Building a local AI workflow for data that cannot leave the laptop

The dataset is a folder of 340 PDFs: client contracts, medical intake forms, internal audit notes. Legal has said, in writing, that they cannot be uploaded to a third-party API. Not "shouldn't" — cannot. Which means the usual architecture, a Python script that chunks documents and posts them to a hosted embedding endpoint, is off the table before it's written.

This is a more common constraint than the AI tooling ecosystem implies. Most tutorials assume the data can travel. The ones that mention local inference usually stop at "run Ollama and chat with a model," which is the easy 10% of the problem. The hard 90% is everything around it: getting documents out of formats that fight you, keeping the pipeline from leaking through a hundred small doors, and accepting that a laptop GPU is not a datacenter.

This is a build guide for that 90%, based on the architecture that actually works when the constraint is real.

What "cannot leave the laptop" has to mean, precisely

Vague constraints produce vague architectures. Before writing code, turn the policy into a list of testable properties.

Data at rest stays on disk. Source documents, extracted text, embeddings, prompts, and outputs. No sync folder, no cloud-backed drive, no ~/Documents that's actually iCloud.

Data in transit never crosses the network interface. That includes the model download step (do it once, on a permitted network, then work offline), crash reporting, update checks, and telemetry from whatever runtime you pick.

Model weights are assets, not data, but they still need a decision. Downloading a 4 GB model from a public registry is usually acceptable; it's outbound-only and contains no client data. Get that in writing too, because "we downloaded a model" and "we sent client data somewhere" get conflated in audits.

The derived artifacts inherit the classification. Embeddings are not anonymized data. A vector index over contracts, plus the ability to invert approximate nearest-neighbor lookups, is a partial reconstruction of the corpus. Treat the index with the same rules as the PDFs.

Logs are the most common leak. A pipeline that writes request payloads to ~/.local/share/.../debug.log and then attaches that log to a support ticket has leaked the corpus. Decide the logging policy before the first run, not after the first incident.

Write those five down. They become your acceptance tests, and they're the reason the rest of this article spends as much time on egress as on models.

Choosing a local runtime

There are four realistic options, and the choice matters less than people think — the differences are in ergonomics and hardware support, not in output quality for the same weights.

Runtime	Best for	Trade-offs
Ollama	Fastest path to a working pipeline; simple HTTP API; good model library	Less control over sampling and memory; server process to manage
llama.cpp	Maximum control, GGUF quantization, embedding + generation in one binary	You build it, you manage flags, more surface area
LM Studio	Non-engineers reviewing outputs; GUI model management	GUI-first; scripting is secondary
MLX (Apple Silicon)	Best throughput per watt on M-series; unified memory	Apple-only; smaller ecosystem

For a pipeline that runs on a work laptop and is maintained by one or two engineers, Ollama is the pragmatic default: it exposes an HTTP API on localhost:11434, manages model downloads, and keeps a model resident in memory between calls. You can swap to llama.cpp later without changing the shape of the pipeline, because both take the same GGUF weights.

The decision that does matter is hardware. Unified-memory machines (Apple Silicon) can hold a model larger than their GPU's nominal VRAM, because there's no separate VRAM. An M-series laptop with 32 GB can run a 13B model at 4-bit comfortably. A Windows laptop with 8 GB of discrete VRAM is limited to roughly 7B at 4-bit, and the model will partly spill to CPU if you push further, at a large speed cost.

The memory arithmetic you need before picking a model

Most "it's too slow" complaints are memory problems. Do the arithmetic first.

Rough working set:

weights_gb  ≈ params_billions × bits_per_weight / 8
kv_cache_gb ≈ 2 × layers × kv_heads × head_dim × ctx_tokens × bytes_per_elem / 1e9
overhead_gb ≈ 0.5–1.5 (runtime, tokenizer, buffers)

For a 7B model at 4-bit, weights are about 3.5 GB. A 13B at 4-bit is about 6.5 GB. The KV cache is the part people forget: it grows linearly with context, and at 32k tokens with a typical 7B configuration it can add several gigabytes on its own. Conversation history and long document chunks both extend it.

Practical rules for a laptop:

• 32 GB unified memory: 13B at Q4_K_M with 8k context is comfortable. 30B at Q4 works but leaves little room for the rest of your desktop.
• 16 GB unified memory: 7B–8B at Q4 with 4k–8k context. Don't plan on 13B.
• 8 GB discrete VRAM: 7B at Q4 with a small context, or a 3B model with a large one. Expect partial CPU offload.
• CPU only: 3B models are usable for summarization at reading speed; 7B is painful for interactive work but fine for overnight batch jobs.

Quantization labels matter. Q4_K_M is the usual sweet spot: 4-bit with a mixed scheme that keeps important tensors at higher precision. Q5_K_M is slightly better quality and noticeably more memory. Q8_0 is near-lossless and roughly doubles the footprint for output differences most people can't measure on extraction tasks. For a pipeline doing structured extraction from documents, Q4_K_M is almost always the right trade.

💡 Practical note: the model that fits is the model you'll actually use. A 30B model that takes 40 seconds per document gets abandoned in favor of a 7B model that takes 4. Pick the size that keeps the loop tight, then measure whether it's good enough.

The pipeline shape

Every local RAG-ish workflow is the same six stages, and the local constraint only changes the implementation of each:

ingest → extract text → chunk → embed → store → retrieve → generate → write

The interesting parts are where local implementations differ from hosted ones.

Ingest is filesystem walking with hashing. Content hash, not mtime — you want to know when a document's content changed.

Extraction is where most projects lose weeks. PDFs are not text. They're drawing instructions with embedded fonts, and the text layer may be missing (scanned), garbled (bad encoding), or in the wrong order (multi-column layouts). Local OCR is non-negotiable if the corpus is scanned, and it's slow enough that you want it cached hard.

Chunking on a laptop should favor smaller chunks than you'd use with a hosted model, because chunks become context, and context becomes KV cache. 600–900 tokens with 15% overlap is a reasonable starting point for prose documents.

Embedding is cheap and fast locally if you pick a small model. A 384-dimension model like all-MiniLM-L6-v2 runs at thousands of chunks per minute on CPU. A 768- or 1024-dimension model is slower but meaningfully better on domain text. Embed once and persist — never re-embed on every query.

Storage can be SQLite with a brute-force cosine scan up to a few hundred thousand chunks. At 340 PDFs averaging 30 chunks each, that's 10,000 vectors: brute force over 10,000 × 384 floats is a few milliseconds in NumPy. You do not need a vector database. You need an array and a dot product.

Generation is one prompt with retrieved chunks and a strict instruction to stay inside them. Keep the temperature low for extraction, higher for synthesis.

Output is written to a local directory the pipeline owns, with a manifest.

A working implementation

The following is a complete-ish pipeline for the extract-embed-answer loop. It assumes Ollama is running locally with nomic-embed-text and a chat model pulled. Everything writes to a single project directory.

# local_rag.py — no network calls except to 127.0.0.1:11434
from __future__ import annotations

import hashlib
import json
import sqlite3
import struct
from pathlib import Path

import httpx
import numpy as np

ROOT = Path.home() / "private-rag"
DOCS = ROOT / "docs"
DB = ROOT / "index.sqlite3"
OLLAMA = "http://127.0.0.1:11434"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "qwen2.5:7b-instruct-q4_K_M"

# A client with no proxy and a short timeout: fails fast if Ollama isn't up.
client = httpx.Client(base_url=OLLAMA, timeout=120.0, trust_env=False)

trust_env=False is doing real work here. It ignores HTTP_PROXY and HTTPS_PROXY from the environment, which on a corporate laptop are often set globally. Without it, a request to 127.0.0.1 can be routed through a proxy that logs it. Small detail, exactly the kind that shows up in an audit.

SCHEMA = """
CREATE TABLE IF NOT EXISTS docs (
    id TEXT PRIMARY KEY,
    path TEXT NOT NULL,
    sha256 TEXT NOT NULL,
    title TEXT
);
CREATE TABLE IF NOT EXISTS chunks (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    doc_id TEXT NOT NULL REFERENCES docs(id) ON DELETE CASCADE,
    ordinal INTEGER NOT NULL,
    text TEXT NOT NULL,
    embedding BLOB NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_chunks_doc ON chunks(doc_id);
"""

Embeddings go in as raw float32 blobs. SQLite has no native array type, and JSON-encoded floats are roughly 4× larger and much slower to parse. This is the kind of thing that matters at 10,000 chunks and decides whether the prototype survives.

def pack(vec: list[float]) -> bytes:
    return struct.pack(f"<{len(vec)}f", *vec)


def unpack(blob: bytes) -> np.ndarray:
    n = len(blob) // 4
    return np.frombuffer(blob, dtype="<f4", count=n)


def embed(texts: list[str]) -> np.ndarray:
    resp = client.post("/api/embed", json={"model": EMBED_MODEL, "input": texts})
    resp.raise_for_status()
    vecs = np.asarray(resp.json()["embeddings"], dtype=np.float32)
    # Normalize once so cosine similarity is a dot product later.
    norms = np.linalg.norm(vecs, axis=1, keepdims=True)
    return vecs / np.clip(norms, 1e-8, None)

Normalizing at write time means the similarity scan is a single matrix-vector product instead of a norm computation per row. It's the difference between 8 ms and 60 ms on 10,000 chunks, which matters when you're retrieving on every keystroke in a UI.

def search(query: str, k: int = 6, doc_prefix: str | None = None) -> list[dict]:
    q = embed([query])[0]
    sql = "SELECT c.id, c.doc_id, c.ordinal, c.text, c.embedding FROM chunks c"
    args: tuple = ()
    if doc_prefix:
        sql += " JOIN docs d ON d.id = c.doc_id WHERE d.path LIKE ?"
        args = (f"{doc_prefix}%",)
    rows = db.execute(sql, args).fetchall()
    if not rows:
        return []
    mat = np.stack([unpack(r[4]) for r in rows])
    scores = mat @ q
    top = np.argpartition(-scores, min(k, len(scores) - 1))[:k]
    top = top[np.argsort(-scores[top])]
    return [
        {"doc_id": rows[i][1], "ordinal": rows[i][2], "text": rows[i][3],
         "score": float(scores[i])}
        for i in top
    ]

Retrieval stays in NumPy and never leaves the process. At 10,000 chunks the matrix load is the slow part, which is why the next step is caching it in memory rather than re-reading blobs per query.

Keeping the machine actually offline

This is the section most tutorials skip, and it's the one that determines whether the architecture satisfies the constraint.

Block egress at the OS level, not in your code. Application-level "I won't call the network" is an intention. A firewall rule is a guarantee. On macOS, pf or a per-app firewall; on Linux, ufw with an allow-list for lo and deny for everything else during processing; on Windows, Windows Defender Firewall outbound rules. The goal is that a mistake produces a connection error rather than a quiet upload.

Put the libraries in offline mode. Hugging Face libraries will phone home for model metadata and telemetry unless told not to:

export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HUB_DISABLE_TELEMETRY=1
export DO_NOT_TRACK=1
export OLLAMA_NOPRUNE=1

Verify rather than assume. Run the pipeline under a packet capture and confirm the only destination is loopback:

sudo tcpdump -i any -n 'not host 127.0.0.1 and not host ::1' &
python local_rag.py --ingest ./docs
sudo kill %1

If that produces output, you have a leak, and you have it before you've processed anything sensitive. This is the single highest-value twenty minutes in the whole project.

Check the model's own behavior. Some runtimes fetch model cards, check for updates, or report anonymous usage. Read the config for whatever you pick and disable what you can. Ollama's server is local by default but does contact its registry on pull operations — so pull models during setup, not during processing.

⚠️ Gotcha: pip install during a "secure" session pulls arbitrary code from PyPI. If the machine must stay clean, build a wheelhouse on a permitted machine, copy it over, and install with --no-index --find-links. Same for the model files.

Extraction: the part that decides whether this works

If the corpus is native PDFs with a clean text layer, pypdf or pdfplumber gets you 80% of the way. If it's scanned, you need OCR, and the quality difference between a naive and a careful setup is enormous.

import subprocess
import tempfile
from pathlib import Path


def extract_text(path: Path) -> str:
    if path.suffix.lower() in {".txt", ".md"}:
        return path.read_text(encoding="utf-8", errors="replace")

    if path.suffix.lower() == ".pdf":
        probe = subprocess.run(
            ["pdftotext", "-layout", str(path), "-"],
            capture_output=True, text=True,
        )
        text = probe.stdout
        # Heuristic: a text layer this thin means a scan, not a document.
        if len(text.strip()) > 200:
            return text
        return ocr_pdf(path)

    raise ValueError(f"unsupported: {path}")


def ocr_pdf(path: Path) -> str:
    with tempfile.TemporaryDirectory() as tmp:
        # ocrmypdf wraps tesseract, deskews, and produces a text layer.
        subprocess.run(
            ["ocrmypdf", "--skip-text", "--deskew", "--optimize", "1",
             str(path), f"{tmp}/out.pdf"],
            check=True, capture_output=True,
        )
        out = subprocess.run(
            ["pdftotext", "-layout", f"{tmp}/out.pdf", "-"],
            capture_output=True, text=True, check=True,
        )
        return out.stdout

The --skip-text flag is important: it OCRs only pages without a text layer, so a 200-page PDF with three scanned inserts doesn't get fully re-OCR'd. On a laptop, full re-OCR of a large PDF is minutes of fan noise.

Two extraction problems that will bite regardless of tooling:

Multi-column layouts get interleaved by naive extractors, producing sentences that alternate between columns. -layout helps; nothing fully solves it except per-document inspection.

Tables become word soup. If the documents are contract schedules or financial statements, extraction quality — not model quality — will cap your accuracy. Handling tables usually means a dedicated extractor (camelot, pdfplumber's table API) and storing them as structured rows alongside the text, not flattening them into prose.

Accept that extraction errors compound. A chunk of interleaved column text embeds poorly, retrieves for the wrong queries, and produces confident wrong answers. When a local RAG system is "bad," the model is rarely the cause.

Prompting a small local model

A 7B model is not a frontier model, and prompts written for the latter underperform. Three adjustments matter.

Narrow the task. "Summarize this contract" produces mush. "Extract the parties, effective date, termination notice period, and governing law as JSON; use null when a field is absent" produces usable output.

Give it the schema, and validate the output. Local models drift from JSON more often, so parse defensively and re-ask once with the error message.

EXTRACT_SCHEMA = {
    "type": "object",
    "properties": {
        "parties": {"type": "array", "items": {"type": "string"}},
        "effective_date": {"type": ["string", "null"]},
        "termination_notice_days": {"type": ["integer", "null"]},
        "governing_law": {"type": ["string", "null"]},
    },
    "required": ["parties", "effective_date", "termination_notice_days", "governing_law"],
}


def extract_fields(chunk: str) -> dict:
    prompt = (
        "Extract contract metadata from the text below.\n"
        f"Return JSON matching this schema exactly:\n{json.dumps(EXTRACT_SCHEMA)}\n\n"
        "Use null for anything not stated. Do not guess.\n\n"
        f"TEXT:\n{chunk}"
    )
    resp = client.post("/api/generate", json={
        "model": CHAT_MODEL,
        "prompt": prompt,
        "stream": False,
        "format": "json",          # Ollama constrains decoding to valid JSON
        "options": {"temperature": 0.0, "num_ctx": 8192},
    })
    resp.raise_for_status()
    return json.loads(resp.json()["response"])

format: "json" constrains the sampler so the output parses. It does not constrain the content to your schema, so validation is still required — a model can return valid JSON with termination_notice_days: 30 when the contract says 60.

Batch for throughput, not latency. A laptop generates maybe 15–40 tokens/second for a 7B Q4 model depending on hardware. If you have 340 documents and each answer is 300 tokens, that's 40 minutes to 2 hours of pure generation. Run it as a resumable batch job with one output file per document and a manifest, not as an interactive loop.

# Resume-safe batch run: skip anything already processed
for f in "$DOCS"/*.pdf; do
  out="$(basename "$f").json"
  [ -f "$ROOT/out/$out" ] && continue
  python local_rag.py --extract "$f" > "$ROOT/out/$out"
done

Reality check on performance

Numbers vary by an order of magnitude across hardware, so treat any single figure with suspicion. What's stable is the shape:

• Embedding is fast and embarrassingly parallel. A small model handles thousands of chunks per minute on CPU. This is never the bottleneck.
• Prompt processing (prefill) scales with input length and is where long retrieved contexts hurt. Feeding 8,000 tokens of context to a 7B model on CPU can take 10–30 seconds before a single output token appears.
• Generation is the visible cost. On unified-memory Apple Silicon, a 7B Q4 model typically produces somewhere in the range of 20–40 tokens/second; on a mid-range laptop CPU, closer to 5–10.

The design consequence: keep retrieved context small. Three to six chunks of 700 tokens beats twelve chunks of 700 tokens, both in latency and in accuracy — a small model gets distracted by irrelevant context faster than a large one. Reranking retrieved chunks with a small cross-encoder, or even a simple keyword-overlap score, is often worth more than a bigger generation model.

🔍 Why this matters: on a laptop, the cost of context is paid twice — once in prefill time and once in KV-cache memory. The instinct to "give the model more information" is actively harmful here.

Improving quality without a better model

When a local pipeline gives weak answers, the reflex is a bigger model. On a laptop, that's usually the wrong lever: you can't afford the memory, the speed, or the thermal budget. The levers that actually move quality, in the order I'd pull them:

1. Fix extraction before anything else. Print ten random chunks and read them. If they contain running headers, page numbers, interleaved columns, or hyphenated line breaks, no retrieval or generation improvement will rescue the output. Cleaning chunks — stripping repeated headers, joining hyphenated words, dropping chunks under 80 characters — is a few lines of code with an outsized effect.

2. Add a lexical retrieval pass. Dense embeddings are bad at exact identifiers: clause numbers, invoice IDs, section references. A cheap BM25 or FTS5 index alongside the vectors, with results merged, catches the queries where semantic similarity retrieves something "related" instead of the exact match the user asked for. SQLite ships FTS5 in the box, so this costs nothing but a second table.

3. Retrieve at chunk level, generate with document context. Small chunks retrieve well and generate badly. The compromise: retrieve the top chunks, then expand each to include its immediate neighbors before sending to the model. You get coherent paragraphs in the prompt without needing giant chunks in the index.

4. Add a refusal path. The most valuable behavior for a local system with a weak model is knowing when to say nothing. A simple guard: if the best similarity score is below a threshold calibrated on your own corpus, answer "I couldn't find this in the documents" instead of synthesizing. Calibrate by running 30 real questions and looking at the score distribution — the threshold is usually obvious in the data and impossible to guess.

5. Keep a small labeled set. Twenty questions with known answers, run on every change. Without it, every tweak is a guess and quality drifts in whichever direction the last anecdote pointed. This is the single practice most local-AI projects skip, and it's why they can never say whether the pipeline got better.

None of these require a larger model. All of them are more effective than upgrading from 7B to 13B, which mostly buys you better prose in exchange for half the throughput.

Where this architecture is the wrong choice

Local inference is the right answer to a real constraint, and it's the wrong answer to a lot of imagined ones.

It's wrong when the task is genuinely hard reasoning over long context. A 7B model reading 50 pages and reconciling conflicting clauses will underperform a frontier model by a margin no amount of prompt engineering closes. If the constraint is policy rather than physics, the better architecture is usually a hosted model with a contractual data-processing agreement and a redaction layer, not a laptop.

It's wrong when the output needs to be produced at a rate the hardware can't sustain. Four hundred documents a week on one laptop is fine. Four thousand an hour is not, and buying a GPU is a different project.

And it's wrong when nobody will maintain it. A local pipeline is a small system with real operational surface: models change, quantization formats change, OCR fails on new document types, disk fills. If the person who built it leaves, the corpus becomes unqueryable. That's a worse outcome than a hosted service with a contract.

The honest framing: local inference trades money and convenience for control, and the trade is worth it exactly when the control is required and the workload fits a single machine. Everything else is a preference dressed up as a constraint.