Local LLM: The Complete Private AI Setup Guide

A local LLM is a language model running on hardware you control, with no request leaving your machine. In 2024 that meant a toy — a quantized 7B model answering a question in twelve seconds. In 2026 it means a 70B model, a 128 GB desktop, and a real coding assistant that never phones home.

The demand curve says the same thing. Google’s Keyword Planner puts local llm at 40,500 average monthly searches, up from 2,900 twelve months earlier — a 14x climb in a year, with a single-month spike to 246,000. Ollama carries 135,000. LM Studio carries 60,500. This is no longer a hobbyist niche; it is a category.

This guide is the long version. It covers the software that actually runs models, the hardware worth buying and why memory bandwidth decides everything, the model sizes that fit, what quantization really costs you, how inference speed works, and a setup walkthrough you can follow end to end.

The case for running your own

Three arguments, in descending order of how much they should move you.

Privacy is the first. If your prompts contain client code, medical text, financial records, or anything under an NDA, sending them to a vendor is a data-processing decision with legal weight. A private LLM makes that decision disappear. Nothing egresses. You can run it on a plane.

Cost predictability is the second. A self hosted LLM has a fixed cost — the machine — and a marginal cost near zero. It does not become expensive because you batched a repository, and it does not rate-limit you at 3am when you are trying to finish something. Token economics of agent loops is exactly where hosted inference gets uncomfortable: agentic work multiplies context length, and context length is what you pay for.

Control is the third. You pick the model, the quantization, the system prompt, and the sampling parameters. When a vendor deprecates your model or changes its behavior between versions, your pipeline does not break.

What you give up is real: frontier capability. A 70B open-weights model in 2026 is excellent at coding and summarization and noticeably worse than a frontier model at hard multi-step reasoning. If that trade reads badly, stop here. If it reads fine — which it does for the majority of daily work — continue.

Software: the runtimes compared

The number of ways to run a model locally is small, and they divide cleanly by what they optimize for: ease, speed, or control.

Runtime	Interface	Best at	Formats	Real limitation
Ollama	CLI + local HTTP API	Getting running in minutes	GGUF	Opinionated defaults; less headroom for tuning
LM Studio	Desktop GUI	Non-terminal users, model browsing	GGUF, MLX	Heavier than a CLI; GUI-first workflow
llama.cpp	C++ CLI, libraries	CPU+GPU hybrid, tight memory budgets	GGUF	You configure it; no hand-holding
vLLM	Python server	Throughput with concurrent users	HF safetensors, AWQ, GPTQ	Wants a real datacenter GPU
MLX	Python/C++/Swift API	Apple silicon natively	MLX, GGUF via conversion	Apple hardware only
Open WebUI	Web app	A ChatGPT-like front end over any backend	N/A (front end)	It is a UI; bring your own engine

Ollama

Ollama is the default recommendation for anyone who has not done this before. It runs a background service, exposes an OpenAI-compatible HTTP endpoint on localhost:11434, and turns “download and run a model” into two commands. Nearly every local-LLM tutorial you will find assumes it, which means the error messages you hit have answers.

Underneath, it is llama.cpp plus a model registry plus process management. That inheritance is the point: you get the mature GGUF inference path without compiling anything. The trade is that when you want a specific context length, a particular KV-cache quantization, or a non-default sampling config, you are fighting its abstractions.

LM Studio

LM Studio is the same engine class wrapped in a desktop application. It offers a model browser with size and quantization labels attached, a chat interface, and — importantly for developers — a local server mode that mimics the OpenAI API so existing code works unchanged.

It is the honest answer for someone on a laptop with 16 GB who wants to try a model before committing to hardware. It is a worse answer for automation, because a GUI is not a deployment target.

llama.cpp

The reference implementation. GGUF, the quantization formats in this article, and the per-format performance numbers everyone quotes all come from this project — it is usually written llama.cpp and just as often searched as llama cpp. When you need to squeeze a 30B model into 19 GB by offloading some layers to system RAM and the rest to the GPU, llama.cpp is the thing that does it — -ngl (number of GPU layers) is the knob.

Use it when Ollama cannot express what you need. Do not use it as your first tool.

vLLM

vLLM is a different animal: a serving system for many concurrent requests, not a desktop toy. Its contribution was PagedAttention, which manages the KV cache the way an operating system manages virtual memory. The paper reports:

vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems

That matters if you are serving a team or running batch evaluation. It does not matter if you are the only user, because vLLM’s batching advantage requires concurrency to exploit. It also wants VRAM that fits the whole model without offload — a datacenter or high-end workstation GPU.

MLX

MLX is Apple’s own array framework, built for the unified memory architecture of M-series chips:

MLX is an array framework for machine learning on Apple silicon, brought to you by Apple machine learning research.

Because GPU and CPU share one memory pool, MLX can do something a discrete-GPU setup cannot: hold a model too large for any consumer card’s VRAM entirely on the GPU, because the GPU is not using separate VRAM at all. If you buy a Mac for this, MLX is the path that exploits the purchase.

Open WebUI

A front end, not an engine. Point it at Ollama or an OpenAI-compatible endpoint and you get conversations, model switching, document upload, and user management in a browser. It is what makes a self hosted LLM feel like a product rather than a terminal.

Choosing

First time: Ollama. Want a GUI: LM Studio. Mac: MLX, then Ollama. Constrained memory: llama.cpp with layer offload. Serving other people: vLLM. Every one of these benefits from Open WebUI in front, and choosing among llama.cpp versus ollama versus vllm is really a question about how much control you need versus how much setup you accept.

Hardware: memory bandwidth is the whole game

Most hardware guides lead with TFLOPS. That is the wrong metric for this workload, and understanding why is the single most useful thing in this section.

Generating text is memory-bandwidth bound, not compute bound. Each token produced requires reading every active weight in the model at least once. So the ceiling on tokens per second is roughly:

tokens/sec  ≈  memory bandwidth (GB/s)  ÷  model size in memory (GB)

For a 4.5 GB quantized 8B model on a machine with 153 GB/s, that predicts about 34 tokens/sec. The measured figure in the llama.cpp README for a Q4_K_M Llama 3.1 8B is 71.93 tokens/sec of generation — roughly 2x the naive prediction, because the formula is a floor, not a simulation, and caching, batching, and compute overlap all help. But the shape is right, and it explains the entire hardware market.

A card with more FLOPs but the same bandwidth will not generate faster. A machine with more bandwidth but fewer FLOPs often will.

Apple silicon: the unified memory play

Apple’s advantage is not raw speed. It is that system RAM and GPU memory are the same RAM. A 64 GB Mac mini can address a 64 GB model; a 24 GB NVIDIA card cannot address anything larger than 24 GB without offloading.

Current Mac mini specifications from Apple list three relevant configurations:

Chip	Memory bandwidth	Unified memory
M6 (12-core CPU, 12-core GPU)	153 GB/s	16 GB, configurable to 24 or 32 GB
M6 (higher memory tier)	170 GB/s	24 GB, configurable to 32 GB
M5 Pro (15-core CPU, 16-core GPU)	307 GB/s	24 GB, configurable to 48 or 64 GB

Read that table against the bandwidth formula and the buying decision falls out. At 153 GB/s you can generate quickly from a 7-8B model and slowly from a 30B. At 307 GB/s, a 30B model becomes pleasant and a 70B at Q4 — roughly 43 GB — fits and runs, slowly.

The M5 Pro spec is the interesting row. 307 GB/s in a machine that sits on a desk and draws a fraction of a workstation’s power, with 64 GB addressable, is the best price-per-usable-model-size in the current lineup for single-user inference.

NVIDIA DGX Spark

The DGX Spark inverts the trade. It puts 128 GB of coherent unified memory behind a Grace Blackwell superchip:

With up to 128 GB of unified system memory, fine-tune models up to 70 billion parameters to customize AI models and solutions for specific needs and use cases.

NVIDIA’s page specifies the GB10 Grace Blackwell Superchip with up to a 20-core Grace CPU, up to 1 petaFLOP of FP4 AI performance, up to 128 GB unified memory (64 GB through OEM partners), and ConnectX networking that lets up to four units cluster. Its stated inference envelope is models up to 200 billion parameters; its fine-tuning envelope is 70B.

The split matters. 128 GB plus 1 petaFLOP at FP4 means it is not merely a bigger Mac mini — it is the only desktop-class box in this list whose stated workload includes fine-tuning a 70B model, and whose networking story is clustering four of them into a small node. That is a different product category, at a correspondingly different price.

Discrete GPUs

An RTX-class card with 24 GB or 32 GB of VRAM is the fastest single-token-latency option per dollar, and bandwidth on the order of 1 TB/s means 8B models generate very quickly. The ceiling is hard: 32 GB is 32 GB. You either run a model that fits, or you offload layers to system RAM and watch generation speed collapse because the PCIe link is roughly an order of magnitude slower than VRAM.

For a model that fits, a discrete GPU is excellent. For a model that does not, it is the wrong tool and unified memory is the right one.

The comparison

Platform	Memory	Bandwidth	Strength	Ceiling
Mac mini M6	up to 32 GB	153-170 GB/s	Cheap entry, silent, low power	Bandwidth limits large models
Mac mini M5 Pro	up to 64 GB	307 GB/s	Best single-user price/size ratio	Not a serving box
DGX Spark GB10	up to 128 GB	NVIDIA does not publish a single figure for it	Fine-tune 70B, infer 200B, clusterable	Cost
RTX 24-32 GB	24-32 GB	~1 TB/s	Fastest per-token latency	Hard VRAM wall
Mac Studio Ultra	96-512 GB	Highest in the Mac line	Largest models on one desk	Cost per token

Which models fit

The arithmetic is unforgiving and easy to do. Boundary costs per parameter:

Precision	Bytes per parameter	8B	32B	70B
FP16 (16-bit)	~2.0	~16 GB	~64 GB	~140 GB
Q8_0 (8-bit)	~8.5 bits	~8 GB	~34 GB	~72 GB
Q5_K_M	~5.7 bits	~5.3 GB	~23 GB	~48 GB
Q4_K_M	~4.9 bits	~4.6 GB	~19 GB	~43 GB
Q3_K_M	~4.0 bits	~3.7 GB	~16 GB	~34 GB

The llama.cpp project publishes the real before-and-after for Llama 3.1:

| Model | Original size | Quantized size (Q4_K_M) | | 8B | 32.1 GB | 4.9 GB | | 70B | 280.9 GB | 43.1 GB | | 405B | 1,625.1 GB | 249.1 GB |

Those numbers carry their own warning, also from that README:

As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them.

So: a 70B at Q4_K_M needs roughly 43 GB of addressable memory plus room for the KV cache and the application. On a 64 GB Mac mini M5 Pro, that fits with headroom. On a 128 GB DGX Spark, it fits several times over — which is how a 70B fine-tune becomes possible there.

The tiers that actually work

8B class (7-9B dense): the workhorse. Runs well on 16 GB anything. At Q4_K_M this is the size to start with, and on a fast GPU it generates faster than you can read.

14B-24B class: the sweet spot for a 32 GB machine. Meaningfully better at code and instruction-following than 8B, still fast.

30-32B class: needs 24 GB VRAM or 48 GB unified memory. This is where local models start replacing hosted ones for daily coding.

70B class: needs 48-64 GB unified memory, or a 128 GB box. Slower, noticeably more capable, and the practical ceiling for a single desktop.

Mixture-of-experts models: a different deal entirely. An MoE activates a fraction of its total parameters per token, so a model with a very large total parameter count can run at the speed of its active parameters while requiring memory for all of them. This is how a “huge” model becomes usable locally — you pay in memory for capacity and in bandwidth only for the active slice.

Every model family worth running ships multiple sizes. Pick the size your memory can hold at Q4_K_M with room for KV cache, not the size that sounds impressive.

Quantization: what the bits actually cost

Quantization is the practice of storing weights at lower precision than the training run used. It exists because a 70B model at 16-bit is 140 GB and at 4-bit is 43 GB, and 43 GB fits on a desk.

The Hugging Face overview defines it plainly:

Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible.

The phrase doing the work is “trying to preserve.” Every quantization format is a different answer to where you spend the accuracy.

GGUF and the k-quants

GGUF is llama.cpp’s container format, and the Q4_K_M notation is a k-quant: a mixture scheme where different tensors get different bit widths based on how sensitive they are. The _M means medium; _S is small (more aggressive), _L is large (less aggressive).

The README lists measured bits-per-weight and speed for one 8B model. Three rows are worth studying:

Format	bits/weight	Size	Prompt processing	Generation
Q4_K_M	4.89	4.58 GiB	821.81 t/s	71.93 t/s
Q8_0	8.50	7.95 GiB	865.09 t/s	50.93 t/s
F16	16.00	14.96 GiB	923.49 t/s	29.17 t/s

(The README does not state the hardware those figures were produced on, so treat them as relative, not absolute.)

Two things fall out of that table. First, quantization makes generation faster, not slower — 71.93 versus 29.17 tokens/sec, because fewer bytes cross the memory bus per token. That is the bandwidth formula again. Second, quantization barely touches prompt processing, because prompt processing is compute bound, not bandwidth bound.

Practical guidance: Q4_K_M is the default. It is where the size savings are largest relative to visible quality loss. Go to Q5_K_M or Q6_K if you have memory to spare and care about long-context fidelity. Use Q8_0 when you want near-lossless and have the RAM. Avoid Q2 and Q3 for anything you depend on — the degradation shows up first in instruction-following and code correctness, which is exactly what you cannot see from a demo prompt.

Importance-matrix quantization is the upgrade path. llama-quantize accepts an --imatrix file so the quantizer knows which weights matter, and --tensor-type lets you pin specific tensors to higher precision. That is how people run a 4-bit model that behaves much closer to 6-bit at 4-bit cost.

GPTQ and AWQ

Both are post-training weight-quantization methods for GPU inference, and both target a specific problem: 4-bit GGUF on a datacenter GPU leaves performance on the table, because the card can do more with better-formatted 4-bit math.

GPTQ’s contribution, from the paper: it can quantize a 175B model in approximately four GPU hours, down to 3 or 4 bits, with negligible accuracy degradation — and reported end-to-end inference speedups over FP16 of about 3.25x on an A100 and 4.5x on an A6000. Four GPU hours for a model that large is why GPTQ became the default GPU-side format so quickly.

AWQ (activation-aware weight quantization) takes a different tack, scaling weights to protect the channels that matter most. It tends to hold up slightly better at low bit-widths on some model families.

For a local user these matter less than GGUF, because they assume a GPU-only workload with no CPU offload. If you have the VRAM, use them with vLLM. If you do not, GGUF with offload is the only thing that works.

MLX quantization

On Apple silicon the natural format is MLX’s own, quantizing to 4, 6, or 8 bits with M-series-specific kernels. The advantage is direct access to unified memory with no copies between address spaces. On a Mac, this is usually the fastest path for a model that fits.

Precision in 2026: FP8 and FP4

Newer hardware exposes lower-precision floating point formats — FP8, and FP4 on Blackwell-class silicon — which occupy half the space of their integer equivalents with better dynamic range. The DGX Spark’s headline “up to 1 petaFLOP” is explicitly an FP4 figure, which is a reminder to compare numbers at the same precision. A format that stores weights in half the bytes is not automatically half as good.

Inference speed: prefill, decode, and what you will actually feel

Two phases, two different bottlenecks — and almost every “why is this slow” question is really a question about which phase you are in.

Prefill (prompt processing) ingests your input and builds the KV cache. It is a large matrix multiply, so it is compute bound and parallelizes beautifully. Measured in hundreds or thousands of tokens per second. Its job is to get through your prompt; you experience it as the pause before the first token appears.

Decode (generation) produces one token at a time, autoregressively. Each token requires reading the weights, so it is memory-bandwidth bound and barely parallelizes at all. This is the number people mean by “tokens per second,” and it is the one that determines whether a model feels usable.

The practical consequences:

• Long prompts are cheap, long outputs are expensive. A 20,000-token prompt costs a few hundred milliseconds of prefill. A 2,000-token answer costs 30 seconds at 67 tokens/sec.
• Bandwidth buys generation speed; FLOPs buy prompt speed. If your complaint is “reading my repo takes forever,” you have a prefill problem. If it is “the answer types out slowly,” you have a decode problem, which means bandwidth or a smaller model.
• Offloading destroys generation speed. A layer on the CPU reads from system RAM over PCIe at roughly an order of magnitude less bandwidth than VRAM. Prefill degrades gracefully; decode collapses. This is why fitting the model entirely in fast memory is worth real money.
• Batching helps throughput, not latency. vLLM’s 2-4x gain comes from serving many requests at once. For one user, it is not why you would choose vLLM.

What feels acceptable: below about 10 tokens/sec, reading output is painful. 20-30 tokens/sec matches reading speed. Above 50 tokens/sec, the machine is faster than you are, and going higher buys little for interactive use.

Hardware requirements: sizing table

Goal	Memory needed	Realistic hardware	Model class at Q4_K_M
Try it out, light Q&A	16 GB	Mac mini M6, entry laptop	7-8B
Daily assistant, good quality	32 GB	Mac mini M6 32 GB, RTX 24 GB	8-14B
Serious coding assistance	48-64 GB	Mac mini M5 Pro 64 GB	24-32B
Local frontier, slow but capable	64-128 GB	Mac Studio, DGX Spark	70B
Fine-tuning as well as inference	128 GB	DGX Spark	70B fine-tune
Serving multiple users	Datacenter VRAM	vLLM on high-end GPU	Fits-in-VRAM models

This is a local LLM hardware requirements map in one table, and the honest summary is: 32 GB is the entry point that does not disappoint, and 64 GB is the point where local starts beating hosted for daily work.

Two specs to check before buying anything: memory bandwidth (generate speed) and total addressable unified memory or VRAM (model size ceiling). Ignore core counts.

Setup: from zero to a private AI

This is the local LLM setup path, in order. Commands assume macOS or Linux; Windows equivalents are on each project’s site.

Step 1 — Install the runtime

Ollama is the shortest path:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

Start the service, then pull a model sized to your machine:

ollama serve          # in one terminal, or let the service manager run it

ollama pull llama3.1:8b        # ~5 GB — works on 16 GB
ollama pull qwen2.5-coder:32b  # ~20 GB — needs 32 GB+

Run it interactively, or hit the API:

ollama run llama3.1:8b

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain KV cache in two sentences.",
  "stream": false
}'

That endpoint is OpenAI-compatible in shape, which means existing client code — including agent frameworks and editor plugins — points at http://localhost:11434/v1 and works.

Step 2 — Verify you are actually offline

The whole point is that nothing leaves. Prove it rather than assuming:

# Watch for outbound connections while you chat
sudo lsof -i -P -n | grep -i ollama

You should see only loopback (127.0.0.1) listeners. If you see an established connection to a remote host, something is proxying and you should find out what.

For a genuinely air-gapped setup, download the model on a connected machine, copy the blobs, and run the host with no default route. Local models work fine with no network; that is the entire premise.

Step 3 — Add a real interface

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000, create the first account (it becomes the admin), and point it at your Ollama endpoint. You now have model switching, conversation history, and document upload without a cloud account.

Step 4 — Tune for your hardware

Defaults are conservative. Three knobs matter:

• Context length. Ollama defaults low. Raising it costs KV-cache memory, which comes out of the same pool as the model. Doubling context can double memory footprint on a long conversation.
• GPU layers (llama.cpp / -ngl). Push as many layers onto the GPU as fit. The last layer that does not fit costs far more than its share of speed.
• Quantization level. If the model fits at Q5_K_M, use it over Q4. Quality is free when memory is available.

If you are lowering the context, shrinking the model, or dropping GPU layers, you already have your answer: your machine is at its limit and the next upgrade is memory.

Step 5 — Wire it into your tools

Anything speaking the OpenAI API works. Set the base URL to your local endpoint, set any non-empty API key placeholder, and pick the model name your runtime exposes. Editor assistant, terminal agent, chat front end, batch script — they all take the same three settings.

Set them once with environment variables and every tool inherits them:

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=local        # required by clients, ignored by Ollama

Local LLM for coding

Coding is the workload where local models cross the usefulness line first, for three reasons: the tasks are short and well-specified, the feedback loop is deterministic (code runs or it does not), and the context is yours — a repository you cannot paste into a hosted service.

What works well locally: generating a function from a docstring, writing tests for existing code, explaining an unfamiliar module, refactoring a file you paste in, converting between languages.

What does not work well locally: long-horizon agentic refactors across many files, architectural judgment, and anything requiring the full repository in context. A frontier hosted model still wins those, and pretending otherwise wastes your afternoon.

The practical setup for best local LLM for coding results is a 30B-class code-tuned model at Q4_K_M on 24-32 GB. That configuration handles the useful cases well and is fast enough at roughly 20-30 tokens/sec that you are not waiting.

Security and privacy: the parts people skip

Running locally removes the vendor from your trust boundary. It does not remove every risk, and three are worth naming.

Prompt injection still works. A local model reading a file, a web page, or a tool output can be told to do something by that content. The boundary moved; it did not disappear. If your local agent has shell access, that is a real attack surface regardless of where inference happens.

Model files are executable input. A GGUF is data plus a runtime, and untrusted model files have been a vector for supply-chain problems before. Pull from official registries and check sources rather than downloading a quantized build from a random mirror.

Your logs are now yours to protect. The reason hosted inference was safe for a sensitive prompt was that you did not store it either. Running a private LLM means the transcript sits on your disk, unencrypted by default, in a Docker volume or an app data directory. Decide deliberately whether that is better or worse than the alternative — for most people it is better, but only if the machine is actually secured.

Where a private LLM is clearly the right answer: regulated data, unreleased client work, medical or legal text, and anything where the question itself is the sensitive part.

FAQ

What is a local LLM?

A local LLM is a language model whose weights live on hardware you own and whose inference runs on that hardware. No prompt or completion crosses the network. The model file is a few gigabytes, the runtime is a small executable, and the whole thing works offline once downloaded.

How much RAM do I need to run a local LLM?

For an 8B model at Q4_K_M, 16 GB is enough. For 30B-class models, 32 GB. For 70B, 48-64 GB of unified memory. Add headroom above the model file size for the KV cache, which grows with context length, and for the operating system.

What is the best local LLM to run?

The largest model that fits your memory at Q4_K_M with room for context, in the class suited to your work. There is no single best local LLM — an 8B is best on a 16 GB laptop and a 30B is best on a 32 GB machine. Size relative to hardware determines quality far more than brand.

Does quantization make models worse?

Slightly, and less than most people fear. At Q4_K_M, quality loss is small enough that most users cannot distinguish it from the full-precision model on ordinary tasks. Below Q4 the degradation becomes visible, particularly in instruction-following and code correctness — which is precisely where it is hardest to notice from casual use.

Is a local LLM faster than ChatGPT?

No, not for latency. A hosted frontier model answers faster than a local model of comparable quality running on consumer hardware. The local advantage is privacy, predictable cost, offline operation, and freedom from rate limits — not speed.

Can I fine-tune a model locally?

Yes, with memory to spare. QLoRA showed that 4-bit quantization plus low-rank adapters is enough to fine-tune a 65B model on a single 48 GB GPU. On desktop hardware, fine-tuning a 7-8B model is routine on 24 GB. Fine-tuning is a different job from inference, with its own memory profile — importantly, training needs optimizer state and activations, not just the weights.

Does a local LLM need a GPU?

No. Quantized models run acceptably on CPU alone, especially 8B-class models, at perhaps 5-15 tokens/sec. A GPU or Apple silicon GPU raises throughput substantially, and a machine with high memory bandwidth raises it more.

What to actually do

If you are starting today with no hardware purchase: install Ollama, pull an 8B at Q4_K_M, and use it for a week on real tasks. You will learn what you actually need, which is far cheaper than guessing.

If you are buying: 32 GB is the floor worth spending on, and memory bandwidth is the spec that predicts satisfaction. A Mac mini with 32 GB gets you a good 14B-class daily assistant. A Mac mini M5 Pro with 64 GB at 307 GB/s gets you a 30-70B model and the end of the hosted-API habit for everyday work. A DGX Spark with 128 GB gets you a fine-tuning box and the ability to cluster four of them.

If you want the training side of this — what a model is before you quantize it — the pipeline from tokenizer through post-training is covered in how LLMs are built. And if your reason for going local is cost, the arithmetic of where hosted inference stops being cheap is in cheap intelligence is a cost curve problem.

The gap between local and frontier is real and it is not going to close. The gap between local and good enough for most of your work closed in 2026, and that is the one that matters.