A local AI stack has two layers:

• Backend (inference engine): loads model weights, manages VRAM/RAM and the KV cache, applies quantization, and exposes an API.
• Frontend (interface or workspace): handles chat, files, RAG, users, tools, and agent orchestration.

MCP and OpenAPI add a tool boundary between the frontend or agent and the outside world. They do not replace the backend. Model quality and throughput come from the model and inference engine; workflow quality comes from the client and its tools.

Local AI changes quickly. Verify model cards, licenses, and runtime support before deployment.

Getting started

Pick the persona closest to yours and run its steps in order. The rest of the guide explains each choice.

First-time desktop user

Private chat on one machine.

1. Install LM Studio.
2. Download a model that the hardware-fit estimate marks as safe.
3. Start with 8k-16k context and increase only after measuring memory.
4. Enable the local server if another app needs the model.

The bundled lms CLI can automate model and server management:

lms get openai/gpt-oss-20b
lms load openai/gpt-oss-20b
lms server start

Windows developer with a coding agent

Claude Code running against a local Ollama backend.

1. Install Ollama: winget install Ollama.Ollama.
2. Download a tool-capable coding model: ollama pull qwen3-coder.
3. Raise the context window; the 4,096-token default truncates repository prompts: setx OLLAMA_CONTEXT_LENGTH 65536, then restart Ollama.
4. Optional: keep the model loaded between sessions: setx OLLAMA_KEEP_ALIVE -1.
5. Install Claude Code: npm install -g @anthropic-ai/claude-code.
6. Run ollama launch claude, pick the model, and keep command approvals enabled.

Ollama exposes no MTP or speculative decoding switch as of v0.32.x; those paths run through llama.cpp or vLLM (see Runtime configuration).

Household or small-team chat host

One shared backend with a web frontend and a few users.

1. Install Ollama and verify a model runs:

ollama run gpt-oss:20b

2. Install Open WebUI and point its Ollama connection to http://host.docker.internal:11434 (or the host address appropriate to your container runtime).
3. Add authentication before allowing other users.
4. Set OLLAMA_KEEP_ALIVE on the server, or keep_alive per request (a duration, -1 to keep the model loaded, 0 to unload immediately); the default unloads an idle model after five minutes, so the first request after a pause pays a full reload. (Ollama FAQ )

Multi-user server operator

A vLLM endpoint serving several people or agent clients.

1. Install vLLM on Linux or WSL2.
2. Start the server with explicit bounds:

vllm serve openai/gpt-oss-20b \
  --host 127.0.0.1 \
  --api-key "$VLLM_API_KEY" \
  --max-model-len 32768 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.85

• --host 127.0.0.1: vLLM binds all interfaces by default; keep the engine reachable only from the gateway or reverse proxy in front of it.
• --api-key: the server is unauthenticated without it, it covers only /v1-prefixed routes, and it is a single shared key, so per-user keys, quotas, and request logs belong in the gateway.
• --max-model-len: bounds per-request context length and the worst-case KV demand per sequence; set it to the context you measured, not the model’s advertised maximum.
• --max-num-seqs: caps concurrent sequences; size it from the capacity level found in Capacity measurement.
• --gpu-memory-utilization: the fraction of VRAM vLLM manages for weights and KV cache (0.90 by default); lower it when anything else shares the GPU.

3. Front the endpoint with a gateway or reverse proxy for per-user keys and rate limits.
4. Follow the exact runtime recipe in the model card; if it recommends speculative decoding or an MTP method, benchmark with and without it at the deployment’s real concurrency (see Runtime configuration).

Runtime configuration

The settings below trade memory, latency, and throughput against each other without changing which model runs. Enable them per runtime, then benchmark against the unmodified baseline.

Speculative decoding

Speculative decoding drafts several candidate tokens cheaply, then verifies them with the target model in one forward pass. Accepted tokens are emitted together; rejected drafts are discarded, and the target model’s own token from the verification pass is used instead. The standard rejection-sampling scheme preserves the target model’s output distribution, so only speed changes, in either direction; some head-based methods also offer faster lossy acceptance modes, so check which acceptance scheme the runtime uses before assuming outputs are unchanged. Multi-token prediction (MTP) is the model-integrated variant: prediction heads trained into the model draft its own continuation, with no second model involved.

Where it helps and where it hurts. Speculation trades spare compute for latency. At low concurrency, decoding is memory-bandwidth-bound and compute sits partly idle, so verifying several drafted tokens per pass cuts per-request latency. At high batch sizes, continuous batching already keeps compute busy, and drafted-then-rejected work competes with other requests, so aggregate throughput can drop below the non-speculative baseline. A single-user coding endpoint and a busy shared chat server can reach opposite conclusions on the same hardware, so benchmark with and without speculation at the deployment’s real concurrency.

Watch the acceptance rate. The number that shows whether speculation is working is the draft acceptance rate, together with mean accepted tokens per step. vLLM reports both in its logs and metrics; exact metric names change between versions, so check the current docs. (vLLM ) A low acceptance rate means most drafted tokens are discarded, and end-to-end latency can be worse than running without speculation. Deeper drafting raises the ceiling on tokens per step, but the chance the whole draft is accepted falls with depth, so measured speedup saturates; tune to the measurement. Sampling also matters: greedy and low-temperature decoding generally accept more drafted tokens than high-temperature sampling.

Enabling it per runtime.

• vLLM configures speculation through a speculative_config block, passed as JSON on the CLI or as a dict in Python. The method field selects the approach: a separate smaller draft model, which must share the target’s tokenizer; prompt-lookup ngram, which drafts from the prompt itself, adds no weight memory, and helps most when output repeats input (code edits, extraction, answers that quote retrieved context); or model-integrated heads such as EAGLE, Medusa, and DeepSeek-style MTP. vLLM has renamed this configuration surface across releases, so copy the key names from the current docs rather than from an older example. (vLLM )
• llama.cpp takes a draft model in llama-server and llama-cli via --model-draft, with drafting depth set by --draft-max/--draft-min and a stop threshold by --draft-p-min; --gpu-layers-draft offloads the draft’s layers separately from the target’s. The draft and target must have compatible vocabularies (mismatched pairs are rejected at load), so the usual pairing is a small quantized model from the same family, such as a 0.5B-1.5B draft for a 27B target. Flag spellings drift between releases; confirm against llama-server --help. (llama.cpp )
• Ollama documents no speculative decoding control as of v0.32.x. Check the current docs before planning around it, or serve the model through llama.cpp or vLLM when drafting matters. (Ollama )

A separate draft model spends memory of its own: its weights plus its KV cache join the total, which matters on the 24 GB targets in the sizing table. ngram and model-integrated heads add little or none.

MTP runtime support

MTP training gives a model prediction heads that a supporting runtime uses as a self-drafting speculative path, with the same acceptance-rate economics as above. A runtime without that support, or with it left unconfigured, runs the same model as an ordinary one-token-per-step decoder: output is identical, and the speedup does not occur. This applies to Qwen3.6-27B’s MTP training and DeepSeek V4 Pro’s DSpark decoding; the enabling method names and flags belong to each model card’s runtime recipe, so take them from there rather than guessing. Two further checks: community GGUF conversions do not always retain the MTP head tensors, in which case no runtime can use them, and GGUF runtimes have historically decoded MTP-trained models on the ordinary next-token path. Verify that your build supports the model’s MTP head before choosing a model for that feature.

KV-cache formats

The KV cache grows linearly with context length (and with layer count and KV-head width), so at 128k+ it can rival or exceed the quantized weights. For 262k-context models such as Qwen3.6-27B and Laguna XS 2.1, cache format is a first-order sizing input.

As rough working numbers, q8_0 halves cache memory versus f16 with small measured quality impact in most reports, and q4_0 quarters it with larger degradation. The losses compound with distance: degradation invisible at 8k can surface as long-range recall failures at 64k+, and Q4 weights plus a q4 cache stack two lossy quantizations. Try q8_0 before q4_0, and test long-context retrieval on the real workload. Combinations are also version-specific: whether a given speculative method runs with a quantized KV cache depends on the runtime’s kernels, so verify the pairing in the current docs before stacking both.

Ecosystem overview

• Agentic workflows run locally. Coding clients read repositories, edit multiple files, run tests, and invoke subagents against local endpoints.
• Local engines expose cloud-compatible APIs. Ollama 0.32.x exposes Anthropic Messages compatibility in addition to its local and OpenAI-style APIs, and ollama launch configures supported coding clients directly. (Ollama Anthropic compatibility , Ollama launch )
• MCP is available in self-hosted UIs. Open WebUI supports admin-managed Streamable HTTP MCP servers, while retaining OpenAPI as the safer default for conventional integrations. (Open WebUI MCP )
• Open-weight models span a wide hardware range. Qwen3.6/3.8, Laguna XS 2.1, gpt-oss, and DeepSeek V4 target tool use and long-horizon tasks; hardware needs vary from one 16 GB machine to data-center GPU nodes.
• Memory optimization matters as much as weight quantization. Long contexts make KV cache format, speculative decoding, and multi-token prediction (MTP) important runtime selection criteria. How to enable and measure them is covered under Runtime configuration.

Strategy: bundled vs modular

Default recommendation

• New to local AI: use LM Studio or Jan.
• Building developer workflows: use Ollama plus the client you prefer.
• Serving several users: use vLLM behind an authenticated frontend or gateway.
• Running on Apple Silicon: compare MLX and Metal-backed GGUF builds for the exact model you want instead of assuming one runtime always wins.

OS and hardware guidance

Windows and Linux

• NVIDIA desktop GPU: Ollama is the simplest daily driver. Use vLLM on native Linux or WSL2 when concurrency and continuous batching matter.
• AMD GPU: prefer supported ROCm builds; otherwise test Vulkan through llama.cpp, KoboldCpp, or LM Studio. Backend/model combinations still vary in ROCm maturity.
• Intel Arc or integrated GPU: Vulkan offload can make 7B-14B GGUF models practical, but shared system memory bandwidth remains the limit.
• Ryzen AI NPU: AMD Gaia targets local agent and RAG workflows across Ryzen AI NPUs, iGPUs, and CPUs. Treat NPU support as model- and operator-specific rather than a universal replacement for GPU inference. (AMD Gaia )

Apple Silicon

Unified memory lets the CPU and GPU share one large pool, which is useful for quantized models that exceed ordinary consumer VRAM. The main paths are:

• MLX / MLX-LM: Apple-native runtime and model format.
• LM Studio: supports both MLX and GGUF runtimes, plus the lms CLI and Python/ TypeScript SDKs. (LM Studio , LM Studio CLI , LM Studio TypeScript SDK , LM Studio Python SDK )
• Ollama or llama.cpp: Metal-backed GGUF inference with broad model availability.
• Atomic Chat: bundles upstream llama.cpp, a TurboQuant-enabled llama.cpp fork, and MLX-VLM behind one OpenAI-compatible endpoint. (Atomic Chat )

Do not size an Apple system by “active parameters” alone. All model weights still need storage and usually memory, even when a Mixture-of-Experts (MoE) model activates only a subset for each token.

Backends (inference engines)

Backends should be evaluated on model support, tool-call parsing, context behavior, continuous batching, speculative decoding, MTP support, and hardware acceleration.

Serving under concurrency

For a single request, Ollama wraps a llama.cpp-derived runner, so decode speed is roughly that of llama.cpp on the same hardware and quantization, and switching between the two would not change it. Once requests overlap, scheduling separates the engines:

• Ollama serves a loaded model through a fixed number of slots (OLLAMA_NUM_PARALLEL, default 1 as of 0.32.x). Each slot reserves its own KV-cache allocation, so memory grows with slots times context length. Requests beyond the slot count queue up to OLLAMA_MAX_QUEUE and are rejected past that, and OLLAMA_MAX_LOADED_MODELS bounds how many models stay resident. (Ollama FAQ )
• vLLM admits new requests into the running batch at token granularity (continuous batching) and pages the KV cache in blocks (PagedAttention). Aggregate tokens/s rises with load while per-request tokens/s falls. --max-num-seqs, --max-model-len, and --gpu-memory-utilization bound concurrent sequences, per-request context, and cache memory; when the cache fills, vLLM preempts sequences and recomputes them later rather than failing the request. (vLLM )
• llama.cpp’s llama-server splits its context budget across a fixed slot count (--parallel), so its scaling behavior matches Ollama’s rather than vLLM’s. Its advantages are device coverage and direct control of offload and cache settings.

Under shared serving, KV-cache demand grows with concurrent sequences (the serving formula is under Hardware and model sizing), so a model that fits one long-context session may not sustain several. Agent clients raise the multiplier: one coding-agent session can hold multiple in-flight requests through subagents and retries, so plan for peak concurrent requests rather than user count, and pick client timeouts knowing whether the engine queues then rejects (Ollama), preempts and recomputes internally (vLLM), or waits on busy slots (llama.cpp).

None of these engines provides per-user identity. Ollama’s API has no authentication, and vLLM and the llama.cpp server accept only static API keys with no per-user identity attached. Per-user keys, quotas, and audit logs therefore live in a gateway or frontend (LiteLLM virtual keys, Open WebUI accounts), and the engine endpoint should be network-reachable only from that layer.

Atomic Chat and TurboQuant

Atomic Chat is both a desktop workspace and an inference server. Its documented TurboQuant support compresses the KV cache through turbo3/turbo4 modes, with up to roughly 4.3x smaller KV-cache footprint in the project’s tests. That can make longer contexts practical, but it does not shrink a 70B model’s weights into 6 GB of VRAM. Atomic Chat also exposes an OpenAI-compatible endpoint and can launch supported coding agents from the UI. (Atomic Chat )

Frontends and workspaces

AirgapAI markets deployments for disconnected and SCIF-style environments. That is not the same as a universal government certification. Require written evidence for the specific compliance regime, deployment, hardware, and version you intend to buy.

MCP and tool execution

MCP standardizes how a client discovers and calls tools. Examples include file access, desktop automation, databases, web search, and code execution.

Open WebUI supports Streamable HTTP MCP servers starting with v0.6.31. Its documentation keeps MCP server registration admin-only because MCP is stateful and can cross a much larger trust boundary than a typical stateless OpenAPI endpoint. (Open WebUI MCP )

Use this decision rule:

• Choose OpenAPI for conventional services that benefit from gateways, typed schemas, audit, quotas, and ordinary HTTP controls.
• Choose MCP when the client and tool server need MCP-native capabilities or a shared tool protocol.
• Use neither until you understand what data and host capabilities the integration can reach.

For Open WebUI, set a persistent WEBUI_SECRET_KEY, restrict MCP setup to administrators, scope tools with access control, and avoid exposing unauthenticated MCP servers outside a trusted network.

Coding and agent workflows

A coding agent inspects and changes a repository directly: it reads files, edits them, and runs tests, instead of returning snippets from a chat UI.

Claude Code through Ollama

Ollama’s Anthropic-compatible /v1/messages endpoint supports messages, streaming, vision, tool calls, tool results, and basic extended thinking. The easiest supported setup is:

ollama launch claude

The guided flow selects a compatible model and configures Claude Code. For larger repositories, Ollama recommends a context length of at least 64k. Manual configurations must raise the window explicitly (OLLAMA_CONTEXT_LENGTH at startup or num_ctx per request): the default is 4,096 tokens (Ollama FAQ ), and input past the configured window is truncated rather than rejected; the server logs the truncation, but the API returns no error. The compatibility layer does not implement every Anthropic feature, so check the current support matrix before depending on token counting, prompt caching, batches, or forced tool_choice. (Ollama Claude Code , Ollama Anthropic compatibility )

Other local coding clients

OpenClaw can connect an assistant to WhatsApp, Telegram, Slack, Discord, Signal, iMessage, and other channels. Its main-session tools run on the host unless sandboxing is configured, so review its security and exposure runbooks before enabling remote users or messaging integrations. (OpenClaw )

Agent safety baseline

• Run code tools in an isolated container or VM with a narrow workspace mount.
• Require confirmation for shell commands, package installation, network access, and destructive file operations.
• Do not give a model access to production credentials because it is “local.”
• Treat repository text, web pages, issues, and RAG documents as untrusted prompt input.
• Review diffs and run tests before accepting agent changes.

State-of-the-art local models

“Open weight” does not always mean “fits on a desktop.” Use the hardware column as the starting point, then check the model card and runtime recipe.

Model selection notes

• Best first reasoning model on 16 GB: gpt-oss-20b, if your runtime supports its Harmony format and MXFP4 path.
• Strong 24 GB coding target: a Q4 Qwen3.6-27B build, with context kept realistic.
• Local long-horizon coding on 36 GB+ unified memory: Laguna XS 2.1 is specifically sized and trained for this niche.
• Large-server experimentation: Llama 4 Scout and DeepSeek V4 Pro are open-weight options, but active parameter count understates their storage and memory requirements.
• MTP and DSpark depend on the runtime: Qwen3.6’s MTP heads and DeepSeek V4 Pro’s DSpark path speed up decoding only when the serving engine loads and enables them; on any other runtime the same weights decode one token per step. The enabling flags are in each model card’s runtime recipe.
• Newest is not always best: verify tool-call templates and runtime support before replacing a stable model in an agent workflow.

Hardware and model sizing

Raw quantized weights are only the first part of memory use:

weight memory ~= parameter count x bits per weight / 8
total memory  = weights + KV cache + runtime buffers + vision encoder + context overhead

Long context is not free. A model that loads at 8k context may run out of memory at 64k or 256k. The KV cache grows linearly with context length (per-runtime cache format flags are covered under Runtime configuration). KV-cache quantization, sliding-window attention, and prompt caching can help, but test your real workload instead of relying on the advertised maximum context.

The formula and table above size a single active conversation. Under shared serving, KV cache scales with concurrent sequences:

per-token KV size ~= 2 x layers x KV heads x head dim x bytes per element
serving KV cache  ~= per-token KV size x context length x concurrent sequences

A dense model with 48 layers, 8 KV heads, and a head dimension of 128 stores roughly 0.19 MB per token at FP16: about 6 GB for one 32k-token sequence and about 26 GB for four of them, before weights. FP8 or quantized KV cache reduces this by 2-4x. vLLM logs how many KV-cache blocks fit at startup and reports cache utilization while serving. Measured concurrent capacity, not the single-user table, is what sizes a shared endpoint.

Capacity measurement

Single-stream numbers do not predict shared serving behavior. Four metrics cover most decisions:

• Time to first token (TTFT): queueing plus prompt processing; what an interactive user experiences as responsiveness.
• Inter-token latency: the gap between streamed tokens within one request.
• Per-request decode rate: tokens/s for a single request once generation starts.
• Aggregate throughput: total tokens/s and requests/s across all concurrent requests. Continuous batching raises the aggregate while lowering each request’s rate, so the two must be measured separately.

Procedure: replay a fixed prompt set with realistic prompt and output lengths at increasing concurrency, record p50/p95 TTFT and end-to-end latency at each level, and take capacity as the concurrency step just before your latency target breaks. Test at the context length you will actually run, not the runtime default; Ollama’s 4,096-token default window truncates silently, so a default-configuration test can measure a shorter prompt than the one you intend to serve.

The stacks above ship their own tools: vllm bench serve drives a live endpoint, llama-bench baselines a device for llama.cpp, and ollama run --verbose prints prompt and decode rates for a single run. Results are specific to the model, quantization, context length, and driver/runtime version; changing any of these invalidates the numbers. Cross-engine comparisons are only meaningful at matched quantizations (see Model formats).

Model formats

Never infer license terms from a community quantization. The upstream model license still applies. Quantization changes behavior as well as licensing exposure: a Q4 GGUF can score differently from the upstream safetensors release, and embedded chat and tool templates vary between conversions. When comparing models or engines, hold the quantization constant.

RAG and hybrid routing

For local document workflows, common building blocks include Chroma, Qdrant, LanceDB, and SQLite with sqlite-vec. Keep source metadata, version the index, and record the embedding model and revision with it: an index only answers queries embedded by the model that built it, so changing or upgrading the embedding model means a full re-index. Include retrieval cases in the pre-rollout evaluation suite, and assume retrieved documents can contain prompt injection.

Use LiteLLM when several clients need one endpoint for local and cloud models:

model_list:
  - model_name: local-reasoning
    litellm_params:
      model: ollama/gpt-oss:20b
      api_base: http://host.docker.internal:11434

  - model_name: local-coding
    litellm_params:
      model: ollama/qwen3-coder
      api_base: http://host.docker.internal:11434

Point Open WebUI, IDE clients, and automation at the gateway, then centralize authentication, rate limits, logs, and fallback policy there. Keep provider keys in environment variables or a secrets manager, not in the routing file.

Running it as a service

A starter stack becomes a service the first time something else depends on it. The points below assume containers or Kubernetes; the pinning and lifecycle practices apply to bare metal too.

• Pin what actually changes. An image tag and a model name are both mutable references. Pin the container image digest, and pin the model revision: a Hugging Face commit hash via --revision for vLLM/TGI, and for Ollama record the digest from ollama show and verify it after pull. A redeploy should pull identical bytes.
• Match the host driver to the image. A pinned vLLM image can still fail if the host NVIDIA driver is older than what the image’s CUDA build requires. Record the driver version next to the image digest and upgrade them together.
• Containers need the GPU runtime. Docker requires the NVIDIA Container Toolkit; Kubernetes requires the device plugin or GPU Operator, which can also manage drivers and node metrics. (NVIDIA Container Toolkit , NVIDIA GPU Operator )
• Assume one model server owns the whole GPU. Sharing through MIG or time-slicing is a deliberate configuration with its own isolation trade-offs, not a default.
• Size probes for model load. Loading tens of GB of weights takes minutes. Without a startup probe, a liveness probe with default timings kills the pod mid-load and the kubelet restart-loops it. Use a startup probe sized for weight loading, a readiness probe on the API, and liveness timings that assume the model is already loaded.
• Keep weights on a persistent volume. Otherwise every pod restart re-downloads tens of GB. Point HF_HOME or the Ollama model directory at the volume.

Observability

vLLM exposes Prometheus metrics at /metrics: TTFT and latency histograms, running and waiting request counts, KV-cache utilization, and preemption counts. (vLLM metrics ) The llama.cpp server exposes aggregate Prometheus metrics behind --metrics (queue depth, KV-cache usage, token counters) but no per-request latency histograms; Ollama exposes no native metrics endpoint. Per-request visibility for both comes from the gateway. GPU metrics need their own exporter: DCGM exporter under Kubernetes, or nvidia-smi polling elsewhere. (DCGM exporter )

Watch waiting-queue depth, KV-cache utilization, preemption count, and TTFT percentiles. Sustained queue growth means the endpoint is past capacity; cache near 100% with rising preemptions means context or concurrency limits are set too high. Alert on both, and on restart loops.

Log per-request model, token counts, latency, and caller identity at the gateway. Request logs contain user prompts, so retention is a data-handling decision, not a logging default. vLLM and LiteLLM support OpenTelemetry tracing when a per-request breakdown across the gateway hop is needed.

Changing models and runtimes

Weights, quantization, runtime version, and chat template are one versioned unit. A runtime upgrade alone changes behavior with identical weights: chat template revisions, tokenizer fixes, and quantization kernel changes all alter outputs, and agent workflows notice first.

• Point clients at a gateway alias, such as the local-coding example above, rather than an engine model name, so a swap or rollback is a routing change.
• Stage the candidate next to the incumbent and shift a small share of traffic first where the gateway supports it. Two resident models need memory for both; on a single 24 GB GPU that usually means staging on separate hardware, or swapping with a tested rollback instead of a true side-by-side.
• Keep the previous weights on disk until the new configuration passes evaluation. A rollback that begins with a re-download is not a rollback path.
• Any change to weights, quantization, runtime, or template re-runs the evaluation below.

Pre-rollout evaluation

Public benchmark ranks exercise none of your templates, parsers, or tools. Keep a fixed suite drawn from real usage and run it through the same client and gateway path used day to day:

• Representative prompts from actual workloads.
• Tool-call cases, checked for parseable calls with correct arguments against the runtime’s actual parser and templates.
• Structured outputs validated against their schemas.
• A few scripted end-to-end agent tasks, such as a small repository fix whose tests must pass.

Compare pass rates against the incumbent before switching, and repeat each case several times, because sampling makes single runs noisy. promptfoo runs such suites against an OpenAI-compatible endpoint; a short script against the endpoint also works.

Security and licensing checklist

• Bind local APIs to 127.0.0.1 unless remote access is intentional.
• Put shared endpoints behind SSO/VPN, TLS, authorization, and rate limits.
• Pin container image digests and model revisions, and review model/runtime update notes before rollout. A Hugging Face repository or an Ollama tag can change content under a stable name.
• Download weights only in non-executable formats such as safetensors or GGUF; avoid pickle-based checkpoints.
• Leave trust_remote_code off unless you have read the repository code it executes at load time.
• Record and verify file hashes for weights downloaded outside a pinned revision or digest.
• Treat community quantizations as third-party redistributions: verify integrity as well as license.
• Engine APIs provide no per-user identity: Ollama’s has no authentication, and vLLM and llama.cpp accept only static keys. Put per-user keys, quotas, and audit logs in the gateway or frontend, and make the engine reachable only from that layer.
• Treat MCP servers, browser agents, code sandboxes, and shell tools as privileged software.
• Isolate tool execution from secrets, SSH agents, browser profiles, and unrestricted home directories.
• Read both the application license and model license:
  • Qwen3.6/3.8 and gpt-oss: Apache-2.0.
  • DeepSeek V4 Pro: MIT.
  • Laguna XS 2.1: OpenMDW-1.1.
  • Llama 4 Scout: Meta’s Llama 4 Community License, not Apache-2.0.
• Do not equate “open weight,” “local,” “air-gapped,” or “open source” with independently audited security.
• Back up chats, prompts, indexes, and model configuration if the frontend stores state.

Common pitfalls

• Choosing by benchmark rank without checking license, tool template, or runtime support.
• Sizing MoE models by active parameters instead of total weights and KV cache.
• Enabling a 128k+ context window before measuring memory and latency.
• Exposing Ollama, vLLM, Open WebUI, or MCP directly to the internet.
• Letting an agent execute repository instructions without treating them as untrusted input.
• Assuming a frontend can fix a slow or incompatible backend.
• Using a cloud-tagged model in Ollama and assuming inference is still local.
• Leaving Ollama at its default context window (4,096 tokens as of 0.32.x); input past the configured window is truncated without an error.
• Sizing a shared endpoint from single-request benchmarks instead of measuring at peak in-flight requests, which agent clients multiply beyond user count.
• Comparing engines under different quantizations and attributing the difference to the engine.
• Enabling speculative decoding on a saturated shared server without comparing aggregate throughput against the non-speculative baseline.
• Choosing a model for its MTP training on a runtime that never loads the MTP path, leaving decoding at one token per step.
• Swapping a model or upgrading a runtime without re-running the evaluation suite or keeping a rollback path.