You can run AI locally with two parts:

• Backend: the engine that loads and runs models.
• Frontend: the UI you click on. Historically, frontends just talked to a backend and did not decide GPU or VRAM. In 2025 many ship an embedded runner or RAG engine, but performance still comes from the backend you configure.

For example, Open WebUI can run fully offline, connect to Ollama / llama.cpp / vLLM / TGI / LM Studio / other OpenAI-style APIs, and now includes multi-user channels, DMs, knowledge bases + RAG, and tool integrations. (Open WebUI )

Updated December 2025 for GPT-OSS, newer Open WebUI releases, AMD Gaia, and recent Ollama/LM Studio changes.

Quick start

1. Pick a backend that fits your hardware and model format.
2. Pick a frontend to chat or manage models.
3. Or choose an all-in-one app that bundles both.
4. Download a model from a catalog you trust.
5. Check licenses (model + app). Bind services to localhost unless you add auth.

Windows vs Linux: what to install first

Windows

• Discrete NVIDIA GPU (desktop/server)

  • Use Ollama, LM Studio, llama.cpp, or text-generation-webui.
  • Ollama now ships an official desktop app for Windows/macOS with GUI chat, history, and drag-and-drop files, while still exposing the same local API/CLI.
  • For high-throughput multi-model serving, run vLLM on WSL2 or a Linux host and expose an OpenAI-style endpoint.

• Laptops / mini PCs with iGPU (AMD/Intel) or Intel Arc

  • Prefer LM Studio or llama.cpp / KoboldCpp builds that use Vulkan. LM Studio can offload layers to AMD and Intel iGPUs via Vulkan, which is significantly faster than CPU-only on integrated-graphics systems.
  • Start with 7B–13B GGUF models; 20B (e.g., gpt-oss-20b) is realistic on 16 GB-class machines with good offload.

• Ryzen AI laptops / NPUs

  • Consider AMD Gaia: GUI + CLI that runs local LLM agents on Ryzen AI NPU + iGPU, using a RAG pipeline and an OpenAI-compatible REST API. It also runs on non-Ryzen systems (at lower performance).

• Frontends

  • Open WebUI or LibreChat pointed at your local endpoint. Open WebUI is now closer to a self-hosted “ChatGPT + team chat” for your own models (channels, DMs, knowledge bases, tools) and uses a custom BSD-3-based license with a branding clause in v0.6.x.
  • Page-level browser extensions (e.g., Page Assist) can talk to Ollama/LM Studio APIs if you prefer in-browser chat.

Linux

• NVIDIA (server / homelab)

  • vLLM (v1) and Text Generation Inference (TGI) 3.x are the standard high-throughput OpenAI-style servers. vLLM focuses on efficient serving; recent releases add architectural speed-ups and improved multimodal support. TGI adds multi-backend support (TensorRT-LLM, vLLM, etc.).
  • For simpler setups, Ollama or llama.cpp remain practical single-node servers.

• AMD GPUs

  • Use ROCm builds when available (vLLM/TGI/llama.cpp), or KoboldCpp / llama.cpp with Vulkan.
  • Gaia has Linux support as well, but its sweet spot is Ryzen AI laptops with NPU + iGPU.

• Use Docker for servers. Bind to 127.0.0.1 and put a UI (Open WebUI, LibreChat, etc.) in front.

Rule of thumb: VRAM caps the model size. Smaller quantized models still beat larger high-precision models on weak hardware. Start with 7B, then move up.

Frontend

Frontends are thin clients from the user’s perspective. They connect to whatever backend you run (local engines, remote APIs, or both). Some now bundle light backends, RAG, and multi-user features, but you still point them at LLM endpoints.

Backend

Engines that load models and expose a local API.

Both: frontend + backend in one

Image / video UIs

Hardware sizing (plain rules)

These are still rough, but align with 2025 open-weight releases:

• 7B q4: ~4 GB VRAM/RAM.
• 13B q4: ~8 GB.
• 20B (e.g., gpt-oss-20b): ~16 GB VRAM or a mix of VRAM + fast RAM.
• 70B in heavy quant: ≥24 GB VRAM, often more.
• Bigger context windows need more memory. Prioritize VRAM (or NPU-accessible RAM) over raw GPU cores for LLMs.

Model formats you’ll see

Modern “flagship” open-weight families like GPT-OSS-20B/120B, Qwen3, Gemma 3, and DeepSeek R-series/V-series usually ship HF safetensors plus community quantizations in GGUF/GPTQ/AWQ/EXL2.

Local RAG building blocks

Tip: keep chunks ~500–1000 tokens, store sources, and version your indexes. Many frontends (Open WebUI, AnythingLLM, Gaia) now have built-in RAG layers using one of these patterns plus an embedding model.

Speech and media blocks

60-second installs

Windows (NVIDIA, beginner-friendly)

1. Install Ollama (desktop app includes CLI + GUI).
2. Open a terminal: ollama run gpt-oss:20b or ollama run llama3 to test.
3. Install Open WebUI or LibreChat . Point to http://localhost:11434. (GitHub , Ollama , Open WebUI , LibreChat )

Windows (laptop / mini PC, no big GPU)

1. Install LM Studio .
2. Use its model browser to download a 7B–20B GGUF model (e.g., GPT-OSS-20B, Gemma 3 12B, Qwen3-Coder).
3. In the model settings, enable GPU offload to your AMD/Intel iGPU, then enable the local API if you want to connect Open WebUI/LibreChat.

Windows (Ryzen AI)

1. Install AMD Gaia using the Hybrid installer on a Ryzen AI PC.
2. Choose a built-in agent (chat, YouTube Q&A, code) and attach your documents or repos.
3. Optionally call Gaia via its REST API or MCP interface from tools that speak OpenAI-style APIs.

Linux (NVIDIA, server)

1. Run vLLM or TGI via Docker to expose an OpenAI-style endpoint.
2. Put Open WebUI or LibreChat in front for your UI. (VLLM Documentation , Hugging Face , Open WebUI , LibreChat )

Windows or Linux (desktop GUI)

Use LM Studio or GPT4All. Download a 7B GGUF, enable the local API, then connect your frontend if needed. (LM Studio , Nomic AI )

API interop map

• OpenAI-style servers: llama.cpp server, vLLM, TGI, Ollama, LM Studio, AMD Gaia. (GitHub , VLLM Documentation , Hugging Face , LM Studio , Ollama , 30 )
• Kobold API: KoboldCpp, KoboldAI backends. (GitHub )
• text-generation-webui: adapters for multiple backends. (GitHub )

Most 2025 frontends expect an OpenAI-style API; if your backend exposes one, you can usually swap it in without changing the UI.

Model catalogs

Security and licensing

• Bind to 127.0.0.1 by default.
• If you must expose a port, use a reverse proxy and auth (at minimum HTTP auth, preferably SSO or VPN).
• Read model licenses (GPT-OSS: Apache-2.0, Qwen/Gemma/DeepSeek: mostly permissive) before commercial use.
• Read app licenses too. Open WebUI v0.6.6+ uses a custom BSD-3-based license with a branding clause; white-labelling or rebranding may require an enterprise license.
• Store API keys in env vars or a secrets manager.
• Treat LLM backends as you would any network service:
  • Run them under non-privileged users.
  • Keep them patched.
  • Only install images/binaries from reputable sources.
• Be aware that local AI stacks can be abused. For example, security researchers have already shown ransomware using GPT-OSS-20B locally via the Ollama API to generate and run attack scripts. If you allow untrusted code to talk to your local LLM stack, include that in your threat model.

Common pitfalls

• Pulling a model larger than your VRAM/RAM. Start small.
• Mixing GPU drivers or CUDA/ROCm versions. Keep them clean.
• Exposing services publicly with no auth.
• Assuming the frontend controls performance. It does not; the backend and model choice do.
• Forgetting that many “frontends” (Open WebUI, AnythingLLM, Gaia) now include RAG indexes and agents; back them up and secure them like any other data store.