Local AI Stack
Open-source tools to run chat, retrieval and agents entirely on your own hardware.
/ quick answer
Private AI for sensitive data with no per-token cost. Open-source tools to run chat, retrieval and agents entirely on your own hardware.
- Ollama or llama.cpp (runtime)
- vLLM (multi-user serving)
- Open WebUI (chat interface)
- Local embedding model
- pgvector or Qdrant (vector store)
- Quantized open-weight instruct model
- LM Studio for a desktop GUI
- Hosted private endpoints in your cloud account
/ frequently asked
What is the Local AI Stack stack for?
Private AI for sensitive data with no per-token cost.
Which tools are in this stack?
Ollama or llama.cpp (runtime), vLLM (multi-user serving), Open WebUI (chat interface), Local embedding model, pgvector or Qdrant (vector store), Quantized open-weight instruct model.
Are there alternatives to this stack?
Yes — LM Studio for a desktop GUI, Hosted private endpoints in your cloud account.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →Local LLM
A language model run on your own machine or server instead of a cloud API, so prompts and data never leave your infrastructure.
- →Small Language Model (SLM)
A language model with roughly 1–15B parameters, small enough to run on a laptop, phone or single GPU while handling narrow tasks well.
- →Reasoning Model
An LLM trained to spend extra compute thinking through a problem internally before answering, improving accuracy on math, code and planning.
Related workflows
Turn this into a repeatable process.
- →Run an LLM Locally
Set up a private, offline language model on your own computer in under an hour.
Related tool stacks
The tools that run it in production.
- →Local SMB AI Stack
Small local business running marketing + ops with 1 person + AI.
- →Indie SaaS Launch Stack
Everything a solo founder needs to ship and monetize a SaaS in weeks.
- →AI Marketing Ops Stack
The control center for an AI-augmented marketing team of one to five.
- →Agent Architecture Stack
The minimum tooling to design, run and observe a production agent.
Comparisons & alternatives
Pick between the options.
- →Local LLM vs Cloud API
Local models win on privacy and fixed cost; cloud APIs win on quality, scale and zero maintenance.
- →Reasoning Model vs Standard LLM
Reasoning models are more accurate on multi-step problems but slower and pricier; standard calls suit simple, fast tasks.