Quantization
Shrinking a model by lowering weight precision.
/ quick answer
Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop. Shrinking a model by lowering weight precision.
/ frequently asked
What is Quantization?
Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop.
What is an example of Quantization?
A 70B model quantized to 4-bit fits in ~40GB VRAM and runs on a single H100.
Why does Quantization matter?
Shrinking a model by lowering weight precision.
What should I learn after Quantization?
Open the related workflows and prompts further down this page: they show how this idea is applied in a real task, step by step.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →AI Router
A layer that picks the cheapest capable model for each request, saving cost and latency.
- →Inference
Running a trained model to produce outputs.
- →Model Routing
Sending each request to the cheapest model that can handle it.
Related workflows
Turn this into a repeatable process.
- →Run an LLM Locally
Set up a private, offline language model on your own computer in under an hour.
- →How to Create a Website with AI
Go from idea to a live, custom-domain website in one afternoon using AI builders.
- →How to Build an AI Content System
A repeatable pipeline that turns one input into publish-ready content across every channel.
- →How to Start a Niche Website with AI
Pick a niche, validate demand, build the site, and publish ranking content using AI end-to-end.
Related tool stacks
The tools that run it in production.
- →AI Research & Knowledge Stack
Default toolset for analysts, founders and creators doing deep research with AI.
Comparisons & alternatives
Pick between the options.
- →Groq vs Together AI
Fast open-model inference: throughput vs breadth.
- →Chroma vs Qdrant vs Pinecone
Open-source local vs managed cloud vector databases.
- →ChatGPT vs Claude
Two leading conversational AI assistants compared across reasoning, writing, coding, and pricing.
- →Lovable vs Bolt
Two AI app builders compared on speed, backend, deployment, and production readiness.