Dictionary
Quantization
Shrinking a model by lowering weight precision.
1 min readupdated 2026-07-04
/ quick answer
Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop. Shrinking a model by lowering weight precision.
Shrinking a model by lowering weight precision. Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop. In practice: A 70B model quantized to 4-bit fits in ~40GB VRAM and runs on a single H100. This dictionary node is part of the Onexial knowledge graph and links to related concepts, workflows and tools below.
Definition
Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop.
Example
A 70B model quantized to 4-bit fits in ~40GB VRAM and runs on a single H100.
/ frequently asked
What is Quantization?
Quantization converts 16-bit weights to 8-bit, 4-bit, or lower. It shrinks model size 2-4x with a small accuracy hit — the key trick behind running Llama-scale models on a laptop.
What is an example of Quantization?
A 70B model quantized to 4-bit fits in ~40GB VRAM and runs on a single H100.
Why does Quantization matter for AI and automation?
Shrinking a model by lowering weight precision. It connects to the workflows, prompts and tool stacks linked on this page, so you can move from definition to execution without leaving Onexial.
↳ connected nodes
Comparison↳ linked
Groq vs Together AI
Fast open-model inference: throughput vs breadth.
Comparison↳ linked
Chroma vs Qdrant vs Pinecone
Open-source local vs managed cloud vector databases.
Dictionary↳ linked
AI Router
A layer that picks the cheapest capable model for each request, saving cost and latency.
Dictionary↳ linked
Inference
Running a trained model to produce outputs.
Dictionary↳ linked
Model Routing
Sending each request to the cheapest model that can handle it.
Comparison↳ linked
ChatGPT vs Claude
Two leading conversational AI assistants compared across reasoning, writing, coding, and pricing.