Token
The atomic unit an LLM reads and generates — roughly ¾ of a word in English.
/ quick answer
A token is a chunk of text (character, subword or word) that an LLM processes. Pricing, context windows and rate limits are all measured in tokens, not characters. The atomic unit an LLM reads and generates — roughly ¾ of a word in English.
/ frequently asked
What is Token?
A token is a chunk of text (character, subword or word) that an LLM processes. Pricing, context windows and rate limits are all measured in tokens, not characters.
What is an example of Token?
'ChatGPT is great' ≈ 4 tokens. A 4,000-word article is ~5,300 tokens. GPT-4o's 128k context window fits roughly a 300-page book.
Why does Token matter?
The atomic unit an LLM reads and generates — roughly ¾ of a word in English.
What should I learn after Token?
Open the related workflows and prompts further down this page: they show how this idea is applied in a real task, step by step.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →Contextual Compression
Contextual compression is a technique used to reduce the size of the input context for a Large Language Model (LLM) while retaining its most relevant information, typically by summarizing or filtering.
- →Token Budgeting
Token budgeting is the strategic allocation and management of token usage within large language model (LLM) operations to control costs and optimize performance.
- →Token Budget
A token budget is a predefined limit on the number of tokens an AI application or specific request can consume within a given period or for a single interaction. It is a critical mechanism for controlling costs and managing resource allocation for Large Language Model (LLM) usage.
- →LLM (Large Language Model)
A model trained on huge text corpora that predicts the next token to produce human-like language.
Related workflows
Turn this into a repeatable process.
- →How to Create a Website with AI
Go from idea to a live, custom-domain website in one afternoon using AI builders.
- →How to Build an AI Content System
A repeatable pipeline that turns one input into publish-ready content across every channel.
- →How to Start a Niche Website with AI
Pick a niche, validate demand, build the site, and publish ranking content using AI end-to-end.
- →Personal Research Assistant Workflow
A repeatable system to research any topic deeply in under 30 minutes.
Related tool stacks
The tools that run it in production.
- →AI Research & Knowledge Stack
Default toolset for analysts, founders and creators doing deep research with AI.
Comparisons & alternatives
Pick between the options.
- →ChatGPT vs Claude
Two leading conversational AI assistants compared across reasoning, writing, coding, and pricing.
- →Lovable vs Bolt
Two AI app builders compared on speed, backend, deployment, and production readiness.
- →OpenAI API vs Anthropic API
Choosing between the two leading LLM API providers for production apps.
- →Lovable vs Cursor
Prompt-to-app builder vs AI-assisted code editor — which one should you reach for?