What are tokens and the context window?

Tokens and context window: definition

Language models do not process words but tokens. A token can be a whole short word, but is usually a word fragment, a syllable or a punctuation mark. Long or compound German words are split into several tokens; German texts therefore generally need more tokens than English texts of the same length.

The context window is like the model’s working memory: only what fits inside can be taken into account in the answer. That includes your instruction, attached documents, the conversation so far and the answer itself. Depending on the model, the range runs from a few thousand to well over a hundred thousand tokens.

Why tokens determine cost and quality

Providers usually bill their services per token, separately for input and output. If you send long documents or the entire chat history with every request, you pay for them each time. Response time also grows with the length of the context. It therefore pays to estimate during planning how many requests arise per day and how large the inputs typically are.

A larger window is also not a free pass. With very long inputs, a model can overlook details, especially those in the middle of the text. When the limit is exceeded, the application cuts off older content or returns an error. The model then really “forgets” the beginning of the conversation.

Real-world example: long contracts at a law firm

A law firm wants to use AI to check extensive lease agreements for certain clauses. The entire contract does fit into the context window, but the cost per review is high and some passages are overlooked. A staged approach works better: the contract is split into sections, relevant passages are retrieved selectively and assessed together.

For follow-up questions in the conversation, the application summarizes earlier contributions instead of sending the whole history. This saves tokens, keeps the answers focused and makes review easier for the lawyers, because every statement can be traced to a passage. The staged method is also cheaper, because the complete contract is not processed for every question.

What companies should watch for

Choose the model and window size according to the task, not the largest value. Check prices per token against your realistic volume and plan cost limits. For sensitive documents, also clarify where processing takes place. Test with your own documents, because length and structure differ greatly from industry to industry.

Nano Concepts designs AI applications so that only what is necessary ends up in the context, and makes consumption and cost per case transparent. On request we set up monitoring that evaluates consumption and response times, and we check whether a smaller, cheaper model is sufficient for the task.

• Pass only relevant excerpts instead of whole folders
• Split long documents into sections and retrieve them selectively (RAG)
• Summarize conversation histories instead of sending them in full
• Measure consumption per case and set limits
• Use the provider’s caching for recurring text parts to cut costs, where available