Large language model / LLM

Large language models write fluent text about almost anything. “Fluent” is the key word here: it describes how the text sounds, not whether it is true.

The short answer

A large language model (LLM) is a program trained on huge amounts of text to predict which text comes next. That makes it good at producing useful-sounding writing, but it can state wrong things with confidence, and it only works with what is in front of it at that moment. Treat its answers as drafts that need checking.

How it works

1. Tokens: a model doesn’t read letters or whole words. It works with tokens, the smallest units it handles, which can be words, parts of words, single characters or even bytes.
2. Prediction: in pretraining, models like this learn to predict the next word given the previous text. NIST puts it this way: generative models approximate the statistical distribution of their training data, and LLMs predict the next token. That can produce accurate and consistent text, but it can just as well produce text that is wrong or contradicts itself.
3. The architecture: the Transformer, introduced in 2017, is built on attention mechanisms alone, without the recurrent or convolutional networks used before.
4. The context window: it is everything the model can look at while it writes a response, including the response itself. Anthropic calls it a “working memory”, different from the data the model was trained on. System prompt, every message, tool results, documents and tool definitions all count toward it. More context isn’t automatically better: as the token count grows, accuracy and recall degrade.
5. Confabulation: NIST uses this word for confidently stated but false content, also called “hallucinations”. It is a natural result of how these models are designed. Even the explanations and citations a model gives for its answer can be confabulated.
6. No guarantee of repetition: even with the temperature parameter set to 0, identical inputs may produce different outputs.
7. Grounding: retrieval-augmented generation (RAG) places documents found at query time into the context window, so the model’s answer rests on evidence rather than on memory. How well that works depends on the quality of the retrieved material.
8. Tools and agents: when an LLM can call tools, a manipulated or unclear output can trigger damaging actions (OWASP calls this excessive agency), and specially crafted inputs can change the model’s behavior even if humans can’t see them (prompt injection). OWASP recommends a human-in-the-loop check, in which a person approves high-impact actions before they happen.

In practice

A simple pattern when you give a model a text to work with:

Answer only from the text below. If the answer is not in it, say "not in the text".
Quote the sentence you used.

<your text>

It makes a check easier, because you can look up the quoted sentence. It doesn’t guarantee a correct answer, so you still verify it. At Bytearium, language models matter in two projects: Byte, our AI mascot (in progress), and the planned AI agent studio. The AI assistants that help us run Bytearium work under written rules: statements need evidence, and publishing, spending money, contracts and account changes are for Leon alone. Byte follows a similar line in public: he promises nothing that isn’t backed up. None of this says how good or reliable any particular model is, and we make no such claim.

Running one at home: what the numbers on the label mean

Open-weight models are published with a parameter count (7B, 9B, 70B: billions of weights) and, when packaged for local use, a quantization such as Q4_K_M. Quantization lowers the precision of the stored weights, for example from 16-bit numbers to 4-bit integers, which shrinks the file and can speed up inference at the cost of some accuracy. The llama.cpp project, whose GGUF format most home setups use, measures that loss as perplexity or KL divergence. The practical rule: memory need ≈ parameters × bytes per weight, plus room for the context. A 9B model at 16 bit needs about 18 GB; at 4 bit roughly 5 GB, which fits a laptop GPU. That is exactly how we run Byte’s model on a separate laptop in our network rather than on the streaming PC: a 9-billion-parameter model, 4-bit quantized, served by llama.cpp over the LAN.

How to tell a usable local setup from a slow one

Three questions before the first prompt: does the model plus its context fit into GPU memory, or does it spill into system RAM and run several times slower? How many tokens per second does it produce on your typical request, measured rather than guessed? And who can reach the service on the network, because a model server without access control answers anyone with everything it has learned.

What “local” buys you, and what it doesn’t

A model on your own hardware keeps prompts and documents in the house, costs nothing per request and keeps working when the internet doesn’t. It does not make the model more truthful: confabulation, prompt injection and the context limit apply exactly as they do to a hosted one, and a smaller quantized model usually confabulates more, not less. Treat the local model as a fast drafting tool with the same checking rules, and keep a log of model file, quantization and prompt so that a result can be reproduced after the next update.

Checklist

• Check an answer against its source before you pass it on, and keep answer and evidence apart.
• Give the model the text it should work from, and ask it to quote what it used.
• Note which model, instructions and documents produced a result, so you can reproduce it.
• Set limits on tokens, time and cost before you let a model run unattended.
• Decide who approves high-impact actions, and make that step mandatory.
• Test the setup on tasks that look like your real ones, and include the cases where it should refuse.
• For a local model, record file name, quantization and the memory it actually uses.