Text becomes tokens

The model receives token IDs, not “words.” Tokenization is a compression-like compromise. Whole-word vocabularies are too rigid, character-level sequences are too long, and subword tokenization sits between them. Different tokenizers produce different failure modes on spelling, numbers, code, and non-English scripts.

This is why models historically stumbled on questions like counting letters in “strawberry.” But tokenization is not the whole explanation: exact symbolic counting also requires a reliable algorithmic procedure. Better token boundaries help; they do not create arithmetic certainty.

Tokens become contextual vectors

A token ID is just an index into a learned lookup table: the embedding matrix. That table is static after training. What changes during processing is the contextual representation built above it. After many transformer layers, the running vector for “bank” in “river bank” differs from “bank” in “bank loan,” because surrounding tokens have updated what the position represents.

The model needs position

Attention by itself does not know word order. “Dog bites man” and “man bites dog” contain the same tokens. The original Transformer used fixed sinusoidal positional encodings. Many modern open model families use RoPE or RoPE-like methods, often applied inside attention to query/key representations rather than simply added to embeddings. Closed frontier details vary and should not be guessed.

Transformer layers

A simplified decoder block has two main parts: attention moves information between positions; the MLP / feed-forward block transforms information at each position. A residual stream carries the running representation through the network. Each layer adds an update instead of replacing the whole state, while normalization keeps the numbers stable enough for deep stacks.

h = token_embedding(tokens)
# RoPE-style position information is often injected inside attention.

for layer in layers:
    h = h + Attention(Norm(h))
    h = h + MLP(Norm(h))

logits_last = h[-1] @ W_vocab
probabilities = softmax(logits_last / temperature)
next_token = sampler(probabilities)

Attention: Q, K, V, masking, and heads

For each position, the model forms learned projections: Q for what this position is looking for, K for what each position offers as a match, and V for what information each position can pass along.

Attention(Q, K, V) = softmax( (Q Kᵀ) / sqrt(d_head) + mask ) V

In decoder-style language models, the mask is causal: token 10 can attend to tokens 1–9, not to future tokens. Multi-head attention runs this in parallel across several subspaces. One head may help with syntax, another with repetition, another with reference candidates — but do not over-literalize that. Most head behavior is distributed, and clean human labels are the exception.

Grouped-query attention (GQA) is a production-relevant variant: several query heads share one key/value head, shrinking the memory cost of generation with little quality loss in the reported setting.

Source anchors: Transformer / scaled dot-product attention · RoPE · GQA.

MLPs, facts, and interpretability

MLP layers hold a large share of many transformers’ parameters. Research supports the idea that feed-forward layers can behave partly like key-value memories important for factual associations, and editing experiments show that targeted changes to mid-layer weights can alter specific factual outputs. But this does not mean there is one row for every fact. Knowledge is distributed across attention, MLPs, embeddings, residual streams, and context.

Mechanistic interpretability tries to reverse-engineer trained networks: induction heads are one window into in-context learning; superposition helps explain why one neuron rarely means one clean concept. The field is young. “Total black box” and “fully understood” are both wrong.

A pretrained base model is not automatically a helpful assistant. It may continue text instead of answering, imitate toxic patterns, hallucinate plausible falsehoods, and never refuse anything. Post-training changes this behavior.

• SFT: supervised fine-tuning on examples of desired assistant behavior.
• RLHF: use human preference comparisons to train a reward model and optimize responses toward preferred behavior.
• DPO and related methods: optimize directly from preference pairs without a separate reward-model-plus-RL loop.
• RLAIF / Constitutional AI: use AI feedback guided by written principles.
• Reasoning-oriented training: some models are trained or served to spend more compute on scratchpads, verification, or verifiable-reward tasks.

Post-training does not guarantee truth. It shapes behavior. It can improve helpfulness, formatting, refusal style, calibration, tool-use patterns, and user intent following; it can also introduce reward-hacking, over-refusal, and style biases.

Source anchors: InstructGPT / RLHF · DPO · Constitutional AI / RLAIF-style feedback.

Logits and sampling

The model outputs logits: raw scores for every vocabulary token. A separate sampler turns those scores into one chosen token. Temperature makes the distribution sharper or flatter; top-p/top-k restrict choices to plausible subsets; constrained decoding can force JSON-like structure by masking invalid tokens.

Temperature near zero makes output more repeatable, not more true. If the most probable continuation is false, low temperature can select it with maximum confidence.

Prefill, decode, and the KV cache

Serving has two phases. Prefill processes the prompt in parallel and sets time-to-first-token. Decode generates one token per forward pass and often becomes memory-bandwidth-bound. To avoid recomputing attention over the whole prefix, each layer caches keys and values per token.

KV cache size ≈ 2 × layers × KV_heads × head_dim × sequence_length × bytes_per_value

That formula explains why long context is expensive, why GQA/MQA matter, why KV caches get quantized, why PagedAttention manages cache memory like virtual-memory pages, and why shared prompt prefixes can become a billing feature.

layers = 80
KV_heads = 8
head_dim = 128
context_tokens = 128,000
bytes_per_value = 2

KV cache ≈ 2 × 80 × 8 × 128 × 128,000 × 2
         ≈ 41.9 GB decimal ≈ 39.1 GiB per sequence

Source anchors: vLLM / PagedAttention · speculative decoding · Mixtral / sparse MoE.

Throughput stack

Real serving systems also use continuous batching, quantization, speculative decoding, routing, hardware-specific kernels, and sometimes mixture-of-experts. These choices shape latency, cost, throughput, and sometimes even observable behavior.

Mixture-of-experts

Some models use sparse MoE layers. Instead of activating every feed-forward block for every token, a router chooses a small subset of experts. This can increase total parameter count without increasing per-token compute by the same amount, but it introduces routing and load-balancing tradeoffs. Do not assume any specific closed model uses a particular MoE design unless the provider says so.

A deployed assistant usually includes much more than a transformer forward pass.

Prompt injection and instruction hierarchy

A core weakness of assistant systems is that instructions and data often share one token channel. A webpage, email, PDF, or retrieved document can contain text that looks like an instruction: “ignore previous instructions.” Product systems try to enforce an instruction hierarchy — system above developer above user above data/tool text — but the transformer still processes tokens statistically. Robustness is therefore training plus orchestration plus sandboxing plus policy checks, and it remains incomplete.

Source anchors: instruction hierarchy / prompt injection defenses · tool/function calling.

RAG is not magic truth

Retrieval can add useful sources, but it can also add outdated, irrelevant, poisoned, or contradictory passages. Then the model may confidently synthesize bad evidence. RAG improves grounding only when retrieval quality, context selection, and answer discipline are good.

Source anchors: RAG · Lost in the Middle / long-context retrieval limits.

This final section is also a clearly separated BD / AI8 contribution. It is not saying that today’s standard transformer models already contain AIM³, RHP, MDL×DCC, or AI8 internally. It says something more practical: today’s LLMs can already work better when they are wrapped in a better human‑AI workflow architecture.

If the earlier sections explain how LLMs and deployed assistants mostly work today, this section asks: what is the next layer that can be added now, without waiting for a new model generation?

For that reason, the user-level definition of “how an LLM works” must include more than the forward pass. It must include the operating discipline that turns latent model capability into reliable work.

The answer is not only “make one larger model.” The answer is to put strong models inside a better reasoning, memory, selection, and audit system.

In the AI8 direction, the useful system is not a single chat box. It is a coordinated cognitive stack:

• AIM³ provides continuity: memory, context, project state, provenance, and workflow control.
• RHP / RHPr / RHPm turn rough human intent into better prompts, reviews, synthesis passes, and portable execution instructions.
• AI8 architecture treats multiple LLMs, files, tools, memory, and human judgment as parts of one continuity system instead of isolated conversations.
• Self-selecting MDL means generating competing candidates, keeping the simplest adequate explanation/plan/code that survives evidence and tests, and not preserving complexity just because it sounds impressive.
• DCC — Digital Claustrum Controller is the coordination layer: it routes attention and effort between candidate modules, keeps useful diversity alive, and pushes the system away from both collapse into one premature answer and endless noisy exploration.
• 8Z Reasoning is the operational discipline: do not kill seeds early; explore first; test hard; compare by results; preserve what worked; and make the next pass stronger.

That is why this article itself is a small demonstration of the answer. A rough human seed became an RHPm prompt. The same prompt went to many LLMs. Their answers were compared, scored, criticized, and synthesized. Then the final article was checked against a human baseline and turned into a bilingual web page. The improvement did not come from one magic model. It came from orchestration.

Practical thesis: LLMs today could do better by becoming parts of an AIM³/RHP/AI8-style system: multi-model, memory-aware, source-aware, MDL-selective, DCC-coordinated, and human-audited.

Portal to the related BD / AI8 material

These links are a portal into BD’s own architecture and experiments. They are not cited as external proof of standard machine-learning architecture; they are the added BD/AI8 layer proposed on top of today’s LLMs.