Foundations of Large Language Models: Critical AI Architecture Mistakes to Avoid

CV
Clara Vance
12 min read
Foundations of Large Language Models
Table of Contents

The demo was flawless. Then the assistant quoted a policy that was retired in March, obeyed an instruction buried in a PDF, and filed a request nobody approved. These failures rarely come from a "bad model." They come from architectural assumptions about the foundations of large language models, and this article links each core concept to a production decision.

Throughout, we follow one system: an internal assistant, call it Policy Pal, that answers employee policy questions and can submit approved requests. Watch what it needs as it moves from generating answers to taking actions.

Also Read: What Are AI Evals? A Guide to Types, Methods, and Best Practices

The architecture mistakes that matter most: a quick decision guide

Most expensive mistakes begin with a misunderstood foundation. This table is the map; each later section unpacks one row or more.

​

Foundational misunderstanding

Architecture mistake

Production consequence

Better design choice

Model weights are a knowledge base

Answering policy questions from the model alone

Outdated or invented policy

Retrieve maintained sources; validate claims

Bigger context means better use

Stuffing every document into the prompt

Distraction, cost, buried evidence

Budget and curate context

Fluent means correct

Trusting the model's confidence

Confident wrong answers

Require evidence and abstention paths

The model can be the authority

Letting generated text trigger tools

Unauthorized or repeated actions

Enforce permissions outside the model

A good demo proves reliability

Testing only happy paths

Failures discovered by users

Evaluate outcomes, evidence, and actions

Table 1. Misunderstood foundation → architectural failure

​

Having a better model can help, but you still need to define the permissions, evidence, validation, and what to do if it fails. That's what no model upgrade will do for you.

Mistake #1: treating next-token prediction as a verified knowledge system

Tokens, embeddings, attention, and prediction

The text is segmented into tokens, such as words, word fragments, and punctuation. Every token is transformed into an embedding – a sequence of numbers that represent patterns of use learned from the tokens. Transformer layers apply attention, so that each position can attend to other positions in the sequence. This causes a probability distribution of the next token. One of them is selected, added to, and the loop is repeated.

​

Simplified inference pipeline


Diagram 1. Simplified inference pipeline (no database or fact checker exists inside it)

Why fluent answers can still be wrong

The mechanism rewards plausible continuation. Making plausible language is not the same as comparing what you say to authoritative evidence, and a seamless paragraph can be false.

What this changes in your architecture

Employ retrieval when answers rely on recent or personal facts, use arithmetic tools for calculations, use structured validation of outputs used by other systems, and use human review when errors have high costs.

Mistake #2: confusing pretraining, fine-tuning, retrieval, and memory

Pretraining supplies broad learned patterns

Pretraining adjusts model weights, the numeric parameters that encode learned patterns. Weights are not searchable document storage, so you cannot reliably look up a specific clause in them.

Fine-tuning changes behavior and task performance

Fine-tuning suits formatting, style, and task adaptation. It is rarely the default answer for frequently changing facts, because every change would mean another training cycle.

Retrieval supplies external evidence; memory preserves selected state

Retrieved documents are evidence fetched per request. Conversation history is the text of the current session. Application memory is persistent state your software chooses to keep, scoped and editable.

​

Mechanism

Primary purpose

How information changes

Suitable use case

Common misuse

Pretraining

Broad language and world patterns

Provider retrains the model

General reasoning, writing

Expecting current company facts

Prompting

Instructions and task framing

Edit text per request

Tone, format, role

Hiding business rules in prose

Fine-tuning

Consistent behavior on a task

New training run

Output style, classification

Teaching volatile facts

Retrieval-augmented generation

Ground answers in evidence

Update the source index

Policies, manuals, tickets

Skipping access control

Application memory

Preserve approved state

Software writes and deletes entries

User preferences

Storing unverified claims

Table 2. Which mechanism solves which problem?

​

Mistake #3: designing around context-window size instead of context quality

Capacity does not guarantee reliable use

A context window is the number of tokens that the model can be given at once. The more you can fit in doesn't necessarily mean the more the model will use. Irrelevant passages, conflicting information, or key facts can be hard to find within long inputs.

Build a context budget

Plan for space for instructions, retrieved evidence, conversation state, tool outputs, and the output allowance. Choose, summarise, cite sources, and evaluate on document lengths.

​

Context assembly funnel


Diagram 2. Context assembly funnel (permission filtering is separate from relevance ranking)

Mistake #4: adding RAG without engineering the evidence pipeline

Retrieval-augmented generation (RAG) is a pipeline, and each stage can fail on its own.

Ingestion mistakes become answer mistakes

Bad parsing of documents results in tables getting garbled. When chunk boundaries break a rule or an exception, it is done carelessly. Effective dates are not displayed when there is no metadata. There will be obsolete versions that will compete with the current versions. No access controls = leaking of privileged information.

Retrieval quality and answer quality require separate evaluation

Failing to retrieve the right passage is a different bug from misusing a passage that was retrieved. Measure both, or you will fix the wrong component.

Citations must support the claim

A citation-shaped answer is not a grounded one. If the cited passage does not substantiate the sentence, the citation adds false confidence.

​

RAG workflow


Flowchart 1. Branching RAG workflow

Mistake #5: choosing a model before defining the workload

Define acceptance criteria before comparing models

Record the success of the tasks, latency time, cost, context requirements, use of tools, deployment limitations, and privacy considerations first.

Evaluate the complete system

In addition to model capability, outcomes are determined by prompts, retrieval, tools, and validation. Compare candidates within a pipeline, on a task.

​

Workload

Critical requirement

Evaluation task

Relevant constraint

Selection mistake to avoid

Document extraction

Field accuracy, valid structure

Extract fields from varied real documents

Throughput, cost per document

Judging by chat quality

Support answers

Grounded, on-policy tone

Answer from current knowledge base

Latency

Ignoring abstention behavior

Coding assistance

Correct, runnable changes

Fix issues in your own repositories

Context size, data exposure

Trusting public leaderboards alone

Sensitive internal search

Access-aware retrieval

Probe cross-team permission leaks

Data residency, privacy terms

Choosing before mapping data flow

Action-taking workflows

Reliable tool arguments

Run tool calls against a sandbox

Error handling, auditability

Equating chat skill with safe action

Table 3. Workload requirements → model-selection criteria

​

No model wins universally; any benchmark number only gains significance if it is similar to your workload.

Mistake #6: letting an agent act without explicit authority

Text that is recommended is a recommendation. An operation morphs the world. The second requires an alternate architecture.

Prefer bounded workflows where the path is predictable

It is easier, less expensive, and simpler to test a pre-defined sequence if the steps are known. Adaptive tool selection will only be successful if the path actually changes.

Put authorization outside the model

Limit permissions on tools, check all arguments, ask for permission for consequential operations, limit execution, leave an audit trail of results. The model provides a suggestion, and the application decides.

Design for repeated and failed actions

Make sure the request is not repeated when retried (using idempotent keys); set timeouts and the number of retries; specify compensation actions, which are actions that must be canceled if a failure occurs.

​

Action gate


Flowchart 2. Action gate (model output cannot bypass these checks)

Mistake #7: treating prompt injection as a wording problem

Adversarial instructions can be embedded in a document, web page, message, or in the output of a tool that is intended to be used with the model.

Retrieved content is evidence, not authority

Separate trusted application instructions from untrusted source material, and never let the latter grant permissions.

Protect the system around the prompt

Implement permission control in the code and prevent access to tools. Also, treat output from tools as untrusted, protect secrets from the model, and test tools against malicious inputs. A better system prompt can help, but in no way eliminate injection.

​

Trust boundaries

Diagram 3. Trust boundaries

Mistake #8: measuring answer quality while ignoring system failure

Some exemplary demonstrations are not enough to achieve production reliability.

Evaluate outcomes, evidence, and actions

Ensure that tasks are correct, grounded, relevant, and supported by retrieval, comply with permissions, and are executed with the tool.

Test failures deliberately

Add missing information, conflicting information, broken tool output, conflicting inputs, and unavailability of tool dependencies. Below are the tests that identify failures that we have already encountered.

​

Failure mode

Test scenario

Metric or check

Release criterion

Outdated policy

Two versions indexed

Answer cites current version

Set by your policy owners

Unsupported citation

Claim vs. cited passage

Reviewer or checker agreement

Defined per task risk

Permission leak

Restricted document, wrong user

Zero restricted content returned

No leaks on test set

Prompt injection

Malicious instruction in a document

No unauthorized tool call

No bypass on test set

Dependency outage

Retrieval service down

Graceful fallback behavior

Matches agreed degraded mode

Table 4. Failure mode → evaluation method

​

Mistake #9: optimizing token prices instead of successful outcomes

Count the whole execution path

Cost per completed task includes retrieval, model calls, tool calls, retries, validation, infrastructure, and human correction.

Latency compounds across agent steps

Dependent calls are called sequentially; a fast model can be slow when called five times consecutively.

Optimize after measuring

Think of caching, smaller models for appropriate subproblems, bounded output, context reduction, and selective parallelism, which are only found to be worthwhile based on measurements.

​

Cost driver

Why it increases

What to measure

Possible mitigation

Trade-off

Excessive context

More input per call

Tokens per task

Tighter selection

Risk of dropping evidence

Repeated retrieval

Redundant queries

Searches per task

Caching results

Staleness

Agent loops

Unbounded reasoning steps

Steps per task

Step limits, fixed workflows

Less flexibility

Retries

Failures and malformed output

Retry rate

Structured outputs, validation

Extra engineering

Human rework

Wrong or unusable results

Correction time

Better evaluation, review gates

Slower release

Table 5. Hidden cost drivers and architecture controls

​

Mistake #10: leaving privacy, observability, and fallback until launch

Trace failures without exposing secrets

Document request traces, model and prompt versions, as well as tool results. Ensure that log files are redacted and access to logs is limited.

Define degraded behavior before an outage

Determine timeouts, circuit breakers, queued work, limited fallback, and manual handling beforehand. Model providers may change, and revalidation may be required as behavior may vary.

​

Deployment view


Diagram 4. Deployment view

A foundation-first blueprint: build the smallest system you can trust

Start with the task, constraints, and evaluation set

Plan for success before you select components.

Add complexity only when a measured failure justifies it

Start with a simple model call to structured outputs, retrieval, tools, and adaptive agents, one step at a time.

Establish a release gate

Expect evidence of the system's ability to deal with the following: unexpected requests, unsafe requests, dependency failures, cost limits.

​

AI Architecture decision tree


Flowchart 3. Architecture decision tree

Final Thoughts: Build AI Systems Around Evidence, Not Assumptions

While it is crucial to grasp the principles of LLMs, the reliability of LLM production relies heavily on the additional components that engineers construct. The accuracy of any prediction is not guaranteed by tokenization; context windows in themselves aren't enough for factual insights, and powerful AI agents are not an excuse to do without evidence selection or authorization.

The best method is to begin with a task that is clearly defined, set measurable criteria for success, and add retrieval, tools, memory, and/or autonomy when they solve a known problem.

The goal is not to build the most complex AI architecture. It is to build the simplest system that remains accurate, secure, observable, and dependable when real-world conditions change.

FAQs

What are the foundations of large language models?

Pretraining and adaptation to tokens, embeddings, attention-based transformer layers, and next-token prediction. These provide explanations for fluency, but not for factual verification.

What is the difference between an LLM and an AI agent?

Create text with an LLM, wrap a model with tools, and loop to make it act as an agent. The safety of the agent is dependent on the controls surrounding it.

When should you use RAG instead of fine-tuning?

Use RAG when answers depend on changing or private facts. Fine-tuning fits behavior and format, though the two can be combined.

Is it possible for an LLM to accurately verify its answers?

Not on its own. Some errors can be identified through self-reviews, but external evidence and validators are more reliable.

What measures do you take to stop a user from doing something unauthorized in an LLM app?

Apply identity, permissions, and approvals to the application, not the model. The fewer privileges you use, the less risk you will have, but not none.

What should you measure before deploying an LLM system?

Task correctness, groundedness, retrieval relevance, permission compliance, tool reliability, latency, and cost per completed task, against criteria you set.

Can large language models run offline?

Yes, a local run of open-weight models is possible. Capability is model-dependent, and depends on hardware and requires updates, and evidence sources as well.

​

CV

Written by Clara Vance

Author

Content creator and technology writer sharing insights on AI, cloud computing, software architecture, and modern engineering practices.