How an AI agent works, in plain English: harness, context, and memory

What an AI agent is, in plain English

Picture this: you’ve hired a genius chef. Thousands of recipes in their head, can cook anything, works around the clock. One catch: they’re standing blindfolded in the middle of an empty room. No stove, no knife, no fridge. And the second you step out the door, they forget absolutely everything.

That chef is the model (Claude, GPT, and friends). The kitchen around them - the stove, the knives, the recipes taped to the wall, the rule “don’t stick your hand in the fire” - that’s the harness A harness (from the word for a horse's tack - the straps and rigging around it) is everything wrapped around the model: the tools you hand it, the format it sees results in, the rules, and the memory. The model stays the same; everyone's harness is different. . An AI agent = chef plus kitchen. Not one or the other - both.

Why a vibecoder should understand how the agent thinks

You’re a vibecoder. You don’t have to write code by hand. But the moment it clicks for you how an AI agent thinks, a few things change:

• you get frustrated less often and fix the cause, not the symptom;
• you save money - you pay for “thinking” in tokens, more on that soon;
• you get results on the first or second pass, not the tenth;
• you frame the task so the agent doesn’t “drift” halfway through.

“The AI built me a broken site” versus “the AI built me a working site” - most of the time the difference comes down to understanding three things: context, tokens, memory. Let’s start there.

The three pillars: context, tokens, and memory

Context - the AI agent’s desk

The agent doesn’t have human-style memory. What it has is a context window Context (the context window) is all the text the model can 'see' right now: your messages, files, command output, instructions. Anything that didn't fit or got pushed out simply doesn't exist for the model. - think of it as a desk. Only so many “scraps of paper” fit on that desk: the task, chunks of code, command output, instructions.

When the right stuff is on the desk, the agent is brilliant. Bury the desk in clutter (old messages, walls of logs, five versions of the same file) and the chef literally has nowhere to set down the knife. That’s when it starts losing the thread, mixing up details, and hallucinating.

Tokens - the currency you pay for the desk

Every “scrap of paper” on the desk costs money. For the model, text gets sliced into tokens A token is a chunk of text, roughly 3-4 characters or ¾ of a word. The model both reads and writes in tokens. You pay for input tokens (what you gave it) and output tokens (what it generated). . The more text you shove in and the longer the answers you ask for, the more tokens - which means more expensive and slower.

That’s why seasoned vibecoders don’t paste the whole project into the chat. They point the agent at one file or the specific snippet - or ask it to go find what it needs itself. And they pay pennies instead of a fortune.

Memory: short-term and long-term

This is where beginners get tripped up. The agent has two kinds of memory, and they’re easy to confuse:

• Short-term (working) memory is the context itself - that same desk. It lasts exactly one session. Close the chat or hit a “compact” and it’s all wiped.
• Long-term memory is files on disk. Notes, MEMORY.md, project rules. The model holds nothing in its head between runs: every time, it re-reads the files from scratch.

Compact: when to clean up the context

When the desk overflows, a compact A compact is the compression of the conversation history: the agent folds the old stuff into a short summary to free up space. Useful, but details get lost in the process. kicks in: the agent folds the old conversation into a short summary. Space frees up - but details vanish. It’s like sweeping everything off the desk into a “misc” folder: you’ve got space, but now finding what you need is a pain.

The whole trick is when you do it:

A good harness vs. a bad one

The harness is the kitchen. Here’s how to tell a comfortable one from a nightmare at a glance. This’ll come in handy once you get to hooking up tools - MCP, skills, and the rest (that’s in the chapters ahead).

AI memory: what helps and what hurts

A real-world example: why the agent broke a working landing page

You ask: “build me a coffee-shop landing page.” The agent fires up, generates 12 files. Along the way you tweak it 40 times: “button further left,” “different font,” “no, put it back the way it was.” Then an hour in, it suddenly breaks the thing that worked perfectly at the start. And then argues that “it was always like that.”

Here’s what actually happened:

1. The desk got buried under 40 edits and a dozen file versions.
2. A compact hit - and the agent forgot what the working version looked like.
3. The rule “the font is Fraunces” lived only in the chat and got wiped along with the history.

And here’s how someone who understands the kitchen would phrase the same request:

Build a coffee-shop landing page. Before you write any code:

Write the key requirements into a file called DECISIONS.md (font
Fraunces, teal palette, single page, order button up top). Check
against it as you go.
Work one block at a time: hero, menu, contacts. After each block,
show me the result and wait for my “ok.”
Don’t rewrite already-approved blocks without my permission.
If the history overflows, do a compact ONLY after a finished
block, never in the middle.

Agent acting dumb? A mini-diagnosis

Before you lose it, run through the checklist (this is basically the agent-introspection-debugging skill):

1. Is the desk overloaded? Long conversation, agent confused - clean up or start fresh.
2. Did you lose context? Maybe a compact wiped something important - repeat the key requirements.
3. Is the rule actually in a file? Saying it once in the chat doesn’t count.
4. Is a tool failing silently? If the agent is going in circles, maybe a command is breaking and it can’t see the errors.
5. Is the task too big? Slice it into chunks of about fifteen minutes of work.

Common beginner mistakes in vibecoding

• Dumping the whole project into the chat. Expensive, and it clogs the desk. Hand over the chunk you need.
• Keeping rules only in the conversation. After a restart they’re gone. Move them into files.
• Pushing one giant task. The agent chokes. Slice it up.
• Cleaning context in the middle of debugging. You’ll lose exactly what you were chasing.
• Ignoring tokens. That “endless” chat eventually shows up with a bill that makes your stomach drop.
• Arguing with the agent instead of restarting. Sometimes it’s cheaper to start a session with a clean desk.