Eval-first: checks before code when you build with AI

What eval-first means in plain words

Imagine ordering a cake from a wildly fast but slightly unhinged pastry chef. Say “make me a cake” and he will. Some cake. Maybe with mushrooms in it. And he’ll be sincerely proud of himself.

Now phrase it differently: “chocolate cake, eight slices, no nuts - a guest is allergic, and make the candle stand up instead of falling over.” See the difference? The chef now has a checklist he can grade himself against. And you can check too - without taking a bite out of the whole cake.

That’s exactly what eval-first Eval-first (from evaluation) is an approach where you first describe your success criteria and how you'll check them, and only then start the work. The check comes before the code. is: first work out how you’ll know it came out well, and only then fire up the AI agent. Sounds tedious. In reality it’s the whole secret behind why some people get a working product out of AI while others get beautiful garbage.

Why a vibecoder needs eval-first

You’re a vibecoder. You don’t write the code by hand - you hand tasks to an agent and accept the result. And right there, at “accept the result,” eval-first saves you both your nerves and your money:

• no more staring at 12 files thinking “well, looks okay?” - you have a precise list of what should work;
• the agent checks itself before handing the result over - fewer rounds of “no, not like that, redo it”;
• you catch the moment a new feature broke an old one (which happens more often than you’d like);
• you can honestly say “done” - because “done” now has a definition, not just a feeling.

The difference between “the AI did something” and “the AI did the thing I needed” usually comes down to one question asked before the start: “how will we know it worked?”

How the eval-first loop works

In the ECC skills this is called the eval-first loop. There are only four steps, and they’re simple:

1. Describe the checks - what should start working (capability) and what should not break (regression).
2. Take a “before” reading - run the checks against the current version and note what fails right now.
3. Let the agent work - now it writes code toward a clear goal instead of guessing.
4. Run the checks again - compare “before” and “after.” Did it get better? Did anything break?

A check is essentially tests for ordinary code, only adapted for working with AI. In ECC circles they put it bluntly: evals are “unit tests for development with AI.”

Capability vs regression: new against old

They’re easy to mix up - so let’s lay them out clearly.

• Capability A capability eval checks whether the agent can do something new that wasn't there before: a new button, a new function, a new feature. (can it do the new thing). You check whether the thing you started all this for actually showed up. For example: “the user signs up by email,” “the order form gets submitted.”
• Regression A regression eval checks that a new change didn't break things that were already working. A regression is a step backward, when something that worked stops working. (did it break the old thing). You check that yesterday’s stuff still works today. For example: “the old sign-in button is still there,” “the other pages still open.”

The pass@k metric: how to measure AI reliability

AI is a probabilistic beast: the exact same request comes out perfect one time and off the mark the next. So you don’t measure reliability as “yes/no” but as the share of wins across several tries. That’s pass@k pass@k is a reliability metric: it worked at least once in k tries. pass@1 means first try. pass@3 means at least once out of three. pass^3 means all three tries in a row succeed. .

• pass@1 - it worked on the first try.
• pass@3 - it worked at least once out of three tries.
• pass^3 - all three tries in a row succeed (a higher bar, for critical stuff).

The ECC skills give these targets: for new features (capability) - pass@3 of 90% and up, and for critical regressions - pass^3 = 100%. Anything that worked must work always. No exceptions.

Slice the task into 15-minute chunks

Evals only work if the task can actually be checked. A giant “build me a marketplace” can’t be checked in one go - that’s a hundred risks rolled into one. So the agentic-engineering skill has a 15-minute-chunk rule: slice the work into units where each one:

• can be checked on its own, separately from everything else;
• carries one main risk (not “here’s the design, the payments, and the database all at once”);
• has a clear definition of done - you can tell when it’s finished.

Eval-first on a live example: a booking form

You ask the agent: “add a table-booking form to the coffee shop site.” Without eval-first the agent cheerfully slaps a form together, you look - there are fields, there’s a button, “yeah, okay.” And a day later it turns out the form sends empty requests, doesn’t validate the phone number, and on top of that broke the menu next to it.

So how would a vibecoder who gets it do this? Describe the checks first - and only then let it build:

Add a table-booking form to the coffee shop site. But first, BEFORE the code, write out the checks we’ll use to know it’s done.
Checks for the new stuff (what should start working):

The form accepts a name, a phone number, and a number of guests.
An empty form can’t be submitted - a clear error shows up.
A phone number with no digits is rejected.
After a successful submit, a “request received” message is visible.
Checks for the old stuff (what should not break):

The menu on the page still opens.
The “call us” button in the header works.
The other pages still open like before.
Then work like this: build the form, then walk through all the checks yourself and write PASS or FAIL next to each one. If there’s even one FAIL - fix it and check again. Don’t hand me a result with any FAIL in it.

Who judges the result: code, AI, or a human

You can hand the check to different “judges” - in the eval-harness skill there are three:

• The code judge (the most reliable). A precise check with no arguments: tests passed / the build compiled / the needed string is in place. If you can check it with code, check it with code.
• The AI judge (an LLM as the grader). For when “good” can’t be measured with a ruler: “is the copy easy to read,” “is the error clear.” You ask another model to rate it on a 1-to-5 scale.
• The human judge (you). For the gray areas and for anything to do with security. The ECC rule is hard: security checks never go on full autopilot - the last word belongs to a human.

What a PASS/FAIL eval report looks like

To make it crystal clear - here’s a short report on the checks in eval-harness style:

Once you’ve finished the form, give me a short report in this format:
Checks for the new stuff:
accepts the fields     - PASS or FAIL
can’t submit empty     - PASS or FAIL
phone is validated     - PASS or FAIL
success message        - PASS or FAIL
Checks for the old stuff:
menu opens             - PASS or FAIL
call-us button         - PASS or FAIL
other pages            - PASS or FAIL
At the end: how many out of how many passed. And the status: ready to ship or needs fixes.

Common mistakes in AI checks

• Checking only the “happy path.” You confirmed the form works when everything was entered correctly. But the empty fields? The garbled phone number? Real people will type in anything, including emojis in the “age” field.
• Forgetting about regressions. You’re celebrating the new feature and don’t notice the old stuff falling apart. Always keep a “what should not break” list within reach.
• Describing the checks AFTER the code. That way you unconsciously bend the criteria to fit what already came out. Checks are designed strictly before.
• Pushing one giant task. There’s no way to check it as a whole. Slice it into ~15-minute chunks with a clear definition of done.
• “Done” with no definition. “Make it pretty” - the agent guesses, and then you’re unhappy. “Done” = a concrete list of checkmarks.
• Putting security on autopilot. Logins, payments, access - a human always does the final check.
• Chasing percentages and forgetting the cost. You can polish pass@k to a shine and not notice that every try eats a pile of tokens and time. Watch both reliability and spend.