# Skill: Read AI benchmark charts critically and run honest personal model comparisons

> Lets an AI coding/design agent assess model benchmark claims without being misled by marketing, and design a personal blind-test evaluation; applies whenever a new model is announced, a chart is cited as evidence, or a model-selection decision needs grounding in real data.

Source: sodigi·learn — models/kak-chitat-benchmarki · https://sodigi.io/learn/models/kak-chitat-benchmarki

## When to use
- A new model is released with a slide showing it beats competitors on every chart.
- A user asks "is model X actually better than model Y?"
- Recommending a model for a production workflow that will run at scale.
- Explaining to a stakeholder why the "top leaderboard model" is not necessarily the right pick.

## Core rules
- A benchmark is an argument, not proof; a score is one number averaged over tasks that are not yours.
- A gap of a couple of percentage points (e.g., 87 vs. 85) is almost always noise, not a meaningful win.
- Goodhart's Law applies: when a score becomes a target, labs train specifically for that exam, inflating the number without improving real usefulness.
- Contamination (data leakage) means test questions often appear in training data; a high score may mean memorization, not understanding.
- Cherry-picking includes: comparing against an outdated competitor, truncating the chart axis to exaggerate differences, averaging in ways that hide weaknesses.
- Community leaderboards can also be gamed (publishing only best runs from many variants).
- The only honest test for your use case is your own tasks run with identical prompts across all candidates.
- Judge answers blind -- hide model names before reading results to avoid brand bias.

## Procedure
1. When a benchmark slide appears, immediately ask: which tasks, what date, compared against which version of competitors?
2. Check whether the gap is large (clear leap) or small (likely noise); dismiss differences under ~5 points unless independently replicated.
3. Build a personal task set: collect 5-10 real tasks from your own work.
4. Write down ahead of time what a good answer looks like for each task -- this is your reference.
5. Send the exact same prompt (byte-for-byte identical) to every model being compared. Do not rephrase.
6. Run each task 2-3 times per model to check consistency, not just peak performance.
7. Copy all answers into a plain file and label them A, B, C. Judge with model names hidden.
8. After scoring, reveal which label was which model.

## Ready-to-use prompt
```
I am comparing AI models for the following task. Respond fully and do not mention your own name or version.

Task: [describe your real task in 1-3 sentences]
Success criteria: [what a correct, complete answer looks like]
Constraints: [format, length, language, stack, etc.]

Deliver the result only. No meta-commentary.
```

## Pitfalls
- Treating a 2-point benchmark gap as a decisive win -- it is usually within noise.
- Comparing the new model against an outdated version of its competitor instead of the current one.
- Testing on toy examples ("how many r's in strawberry") instead of real work tasks.
- Changing the wording between model runs -- even small prompt differences shift outputs enough to invalidate the comparison.
- Judging while knowing the brand -- the model name in your head creates bias; always blind the labels first.
- Assuming community "human preference" leaderboards are fully neutral -- some platforms allow selective run submission.