Wink Pings

Don't Trust the Demo, Trust the Numbers: A Complete AI Evaluation Package

Sergii Makarevych has bundled four long-form eval posts on X: general evaluation, agent evaluation, eval-driven development, and skill evaluation. In the comments, someone added NVIDIA's open-source tool SkillEvaluator, which turns skill evaluation into an automated check gate. Core takeaway: don't trust demos, don't trust a single number, and check your grader first.

Sergii Makarevych published a post on X titled Full package on evals. Four links, four long-form articles, packaging AI evaluation from general to agent to skill. One commenter said they had been looking for resources to learn evals and found this extremely useful.

The four pieces share one trait: every number has a source, either from a runnable tutorial or from a published paper. If you keep only one sentence: don't trust demos, don't trust a single number, and check your grader first.

Part One, General Evaluation: An Eval Has Four Parts

General evaluation is the foundation. An eval is a repeatable test that uses a trustworthy number to tell you whether an AI system did what you asked it to do, and whether a change made it better or worse.

Why you need it: language models are not like ordinary software. Ordinary software either works or crashes. A language model can give a wrong answer fluently, confidently, and in a polished format. Reading one correct answer has almost no predictive power for the next hundred.

The article runs the full process with a fictional bike-shop customer-service assistant: 60 test tickets, triage accuracy 0.700, 95% confidence interval [0.583, 0.817]. The conclusion really worth remembering is about the grader. The model judge and human labels agreed on only 36/60 cases, with kappa 0.15. A 60% agreement rate sounds okay, but if the judge says “pass” to everything, it would also have 57% agreement, because most responses were passing anyway. Kappa subtracts out that luck. 0.15 means this judge is basically guessing.

The judge also has five known biases: position (answers shown first get an advantage), length (longer wins), self-preference (favoring the style of its own model family), treating satisfaction as success, and judge identity (swap the model and the winner changes). Each one has a corresponding fix.

One frequently cited audit result: GAUGE tested with 25 agents and 3,700 conversations. A judge that sees only user-visible conversation is close to a coin flip at distinguishing healthy agents from broken agents (AUC 0.49). Give it tool calls and database state, and the failure risk is halved (AUC 0.73). How strong the judge model is doesn't matter; what evidence you give it does.

Part Two, Agent Evaluation: Look at State, Not the Reply

Agent evaluation is about agents. A normal model only outputs text; an agent also calls tools and changes databases. The focus of evaluation shifts: what it did, not just what it said.

The strongest example: a grader that reads only the final reply passed 58 out of 60 runs. Meanwhile, an agent that does nothing and only replies “processed” passed all 20 runs under the same read-only check. Looking only at the conversation and not the state is equivalent to not testing at all.

Agent evaluation needs two numbers. pass@k is capability: at least one success in k tries. pass^k is reliability: success every time in k tries. Customers experience the latter. The article reports pass@3 0.85 and pass^3 0.75, but broken down by task category: policy refusal 1.00/1.00, normal operations 0.82/0.73, escalation path 0.50/0.00. The average is real, but useless.

Failures should be attributed to specific steps, not recorded as a single score. In the article, all 12 failure cases could be traced back to the first erroneous tool call: 6 were caused by the task itself being written incorrectly, and 6 by the agent replying to the customer before issuing the refund. One category explained all the real failures, and the direction for fixing them was clear.

Part Three, Eval-Driven Development: Write the Test First, Then Build the System

Eval-driven development is about process. First define “good” as a testable eval, then start choosing models, configuring retrieval, and deciding architecture. After that, every choice turns from an argument into a test.

A few numbers you can copy immediately:

  • Run the same case 3 times. If all 3 agree, it is probably stable; if the 3 disagree, the case is flaky, so give it 5 to 10 repetitions.
  • In a 20-case test set, each case is worth 5 percentage points. With 100 cases, each is worth 1 point. You cannot reliably detect a 5-point regression with 60 cases; the detection floor is 12 points.
  • Let code decide what code can decide; the judge only looks at things that require reading to score: tone, policy boundaries, and clarity.
  • Keep a hidden set out of tuning to prevent overfitting.

Part Three has a good metaphor: a single score is a “point,” and a dashboard is the “line” connecting those points. Only by looking at quality, cost, and latency lines together can you spot bad trade-offs and slow degradation.

Part Four, Skill Evaluation: Most Shared Skills Are Useless

Skill evaluation is the most counterintuitive. A skill is just a piece of text loaded into the agent's context; it does not change the model itself. The author's testing framework: break the skill down into core claims, pair each claim with a testable hypothesis, then measure two numbers. Adherence: did the agent follow it? Effect: did following it help? Low adherence is a problem with the text; high adherence plus flat effect is a problem with the idea. Looking only at outcomes cannot distinguish these two failures.

The data is not pretty:

  • Of 49 public software-engineering skills, 39 had zero gain, the average gain was 1.2 points, and token overhead rose by up to 451%.
  • Of 31,132 marketplace skills, 26.1% carried at least one vulnerability pattern.
  • One development machine had 89 skills installed, and 81 had never been invoked.
  • The model decides on its own whether to load a skill, and trigger rates in multiple tests were only around half; the longer the list, the lower the chance of selecting the right one, dropping from 29.6% with 5 skills to 3.3% with 100.

The conclusion is direct: installing a shared skill is like merging a stranger's PR, and this PR has no tests. Either test it yourself first or don't install it.

Bonus Tool: NVIDIA SkillEvaluator

Christopher K added a project in the comments that turns the methodology from Part Four into runnable code: NVIDIA's SkillEvaluator, Apache 2.0, experimental stage.

It has three tiers. Tier 1 static checks: more than 500 rules checking quality, schema, PII, license, and dependency vulnerabilities, with no LLM key required. One particularly useful check is harness-check: it verifies whether your agent will actually load the skill rather than install it and let it gather dust. Tier 2 checks for duplicates and overlap. Tier 3 runs a real agent and compares results with and without the skill.

Integration is a ready-made CI template. Run skillevaluator ci init in a GitHub or GitLab repository, and it automatically generates a workflow; after that, every skill change automatically runs the checks. The barrier is much lower than the full evaluation described in Part Four, and it fills exactly the gap that “nobody tests shared skills.”

What Is Really Valuable Here

The same piece of advice appears repeatedly across the four articles: read one hundred real outputs before writing a grader. Run an agent that does nothing before trusting any number. Check the agreement rate between the judge and humans before letting the judge decide whether to ship.

SkillEvaluator already handles the parts tools can automate. Reading the data is something machines cannot replace. This package is worth bookmarking not because it is complete, but because it turns evaluation from “it feels fine” into “the numbers say it is okay, and I know why the numbers say it is okay.”

发布时间: 2026-11-20 00:42