NVIDIA Open-Sources SkillEvaluator: It Can Test Agent Skills, but the README Doesn't Provide Results
Writing an agent skill isn't hard; what's hard is knowing whether it changes the agent's behavior. NVIDIA's SkillEvaluator uses three tiers to check quality, overlap, and actual impact, but the README doesn't publish its own results.
NVIDIA has open-sourced SkillEvaluator under Apache-2.0, with a very specific goal: check the quality of agent skills and measure whether they actually affect task outcomes. An agent skill is a folder containing instructions and supporting files, organized according to the Agent Skills specification. Writing a skill that can load isn't hard; what's hard is proving that it changes the agent's behavior. SkillEvaluator breaks this down into three tiers and is also part of the NVIDIA Verified Skills pipeline; the README includes the paper arXiv 2608.20614.
Tier 1 performs validation: it checks whether the skill is safe, correctly formatted, and compatible with the harness. The default passing threshold is 70. The text-analyzer example in the README scores 86.2/100, B, exit code 0. This tier does not require an API key and does not depend on a GPU or Docker. But passing static quality does not mean publication evidence is complete; the README explicitly says publication evidence is still INCOMPLETE, and complete security coverage still requires external scanners.
Tier 2 performs deduplication: it uses embeddings and chat to determine whether guidance content is duplicated or overlapping. Adding --catalog ./skill-catalog.json lets you compare overlap across different skills. This problem will become increasingly practical; once skill libraries get large, skills conflicting with each other is more common than a single skill being written incorrectly. One commenter said the overlap flag may be an underrated feature.
Tier 3 performs live evaluation: it runs once with and once without the skill, compares task results, and measures the skill's impact on outcomes. It requires provider access, a supported agent, and a running sandbox, Docker by default. Tier 3 does not block validate by default unless you add --block-on-agent-eval. The support level is marked Experimental, community-supported, with no SLA.
There are three ways to integrate: CLI, CI, and connecting directly to an agent. The lightest check is a single command:
pip install skillevaluator
skillevaluator validate ./my-skill --tiers 1
For CI, you can use skillevaluator ci init. Claude Code, Codex, or other agents that support skills can use npx skills add NVIDIA/SkillEvaluator.
Community reactions are more direct than the README. One commenter asked what keyless Tier 1 can catch on its own; without results, it's hard to judge how meaningful these gates are. Another asked whether the with/without scores are based on task pass rate or also check whether the skill was actually loaded; a skill the agent never activated looks like zero effect. Another commenter argued that with/without is the only reliable method: if it loads correctly but doesn't change behavior, it's just a prompt file.
This post itself is labeled README read, not run. The tool's direction is clear, and the tiering is reasonable, but the README doesn't include its own evaluation results. For teams that want to put it into CI, the feature table is only the first half; the results page is the second half. Source: https://github.com/NVIDIA/SkillEvaluator
发布时间: 2026-11-16 08:26