He built a TS-like language for codemode, but models still can't write it fluently

Mario Zechner posted a screenshot of a code editor on X. At the top is “Design pico codemod,” and below it are 1763 tool calls running, with 1 failure. The failed one starts with agent eval6 pico. This doesn't look like an ordinary demo; it looks more like an evaluation tool built with spare agent.

The day before, Mario said he wanted to build a language for codemode called pico. When someone asked what it was written in, he answered C. When someone asked about the syntax, he replied sharply: who cares. Netizens joked that he originally wanted to beat Armin, but ended up joining the other side. There was also a side note about the name pico: a netizen said that in Slovak it is a nickname for methamphetamine.

codemode can be understood as: the model writes a script, and the runtime executes it; the script can invoke bash, write files, and spawn subagents. The model doesn't have to call tools one by one. Mario wants pico to become this scripting language. QuickJS is a small JS engine, and djs/TS is Armin's approach. So eval6 compares the three together.

The evaluation tool itself is a p2 codemode script, running in a QuickJS sandbox with an async body. The gist comment says “Not pi-durable.” In the tweet, Mario called it “exceptionally cursed.” Netizen Liu Chao summarized: using code mode to write an eval harness, then spawning agents to test code mode, gives exactly the right amount of cursedness.

The harness logic isn't complicated. It crosses three dimensions: task, model, and language. For each combination, a one-shot subagent writes a script. The subagent has no tools and only returns structured output. The script is written to a file, and then the runner runs a fixed fixture, checking the output, return value, and tool calls. At most 3 attempts. It retries only on compile/parse failures or runtime errors, feeding the error back. If the script runs but the answer is wrong, it doesn't retry. 16 workers run concurrently. Results are appended to results.jsonl.

The table title is Model eval (eval 6), comparing pico, djs, and QuickJS, with 120 runs each. It lists first compile/parse, first-attempt correctness, correctness within three attempts, ran to completion but wrong answer, runtime errors, number of tool calls where the script call failed, total character length, and number of lines described in the prompt. Below, it breaks down first-attempt correctness and correctness within three attempts by model; the models include Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Sol, DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3.

Mario's conclusion: pico is close to TS, but models still struggle. He added that a codemode language with type checking can improve correctness, and the current cost is more first-attempt failures. This problem can be solved by changing the language and changing the language description given to the agent. In other words, when models write incorrect code, part of it is a language design problem and part of it is a documentation problem.

The screenshot lists the test suite before committing: all C tests and 33 golden cases, 136 TypeScript tests each with JIT off, JIT on, and forced mode, all 60 reference solutions in the three languages, plus site tests. The commit consists of the 8th round of eval, the QuickJS and djs evaluation tools, and a 163-line description. The site agent is updating the language reference and the Model evals page. The owner left a message: “keep improving things until pico wins the eval.”

The goal isn't just to win the evaluation. Mario said he wants a durable execution VM with snapshots as small as possible, ultimately moving away from replay-based durable execution. This goal is very specific. pico hasn't won yet, and Mario says he will keep climbing the mountain until it succeeds. Looking at the table, pico is close to TS but still some distance from models writing it fluently. Whether to change the language first or change the documentation first is a choice he has to make himself.