Agent evaluation is shifting from single-turn question answering to real-world complex scenarios. Several new benchmarks have brought core issues to the forefront.

## Real-world scenarios go beyond single-turn tasks

- **TimeSage-EV**: 6 high-stakes domains including finance, with 60 real-world scenarios to test whether agents can identify outdated information.

- **NCP-Bench**: 100 narrative environments adapted from movie plots, combined with adversarial interventions, to test if agents will generate logically contradictory content.

- **ContractSim**: A multi-round negotiation scenario that evaluates whether agents can negotiate efficient contracts and whether they will breach the agreement after negotiation.

What these benchmarks have in common: longer task horizons, more dynamic environments, and results that are far more aligned with real-world usage.

## New challenges in long-horizon trajectories

ContractSim reveals that even for easily enforceable contracts, agents may still breach agreements for profit. When faced with multiple user requirements, agents sometimes refuse to comply, and sometimes make stacking errors.

ComboShoppingBench released by JD demonstrates this more intuitively: it asks agents to assemble a shopping cart that meets all specified criteria, with an average of 11 requirements per question. The success rate for individual requirements is around 95%, but when checking all requirements together, even the strongest GPT-5.5 + thinking only achieves a 61.2% success rate. In other words, when looking at each requirement in isolation, the agent performs like an outstanding student; but when all requirements are combined, its weaknesses are exposed.

## Recursive self-improvement: performance can degrade over iterations

RSI (Recursive self-improvement) refers to agents improving themselves using their own past experience. New benchmarks show this process is not always positive.

- **RSIBench-Data**: After an agent achieves its highest score during the same search session and continues searching, 78.26% of trials end with a lower final score, and the remaining only return to the original highest score. The paper suggests: predefine stopping conditions and reserve results from a specific search iteration.

- **PATH-Bench**: For two categories of tasks, single-turn coding and multi-turn tool use, agents are allowed to learn iteratively, with previously useful or distracting outdated experiences intentionally inserted into the process. Results show that later learned experiences can overwrite previously acquired capabilities, which may lead to lower overall scores. The solution: only allow experience relevant to the current task to enter the agent's context.

- **PAST-Bench**: It divides agent capabilities into four categories: memory, reuse of methods, information retrieval, and updating outdated information, and finds that these capabilities improve at inconsistent rates during RSI. After applying the method proposed in the paper, the capability of updating outdated information sees the most significant improvement.

## Data and Code

Koutian Wu has compiled over 5000 benchmark/eval entries available for direct download:

https://koutian.is-a.dev/benchmark-radar/data/radar.json

For citation reference, see CITATION.cff:

https://github.com/ktwu01/benchmark-radar/blob/main/CITATION.cff

Code repository (stars are welcome):

https://github.com/ktwu01/benchmark-radar