No Agent Yet, Evals First: The Build Order for an Invoice Lookup Project
When Ajitesh Shukla's team was building a supplier invoice lookup Agent, they wrote the eval runner first, then defined the Ports, and finally wrote the Agent. A mock adapter let evals run on day one; the real interface was swapped in later, and a regression suite came along for free.
Harrison Chase retweeted a post from Ajitesh Shukla about the development order for AI agents: write the eval runner first.
Ajitesh Shukla's team is building a supplier invoice lookup Agent for a P2P process using LangChain deep agents. P2P stands for procure-to-pay; the Agent needs to answer questions about invoice status and will later call real systems. Their approach: write the eval runner first and push the Agent to later.
Why write evals first:
- Before writing the first skill, you have to define what “correct” means
- Every change to a skill, model, or tool is rescored against the same golden dataset
- Agent improvements are driven by numbers, not vibes
The key design is Ports. The Agent depends on small Python interfaces; in tests, a mock adapter is plugged in and returns fixed test data. The code snippet is from the tweet:
class InvoicePort(Protocol):
def get_status(self, invoice_id: str) -> InvoiceStatus: ...
class MockInvoiceAdapter:
def get_status(self, invoice_id):
return FIXTURES[invoice_id]
The Agent can't tell whether it's connected to a mock or the real system. This detail makes evals reproducible and lets Agent development proceed without waiting for access permissions. Later, replace the mock with the real adapter, and neither the Agent nor the evals need to change. The frozen mock becomes the regression suite.
Benefits of this order:
- Reproducible evals with consistent results on every run
- Agent work starts on day one, without waiting for permissions
- The real adapter replaces the mock later, with no changes to the Agent or evals
- The frozen mock becomes a regression suite
This approach isn't flashy, but it gets ahead of two common problems: waiting on backend permissions and changing the Agent by feel. Evals first. Ports second. Agent third.
Original link: https://x.com/hwchase17/status/2121980898136801584
发布时间: 2026-11-16 05:26