LangChain Built a Knowledge Factory Internally: Agents Report Their Own Errors, Knowledge Curators Catch Up Overnight
Harrison Chase separates context and knowledge: once models are good enough, what really holds agents back is the tacit knowledge scattered across Slack, Notion, and veteran employees' heads. LangChain's approach lets agents flag gaps, has a curator organize them in the background, and closes the loop with a revenue-report workflow.
On November 21, Harrison Chase tweeted with a link to 'How to build a knowledge factory.' He mentioned that LangChain's internal GTM (go-to-market) and content agents, as well as LangSmith Engine, all have a similar learning loop. The article addresses a very concrete question: when companies want AI to do work, the bottleneck is often knowledge that no one has organized, not the model.
He separates context and knowledge. Context is the information an agent has at hand when doing a task. Knowledge is reusable understanding, such as how a metric is actually calculated, which data source to trust, and what rules govern a certain type of situation. What the knowledge factory does is turn scattered documents and experience into context that future tasks can call on. Knowledge is part of context, but it usually should not be the only context an agent receives.
The analogy comes from the software factory. Over the past year and a half, the software factory idea has become popular: the best engineers should not just write software; they should build systems that make the whole company more efficient at writing software, with agents doing the actual coding. Harrison Chase believes general knowledge work is following the same path. Agents can already make presentations, publish videos, and do outbound calls. To get them to do this work, you first have to hand over tacit, scattered knowledge.
He lists three types of knowledge to maintain:
- How to do the work, Skills. The processes, instructions, and scripts for doing things correctly inside a company. Many processes were never written down before and were passed along by word of mouth among colleagues.
- How the business works, Domain knowledge. Facts, definitions, and policies about products, customers, competitors, and the organization. This type of knowledge does not expire often, but it is usually incomplete or even wrong.
- Where to get current information, Data access guidance. Where the data is, which source to query, and how to query it. Underlying data strategy such as Snowflake and Databricks is the foundation. If the source data is already complete and real-time, do not maintain another copy; what you need to maintain is clear access guidance.
The factory itself has three parts. First, let agents flag missing knowledge. When an agent cannot find information or discovers that information is wrong while doing work, it should report this through a tool, ideally with a suggested correct update. Second, a curator agent. It updates, organizes, and maintains knowledge in the background on a cadence, can access original sources such as Notion and Slack, can look at agent traces, can crystallize how people try to use things into skills, and can handle errors flagged by agents. Third, storage. Harrison Chase says storage is in some ways the least important: skills can go in GitHub, or in a single Notion file, LangSmith Context Hub's OKF format, or a Neo4j knowledge graph. The curator should be able to work with these formats. The harder problem is memory representation. The industry has converged on skills for procedural memory, but semantic memory has no standard yet, so do not define it too rigidly too early.
The article gives an end-to-end example: an agent prepares a revenue report. The raw revenue numbers are in BigQuery and Salesforce. Knowledge is divided into several layers: how to make a report in LangChain's style, stored in GitHub in the skills format; how ARR is calculated, stored in the company wiki; and how to access real-time data in BigQuery and Salesforce, also stored as skills in Context Hub. The agent runs, discovers that a column in BigQuery has been renamed, finds the correct column on the spot, and reports the issue through a tool while suggesting the new column name. After the report is done, the finance lead works with the agent in real time to change a few formatting details and tells it that all LangChain documents need a new header. That night, the curator sees the flagged issue and also checks the agent trace, notices this back-and-forth, and updates the relevant skills: use the new column for data retrieval and use the new header when drafting reports.
The replies surfaced several difficult questions. Adan Zweig asked who decides when a piece of knowledge expires, and whether the learning loop can retire old knowledge rather than only adding to it. Harrison Chase replied that the curator should be able to handle that. Draxx asked whether old corrections would be kept as tests to prevent new experience from overturning old experience. hims shared his own practice: he does not let sub-agents touch skills or knowledge, because if every sub-agent edits them, things quickly become a mess. Now sub-agents only report to the orchestrator, and the orchestrator makes unified changes based on what succeeded and what failed. Jack Schaffer asked who grades the curator. In his harness, the fixer can change many things, but permission gates cannot be touched, and every fix must beat held-out tasks under a token budget. His judgment is blunt: without a locked judge, the learning loop will learn to grade itself. Somi asked whether the curator's overnight changes need human review or go straight into skills.
August Wittorp posted an image in reply:
The article ends with a not-too-complicated starting point: do not wait until all documentation is written before letting agents start work. Pick one recurring workflow, see where the agent gets stuck, fill the gap, and get the loop turning. LangChain says they are already running this internally and are also working with several customers to connect this setup to internal and external agents.
Original article: How to build a knowledge factory
Tweet: Harrison Chase's tweet
发布时间: 2026-11-22 01:27