LearnHarness: Diagnose First, Teach Second—Acceptance Is the Hard Part

LearnHarness puts the first step of a learning agent in diagnostic questioning—a clever sequencing choice. This open-source project, recommended by Xudong Han, first uses questions to establish someone's knowledge base and find gaps in understanding, then uses an agent to plan a learning path, teaching and practicing along the way and adjusting the content based on answers. Its Memory stores concepts learned and their reasoning relationships as a knowledge graph, so next time you can pick up where you left off. You can try it with math, machine learning, programming, physics, economics, philosophy, and even exam prep and reading complex papers. The project is at GitHub, and the original post is here.

The replies are more interesting than the project description. Byron Xiao starts with acceptance criteria: being able to plan a path does not mean the course has been built. Check the same concept again the next day; only if the learner can answer does it pass. Throw out pretty-sounding claims about the graph and keep the comparison baseline. Eaglegon adds a sharp point: the most fragile part of the whole system is the first-step diagnosis. If the blind spots are misidentified, the path planned afterward and the knowledge graph are all built on the wrong map. Math and programming have standard answers, so it is easy to verify whether the diagnosis is accurate; philosophy and paper reading have no standard answers, making them harder to judge.

Several people focused on memory. oocc says it would be very practical if it could remember which concept you got stuck on last time, so you can continue from there next time instead of starting again from "what is machine learning." Cyber Hermit reminds that with teach-and-practice plus dynamic adjustment, the key is how detailed the memory is—by lesson 10, does it still remember the weak spot from lesson 1? Leesanity hopes it can work as a plugin; the hidden issues in engineering show up in conversations, and learning from existing conversations would be more convenient. Miya asks whether Memory is stored locally or in the cloud. The materials do not answer this, so you need to confirm it yourself before trying it.

There is also a practical limitation. Gerald Ian took one look and found that it is under the AGPL license, so it cannot be integrated into a product. AGPL is a strong copyleft license; anyone wanting to put it into a closed-source product should be cautious, and this will put some people off.

Learning agents easily turn path planning into a demo; the real difficulties are whether the diagnosis is accurate, whether it still remembers the next day, and whether the license can be used. If you want to try LearnHarness, you can first test its diagnosis with math or programming, which have standard answers, then switch to paper reading, which has no standard answers, to see whether it will admit that it does not know. No matter how pretty the graph looks, if the learner cannot answer on the next day's spot check, it is exposed.