After Agents Joined the Experiment Loop, Databricks Training Jobs Grew 8x—and the Bottleneck Changed

The Databricks AI research team handed part of its experiment loop to agents. The process is not complicated: an agent proposes a change to a model or training method, submits multiple experiments in parallel, reads training metrics, decides which direction is working, and then starts the next round. Databricks said in its blog that this agent-driven approach to experimentation led to an 8x increase in the number of training jobs for its AI Functions research workload in a year.

An 8x increase sounds like an efficiency story. What was exposed first was infrastructure friction. The Databricks team trains models ranging from several billion parameters to nearly a trillion parameters, using methods including SFT, offline RL, and online RL, on H100 or B300 GPUs. The largest training jobs require hundreds of GPUs running for days to weeks, plus high-bandwidth networking. If one node or the RDMA network has a problem, hundreds of GPUs sit idle and training can only fall back to an old checkpoint and start over. Long-running jobs are most vulnerable to this kind of interruption.

These models support Databricks AI Functions, including ai_parse_document and ai_extract. Databricks listed three challenges: fault tolerance for long training jobs, fast experimentation and prototyping, and allocation of GPU queues across multiple projects. In the context of agent experiments, the problems are even more direct. When researchers manually run a few experiments, slower startup is tolerable. When an agent submits hundreds of jobs at once, startup latency, configuration errors, and bad nodes all become throughput bottlenecks.

AI Runtime is Databricks' product for managing this layer of GPU training infrastructure. The research team has been customer zero since the product's early days. AIR checks GPU node health and communication before training, and during training monitors NCCL communication failures, thermal or power throttling, and GPU driver issues. After a bad node causes a job to fail, the system quarantines that node and no longer schedules subsequent jobs onto it. For RL projects like KARL, which has more than 100 billion parameters and requires thousands of GPU-hours, this is more practical than repeatedly troubleshooting hardware failures.

Fast iteration depends on two things. The AIR CLI lets jobs be launched programmatically and specify GPUs, with AIR handling provisioning and scheduling, including high-bandwidth networking for cross-cloud distributed training. Custom Docker containers keep the local development environment consistent with the training environment; after code is uploaded, it runs in the same container, and images can be registered and shared through Databricks Artifact Registry. Databricks says AIR's investment in startup latency and container optimization can reduce training job startup to seconds, shortening the experiment feedback loop.

Experiment tracking uses MLflow. AIR automatically creates and manages MLflow experiments for training jobs, and researchers can see status, stream logs, and retrieve metrics. Agents also rely on this information to evaluate results and decide the next step. Research data is stored in Unity Catalog and mounted through FUSE, so training code can access it like local files; it also works across workspaces, with data permissions and tracking preserved.

GPU capacity management is another friction point. Debugging experiments may need only a few GPUs, while large training runs need hundreds for several days. AIR's priority and preemption capabilities allocate resources across projects: time-sensitive experiments run first, and idle capacity can take on exploratory experiments. Databricks gives the example of using idle capacity on weekends to run extra experiments without interrupting high-priority work. The goal is not to have researchers manually coordinate machines, but to ensure that the experiments that should run have GPUs available.

The agent loop is influenced by autoresearch, proposed by Andrej Karpathy. Databricks breaks the process into four steps: propose, submit parallel trials, read metrics and learn, update the plan and start experiments again. AIR fits this loop because agents can use the Databricks CLI and databricks-air-skills to programmatically submit concurrent jobs, monitor status, retrieve MLflow metrics, and iterate based on results.

There is an easily overlooked change here. When humans do research, infrastructure problems are friction; when agents do research, infrastructure problems become the ceiling on experiment throughput. Slow startup, vague errors, and inaccessible results all leave the agent spinning in place. Databricks' conclusion is very product-oriented: agent-driven research requires fast job startup, programmatic interfaces, clear and actionable errors, reliable execution, and accessible training results. It sounds like an infrastructure checklist, but it actually determines how many ideas an agent can try in a day.

If you train or fine-tune models yourself, you can read Databricks' AI Runtime documentation. The original article is here, and the autoresearch project is on GitHub.