Running NVIDIA Dory 4B on llama.cpp: Account for Every Single Loop

NVIDIA's Dory 1.0 4B is a type of looped language model. The coding guide published by Sana Hassan on MarkTechPost breaks down the running workflow into four steps: building the pinned version of the llama.cpp branch, analyzing computational costs per loop, testing memory cache performance, and carrying out paired evaluations. Original article link: How to Run NVIDIA Dory 4B with llama.cpp: Loop by Loop Guide.

For readers who are not familiar with looped models, here is a quick background note. Standard Transformer inference proceeds layer by layer, where parameters of each layer are calculated only once. A looped model makes the same set of computations run repeatedly for multiple iterations, trading extra compute for deeper processing. The total parameter count of the model stays unchanged, but inference latency and VRAM usage pressure will shift. The 4B in Dory 4B refers to its parameter scale, while the number of loops determines the actual total computational volume. When running this type of model, judging your user experience solely by the model file size will lead to misjudgment.

The core value of this guide lies in its standardized workflow. The first step is to pin the llama.cpp branch. "Pinned" here means the code version is fixed, so as to avoid upstream code updates skewing your experimental results. This step is non-negotiable for reproducible testing and benchmarking. The second step is to record computational costs on a per-loop basis. For each additional loop iteration, you need to track prefill and decoding latency, VRAM footprint, and throughput separately. Reporting only a total runtime that mixes all loops together lacks sufficient actionable information. The third step is to test memory cache performance. The caching strategy for looped models directly impacts cache hit rate and memory usage. If the cache is not properly configured, the more loops you add, the more obvious the waiting lag becomes. The fourth step is paired evaluation. Only by comparing different loop settings with the exact same input and identical parameter set can you verify whether the extra computation you added brings corresponding quality improvements.

If you plan to run Dory 4B locally, it is recommended that you first build the pinned branch as instructed in the original guide, then start testing with a small number of loops. Record the latency and memory usage curve for each loop, confirm the cache behavior, and finally run a comparison of paired outputs. Getting a single command to run successfully is just a warm-up. You have truly mastered running the model only when you know exactly where the computation of every single loop is spent. The full step-by-step workflow is available in the original article, linked above.