When RAG Gets It Wrong, Don't Blame the Model First: Five Places a Single Request Can Fail

A user gets a confidently worded, nicely formatted answer, but behind it the retrieval step may simply have grabbed the wrong chunk. Once an LLM app is live, inputs and outputs are not enough to debug with. In a single request, embedding, retrieval, context assembly, and generation can all add latency, call paid APIs, fail, and still produce a response that looks valid.

In Daily Dose of Data Science, Avi Chawla breaks a RAG request into a trace and spans. A trace records the full path of one request from query to response. A span records a single operation along that path—the colored blocks in the diagram.

The video below illustrates the relationship: on the left is the path from User Query to Final Response, and on the right the five colored blocks correspond to the durations of five types of spans.

What each of the five span types should capture:

Query span: input, timestamp, session identifier, and request metadata.
Embedding span: model, input size, latency, retries, and rate-limit errors.
Retrieval span: retrieved chunks, document IDs, relevance scores, filters, top-k, and latency. Many RAG failures start here. Without these fields, there is no evidence that retrieval selected the wrong documents.
Context span: the context assembled from retrieved chunks, instructions, and conversation history. It can catch truncated documents, duplicate chunks, missing citations, and prompts that exceed the token budget.
Generation span: model, token count, time to first token, total latency, finish reason, retries, and estimated cost.

With these fields, a bad answer can be traced back to retrieval, context assembly, or generation. One commenter replied that most bad answers he has seen came from retrieval pulling the wrong chunk, with the model simply taking the blame. Another commenter said capturing truncated documents in the context span is the first thing that should be added, because a silently truncated prompt looks from the outside like the model got dumber.

Opik turns this observability approach into an open-source implementation. It is built by Comet, licensed under Apache-2.0, and the full platform can be self-hosted. Opik captures LLM calls, retrieval steps, and tool executions, and attaches latency, token usage, and cost to each operation. All operations in a single request share the same Trace ID. If an app handles 1,000 requests, it generates 1,000 traces, and each trace contains its own spans.

This makes cost analysis more useful. Teams don't have to look only at total spend; they can pinpoint specific model calls, retries, or oversized prompts. Over time, changes in retrieval scores, embedding latency, and context size will surface first, before they turn into broader quality problems.

Getting started is straightforward:

pip install opik
opik configure

from opik import track

@track
def my_function(input: str) -> str:
    return input

GitHub repository: https://github.com/comet-ml/opik

Avi Chawla places observability among the eight areas in the 2026 LLM Engineering Roadmap. Original link: https://x.com/DailyDoseOfDS_/status/2115737924805460022