LangChain Observability: From Agent Traces to Useful Answers

When a LangChain support agent gives a wrong answer, the model is only one possible cause. The retriever may have returned the wrong policy. A tool may have failed. The conversation may have lost a previous constraint. A log containing the final answer cannot distinguish those cases.

LangChain observability connects the user’s question, retrieved evidence, tool activity, model response, and application outcome. Start with one interaction per user turn. Add spans at the boundaries that can explain a failure, and preserve the conversation across turns.

Agnost AI supports this through its conversation SDK and LangChain OpenTelemetry integration. The SDK captures the input and output you provide. Deeper trace coverage depends on the instrumentation enabled in your application.

LangChain monitoring boundaries: Question → Retrieval → Tool / model → Public answer → User outcome — Monitoring model for LangChain. Observe retrieval quality separately from fluent generation.

LangSmith traces and Agnost conversations

LangSmith supports tracing LangChain applications. Existing tracing can remain useful for understanding the execution of a chain or agent. Adding the Agnost wrapper below records the user-facing request and response; it does not import LangSmith history or automatically duplicate its spans.

Use a consistent application-level request identifier when comparing records across systems. First locate the execution that produced a questionable answer, then connect it with the conversation and the evidence of its outcome. If you enable additional OpenTelemetry instrumentation, check for duplicate model or tool spans before calculating usage. One request with several spans should still count as one user request.

Worked example: a cancellation answer uses the wrong policy

Suppose a support agent answers a customer’s cancellation question using a policy for a different plan. This is an illustrative RAG investigation; the events below are evidence to capture, not an account of an actual customer.

Check	Possible observation	Interpretation
Customer scope	Correct account, wrong plan filter	Retrieval context was incomplete
Retrieved documents	Relevant topic, inapplicable policy revision	Similarity did not establish applicability
Prompt context	Wrong policy passed unchanged	The error preceded generation
Final answer	Fluent paraphrase of that policy	Fluency concealed the upstream mistake
Follow-up	User points out their plan	A correction worth preserving

Inspect source applicability before changing the model. Document IDs and revision metadata can explain this case more directly than a larger dump of internal reasoning. If the correct document was retrieved but excluded before generation, inspect context assembly instead.

A candidate change is to constrain retrieval with verified account context and explicitly handle missing policy coverage. Evaluate that change on the same plan and question categories. An aggregate answer score may improve simply because fewer difficult plans appeared in the next sample.

In Agnost, use the conversation as the entry point and the underlying execution records to distinguish retrieval, assembly, and generation. Add your document references at the retrieval boundary if they are not exported already.

Agnost intent evidence view with illustrative demo data — **Product demo · illustrative data.** Intent evidence places the user input next to the agent output. Use this view to find recurring mismatches, then inspect retrieval or tool records to explain them. Select the image to inspect it at full size.

Metrics that answer useful LangChain questions

Use these definitions to design your measurement plan. Calculate them from the required application or tracing evidence; they are not all precomputed dashboard metrics.

Metric	Definition	Interpretation
Observed correction rate	Turns with an explicit correction / reviewed turns	Silence is not proof of a correct answer.
Retrieval coverage	Requests with applicable evidence / reviewed evidence-dependent requests	Requires an applicability judgment, not just a nonempty result.
Tool failure rate	Failed tool operations / tool operations	An HTTP response alone may not represent tool success.

Choose an interaction boundary that matches the product

For a support chat, the useful unit is usually a user turn and the final reply. For a background document review, it may be the submitted document and completed review. A tool call is an internal operation, not another user request.

Keep a stable conversation ID for follow-up questions. Add an agent version so a changed prompt, retrieval configuration, or tool set can be compared with its predecessor. Record retrieval configuration separately from the model name: changing chunking or document freshness can change answers without changing the model.

A compact evidence model might include:

Evidence	Why it helps
Sanitized question and answer	Shows what the user actually experienced
User and conversation IDs	Preserves multi-turn context
Agent/configuration version	Connects behavior to a release
Retrieved document IDs and revisions	Makes evidence freshness inspectable
Tool result status	Distinguishes usable output from an error message
Application outcome	Shows whether the task was completed

These are fields to capture where available, not a promise that every integration emits them automatically.

Add Agnost around an existing LangChain agent

The following adapter uses the current agent-style messages input. It expects an already configured agent from your application. It deliberately accepts an output serializer, because applications using chains, structured responses, and multimodal messages do not all return the same object.

Example scope: the code below is an integration adapter for an existing application. Configure its framework, model, credentials, and output contract as described; the demo screenshots were not generated by running these snippets.

pip install agnost
export AGNOST_ORG_ID="your-org-id"

import os
import agnost

agnost.init(os.environ["AGNOST_ORG_ID"])


def answer_question(agent, prompt, *, user_id, conversation_id,
                    public_answer):
    interaction = agnost.begin(
        user_id=user_id,
        conversation_id=conversation_id,
        agent_name="langchain-support",
        input=prompt,  # Redact sensitive content before this boundary.
        properties={
            "framework": "langchain",
            "agent_version": "support-v1",
            "retrieval_version": "policies-v1",
        },
    )
    try:
        result = agent.invoke({
            "messages": [{"role": "user", "content": prompt}],
        })
        text = public_answer(result)
    except Exception as exc:
        interaction.end(output=type(exc).__name__, success=False)
        raise
    else:
        interaction.end(output=text, success=True)
        return result

For a text-only message agent, your serializer can select the last public assistant message. For a structured-response agent, serialize only the approved output fields. For an existing chain, keep its input contract instead of replacing it with messages. The LangChain quickstart documents agent construction and invocation.

The wrapper records application identity; it does not configure model memory. If your agent uses a checkpointer, pass its required thread configuration too. Otherwise the dashboard may group turns correctly while the agent itself still lacks prior context.

Call agnost.shutdown() when the application process exits. A server should initialize once and keep the SDK available across requests. The captured duration spans the wrapped work, including your output selection. It is not a provider-only inference measurement.

Trace retrieval separately from generation

Consider a question about a cancellation policy. Diagnose it in the order the answer was constructed:

Was the correct account or product scope used?
Did retrieval find the applicable policy revision?
Did the relevant passage reach the model?
Did the answer reflect that passage accurately?
Did any requested action actually complete?

If the wrong document was retrieved, prompt tuning may only make the wrong answer more fluent. Store document identifiers, revision timestamps, and a minimal approved evidence excerpt when permitted. Retrieval scores alone are not a universal relevance measure; their meaning depends on the retrieval system.

Use Agnost’s OTel setup for supported LangChain traces. If a custom retriever or tool remains opaque, add a span around that operation. Inspect received data before building a metric around token usage, retrieval detail, or provider cost. Missing usage is unknown, not zero.

Separate tool success from request success

A tool can return successfully at the transport layer and still fail the user’s task. An HTTP 200 containing “no matching account” is one example. A booking API accepting a request without confirming inventory is another.

In your application, record an explicit outcome from the authoritative system. For a support agent that changes a subscription, useful states could include not_requested, awaiting_confirmation, completed, and failed. Preserve unknown outcomes. The final natural-language message is evidence about what the agent said, not proof that the subscription changed.

In Agnost, conversation review lets you inspect the interaction alongside available events. Intent analysis organizes configured user goals. A matched “cancel subscription” intent measures demand for that action; it should not be reported as a completed cancellation.

Review the slow requests that averages hide

Compare latency within a task category. A retrieval answer, account lookup, and multi-tool investigation do different work. Track the full response time separately from the time until the first visible output in a streaming application.

Repeated retrieval and repeated tool calls deserve inspection, but repetition is not automatically waste. A second lookup after the user clarifies their account is legitimate. A second identical lookup because the first result was lost may be a defect. Conversation context gives the trace meaning.

When requests run tools concurrently, don’t sum child durations and call the result user latency. Measure the outer request directly. Keep model token counts and cost estimates scoped to the operations that produced them, with provider and pricing assumptions visible.

Troubleshoot missing or misleading records

I see the answer but no tools. The conversation wrapper captures only the boundary shown above. Enable the framework instrumentation, and check coverage for your custom tools.

Related messages are split. Pass the same conversation ID from your application on each turn. Also check the separate memory/thread configuration.

One request is counted several times. Distinguish a user turn from internal agent iterations and multiple telemetry sources. Choose one root event for your request denominator.

The dashboard says success but the user asks again. Review how success is assigned. In this example it means normal execution, not resolved intent. Look at corrections and authoritative action outcomes.

A new release looks better. Compare equivalent task categories and traffic before attributing the change to the release. A simpler mix of incoming questions can improve an aggregate without improving the agent.

Frequently asked questions

Can I keep LangSmith and use Agnost?

You can keep an existing tracing setup while adding the Agnost conversation boundary, or configure supported trace export. Decide which records power each metric so dual instrumentation does not double request counts.

What should I monitor first?

Start with complete conversation identity, observed errors, response duration, and the public answer. Add retrieval evidence and tool results for the failure you need to explain. For graphs with persistent execution and human approvals, continue with LangGraph observability.

Browse all agent observability guides by framework.