CrewAI Observability: Monitor Tasks, Delegation, and Real Outcomes

A CrewAI crew can finish without doing what the user asked. The researcher may collect weak evidence, the writer may ignore a constraint, or a tool may return an error as ordinary text. A successful Python return tells you the orchestration finished. It does not tell you whether the report was useful.

CrewAI observability connects a user’s request to the crew run, its tasks, agent activity, tool calls, and final result. Start by recording the request and response around crew.kickoff(). Add framework instrumentation when you need to locate the step that caused a delay or bad answer. Keep the user’s outcome separate from execution status.

Agnost AI provides agent analytics, observability, and monitoring. For CrewAI, its conversation SDK records the interaction you wrap; its OpenTelemetry ingestion can receive the deeper spans your instrumentation exports. These are different levels of coverage. Installing the conversation SDK alone does not automatically trace every task or tool.

CrewAI monitoring boundaries: User brief → Research task → Writing task → Final report → Acceptance — Monitoring model for CrewAI. Trace the tasks; measure the public request once.

Choose your CrewAI monitoring layer

CrewAI’s built-in tracing provides execution detail for Crews and Flows through CrewAI AMP. It is a useful starting point when the question is which agent, task, or tool ran. Follow its account and CLI setup instructions for your installed version.

The Agnost request wrapper below records the public interaction. Add the documented OpenTelemetry instrumentation when you also need internal spans. Keep a stable user and conversation identity so execution evidence can be reviewed alongside the request and result. These approaches capture different boundaries; verify the spans actually received before assuming equivalent coverage.

For example, use task-level evidence to investigate a delegation loop, then review the associated conversation to establish what the user needed. To assess whether the result was correct, record a review or downstream outcome separately. Neither a completed trace nor a returned crew result establishes that on its own.

Worked example: a report finishes with unsupported claims

Consider an illustrative two-agent crew that writes a competitor brief. The researcher gathers evidence; the writer turns that evidence into a report. The user asks for current pricing and links to primary sources.

Follow the evidence through the example:

Observation	What it tells you	Next inspection
The crew returns a report	Orchestration completed	Check the requested evidence
Research output contains an old pricing page	The evidence may be stale	Inspect source date and retrieval inputs
Writer supplies a current price without a source	The final claim is unsupported	Compare writer input with the claim
User asks “where did that number come from?”	The report did not meet the stated requirement	Preserve the correction in the conversation

The useful finding is that the writing step asserted something absent from its evidence. A different finding—fresh evidence existed but was omitted from the writer’s context—would point to the task handoff instead.

In Agnost, start with the public conversation to establish the user’s constraint. Inspect the available task and tool evidence next. Record source references and task version in your application when needed; those details do not appear simply because a crew has two agents.

A practical corrective experiment is to require the writer to leave unsupported fields unresolved and include the source reference with every price. Review comparable briefs after that change. Do not claim an improvement until the resulting reports have been checked.

Agnost tools view with illustrative demo data — **Product demo · illustrative data.** The Tools view separates call volume, recorded success, and P90 duration. Use it to choose the dependency to investigate before reading an entire crew transcript. Select the image to inspect it at full size.

Metrics that answer useful CrewAI questions

Use these definitions to design your measurement plan. Calculate them from the required application or tracing evidence; they are not all precomputed dashboard metrics.

Metric	Definition	Interpretation
Crew execution failure rate	Failed kickoff attempts / all recorded kickoff attempts	Keep retries distinct from user requests.
Delegation per request	Delegation operations / public requests	Segment by crew process and requested work.
Report acceptance rate	Explicitly accepted reports / reports with an observed decision	Show pending and unreviewed reports separately.

What should you record for a crew?

Use three boundaries. A conversation groups related user requests. A run is one execution attempt. A task is work performed inside that attempt. An application retry creates another attempt; it should not become another apparent customer.

Boundary	Useful evidence	Question it answers
User request	Sanitized input, user ID, conversation ID	What did this person need?
Crew run	Run ID, crew version, duration, terminal status	Did this attempt finish?
Task or delegation	Responsible agent, task, inputs, result	Where did the output go wrong?
Tool call	Tool name, duration, result status	Did the agent get usable evidence?
Product outcome	Accepted report, approved action, explicit feedback	Did the work help?

A report-generation product might distinguish generated, accepted, and revised. Those are application-defined outcomes, not built-in CrewAI or Agnost guarantees. If you only observe generation, call the metric generation completion; don’t label it customer success.

Capture the request and final answer with Agnost

This example wraps an existing, working crew. Its tasks must accept a {prompt} input. Keep your current CrewAI model and tool configuration; the instrumentation does not require changing either.

Example scope: the code below is an integration adapter for an existing application. Configure its framework, model, credentials, and output contract as described; the demo screenshots were not generated by running these snippets.

pip install agnost
export AGNOST_ORG_ID="your-org-id"

import os
from uuid import uuid4
import agnost

# Initialize once when your worker starts.
if not agnost.init(os.environ["AGNOST_ORG_ID"]):
    raise RuntimeError("Agnost initialization failed")


def run_report(crew, prompt: str, user_id: str, conversation_id: str):
    # This example expects input already approved for telemetry.
    interaction = agnost.begin(
        user_id=user_id,
        conversation_id=conversation_id,
        agent_name="crewai-report",
        input=prompt,
        properties={
            "framework": "crewai",
            "crew_version": "report-v1",
            "run_id": str(uuid4()),
        },
    )
    try:
        result = crew.kickoff(inputs={"prompt": prompt})
    except Exception as exc:
        # Avoid sending exception messages containing tool credentials or data.
        interaction.end(output=type(exc).__name__, success=False)
        raise
    else:
        interaction.end(output=result.raw, success=True)
        return result

Call agnost.shutdown() when the worker exits so queued events can be sent. In a one-shot script, put that call in an outer finally block. In a web server, use the process lifecycle rather than shutting down the shared SDK after every request.

The success=True above means the invocation returned normally. The SDK measures elapsed time between begin and end; it does not infer report acceptance. Use your authenticated application’s user and conversation IDs instead of the sample names. Strip sensitive fields before recording both inputs and outputs.

CrewAI returns a CrewOutput; raw holds its final textual result. Task-level outputs are separate, which matters when the final response hides an earlier mistake. See CrewAI’s crew and output reference.

Add task and model traces when the final answer is insufficient

Use Agnost’s CrewAI integration for the OpenLIT path. CrewAI also documents OpenLIT instrumentation. Configure the exporter before creating and running your crew:

pip install openlit
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.agnost.ai"
export OTEL_EXPORTER_OTLP_HEADERS="X-Agnost-Org-ID=your-org-id"

import openlit

openlit.init(
    application_name="report-worker",
    environment="production",
    disable_metrics=True,
)
# Construct and run your existing CrewAI application after initialization.

This configures a trace destination. The exact tasks, model attributes, token usage, and tool spans depend on the installed instrumentation and provider versions. Inspect a real exported run before relying on any particular field. A custom tool performing several internal HTTP requests may need its own spans to explain where time went.

If you combine SDK interactions and automatic traces, label their boundaries clearly. A conversation event and a crew span can describe the same invocation. Counting both as completed user requests inflates volume. Pick one source for request-level metrics and use deeper spans for diagnosis.

Diagnose the failures that are specific to crews

The final answer is wrong, but every task completed. Inspect the first task whose output no longer supports the user’s request. In a research-and-writing crew, compare the evidence collected with the claims written. If the researcher had no relevant evidence, changing the writer’s tone will not fix the problem. Record task outputs selectively so you can follow that dependency without collecting an entire sensitive document corpus.

Delegation keeps repeating. Compare agent-to-agent transfers with changes in evidence. Repeated transfers are useful when specialists add information. A cycle that passes the same unresolved question between agents is a different pattern. Capture the delegating agent, receiving agent, and task boundary; investigate unclear ownership or an unavailable tool before increasing iteration limits.

The crew is slow only for some requests. Segment by requested work. A long research report and a short classification task should not share one latency target. Look at the longest dependent sequence, not the sum of parallel task durations. Otherwise concurrency can make your reported latency exceed what the user actually waited.

A tool failed, but the run says success. Some tools return a message such as “document unavailable” rather than raising. Preserve a structured result status in your own tool boundary. A model can write fluent prose after receiving unusable tool output.

A retry produced two reports. Give each attempt a distinct run ID while keeping the same conversation ID. Separately identify any external action, such as sending a report email, so you can tell repeated reasoning from repeated delivery.

Turn traces into a weekly product review

In Agnost’s Users, review related interactions as conversations. In Events, inspect the records and spans that arrived. Configured intents help organize requests by what users wanted; an intent match is not proof of resolution.

Start with questions you can answer from evidence:

Which report types most often lead to a correction or another attempt?
Which crew version produces more unusable tool results for the same task category?
Are slow requests waiting on research, delegation, or final generation?
Does a normally completed run lead to an accepted report?

Record the last outcome in your application when it occurs. Do not invent it from a positive-sounding response. Keep unknown outcomes visible instead of quietly adding them to either successes or failures.

Verify coverage before trusting the dashboard

Run one ordinary request and inspect its input, final output, identity, and timestamps. Then inspect a known failed invocation. The failed attempt should remain distinguishable from a valid response. If you enabled tracing, verify that the task or tool you care about actually appears; a root span alone is not full crew visibility.

Missing records usually call for checking exporter configuration, organization ID, initialization order, sampling, and worker shutdown. Missing context calls for checking identity propagation. Neither is fixed by adding more keywords to an agent name.

Frequently asked questions

Does CrewAI monitoring require tracing every model call?

No. A request wrapper gives a useful starting point for conversation review and execution latency. Deeper tracing becomes necessary when you need to attribute retries, tool failures, or work to a particular agent.

Can I use Agnost with an existing tracing tool?

The conversation SDK can sit at the application boundary while your existing tool handles execution traces. If you export traces to multiple destinations, review data handling and duplicate counting. Choose coverage deliberately rather than assuming two dashboards see identical data.

For the concrete setup, use Agnost’s CrewAI guide. If your workflow pauses and resumes across requests, the LangGraph monitoring guide explains the additional distinction between a persistent thread and an execution attempt.

Browse all agent observability guides by framework.