Build an MCP Observability Dashboard That Reveals Every Tool Execution

byHossein Hezami •
0

The first time an MCP-powered agent does something unexpected, everyone asks the same question:

“Which tool did it call?”

If you cannot answer that immediately, you do not have an agent system. You have a mystery box with a chat UI.

MCP tool execution is where the agent touches reality. It is where documents get read, tickets get created, databases get queried, files get written, and external APIs get called. If your observability stops at model prompts and chat responses, you are watching the shadows instead of the machine.

A serious MCP observability dashboard needs to show every tool execution: which server handled it, which tool was invoked, what arguments were sent, whether it succeeded, how long it took, why it failed, and how it relates to the user request that triggered it.

This article is about building that dashboard without drowning in logs, leaking secrets, or slowing down the agent.

TL;DR

To observe MCP tool executions properly:

  1. Instrument the MCP boundary, not just application logs.
  2. Turn every tools/call into a canonical execution event.
  3. Correlate executions with sessions, users, and agent turns.
  4. Redact and truncate payloads before storing them.
  5. Distinguish transport errors, protocol errors, tool errors, and policy denials.
  6. Use an async collector so observability does not block tool execution.
  7. Store execution records for debugging, not only aggregate metrics.
  8. Build dashboard views around real investigations.
  9. Secure the observability plane like an admin system.

If you only build one thing first, build the execution timeline. It is the MCP equivalent of a distributed trace view.

📋 Table of Contents

The Real Problem

MCP makes tool integration cleaner, but it does not automatically give you operational visibility.

A client may connect to several MCP servers. Each server may expose multiple tools. The model may call those tools in sequences you did not explicitly script. A single user request can produce one tool call, ten tool calls, or none at all.

When something goes wrong, the useful question is rarely:

“Did the model respond?”

The useful question is:

“Which tool execution caused the outcome?”

That requires a different kind of observability.

Traditional application logs usually capture what your code did. MCP observability needs to capture what the protocol did.

You need to know:

  • Which MCP server received the call?
  • Which tool name was invoked?
  • What arguments were passed?
  • Did the call reach the server?
  • Did the server return an MCP error?
  • Did the tool result include isError: true?
  • Did the call time out?
  • Was the call denied before execution?
  • Was it retried?
  • How large was the response?
  • Which session or conversation triggered it?

That is not a logging problem. It is an execution telemetry problem.

The dashboard you build should answer operational questions like:

  • “Show me every call to billing.refund_invoice from the last hour.”
  • “Which MCP server has the highest tool timeout rate?”
  • “Why did this agent turn take 18 seconds?”
  • “Did the failed tool call happen before or after the successful database query?”
  • “Which user session triggered this destructive tool?”
  • “What changed when tool failures spiked?”

Those are the questions that save you during an incident.

1. Instrument the MCP Boundary Not Just Your App Logs

Scenario

Your agent creates a support ticket even though the user only asked for a status update. The chat transcript shows the final response. Your application logs show that the agent loop ran. But nobody knows which tool was called, with which arguments, or whether the model tried three tools before the damaging one.

Why normal logs are insufficient

Application logs usually live above the MCP layer. They know that an agent responded, but they do not necessarily know how the MCP protocol messages flowed.

MCP tool execution happens through protocol operations. The one you care about most is the tool call operation, commonly represented at the protocol level as:

tools/call

If you only log inside your business logic, you may miss:

  • tool calls rejected before reaching your handler
  • invalid arguments caught by the server
  • protocol-level errors
  • transport failures
  • retries performed by the client
  • tool results marked as errors by the server
  • calls made by a different client component

Solution

Instrument the boundary where tool calls enter and leave.

There are three useful instrumentation points:

  1. Client side, where the MCP client invokes a tool.
  2. Server side, where the MCP server executes a tool.
  3. Proxy or middleware, where MCP messages pass between client and server.

For most teams, client-side and server-side instrumentation gives the best balance of clarity and control.

At a minimum, capture this when a tool call starts:

type JsonRpcRequest = {
  jsonrpc: "2.0";
  id: string | number;
  method: string;
  params?: unknown;
};

function isToolCallRequest(message: unknown): message is JsonRpcRequest {
  if (typeof message !== "object" || message === null) {
    return false;
  }

  const candidate = message as Partial<JsonRpcRequest>;

  return candidate.method === "tools/call";
}

If your MCP client SDK exposes a tool-call method, wrap it. If your server framework gives you a tool handler, wrap that too.

The key is to observe the operation itself, not only the surrounding application code.

Why this works

The MCP boundary is the choke point.

If you instrument the boundary, you see:

  • attempted calls
  • denied calls
  • failed calls
  • successful calls
  • calls that never reached the server
  • calls that reached the server but returned tool-level errors

That gives you a complete execution picture instead of a partial narrative.

💡 Practical note: If you run local MCP servers over stdio, remember that stdout is part of the protocol. Do not write diagnostic logs to stdout. Use stderr, files, or a separate collector transport.

2. Turn Every Tool Call Into a Canonical Execution Event

Scenario

You have three MCP servers. One logs JSON to stdout. Another logs plain text. The third emits metrics but no structured logs. When an incident happens, you spend more time translating log formats than understanding the failure.

Why ad hoc logs fail at scale

Every MCP server should not be allowed to invent its own observability shape.

If one server records:

{
  "tool": "search",
  "ok": true
}

and another records:

TOOL_EXEC search success

you cannot build a reliable dashboard.

You need one canonical event model.

Solution

Define a structured ToolExecutionEvent.

A practical schema looks like this:

type ToolExecutionStatus =
  | "started"
  | "completed"
  | "failed"
  | "timeout"
  | "denied"
  | "cancelled";

type ToolExecutionEvent = {
  eventId: string;
  traceId?: string;
  sessionId?: string;
  clientId?: string;
  serverId: string;
  transport: "stdio" | "http" | "websocket" | "unknown";
  toolName: string;
  status: ToolExecutionStatus;
  startedAt: string;
  endedAt?: string;
  durationMs?: number;
  attempt: number;

  request: {
    argumentKeys: string[];
    argumentPreview?: Record<string, unknown>;
    argumentsSha256?: string;
    sizeBytes?: number;
  };

  response?: {
    mcpIsError?: boolean;
    contentTypes?: string[];
    sizeBytes?: number;
    preview?: unknown;
  };

  error?: {
    kind: "transport" | "protocol" | "tool" | "policy" | "timeout";
    message: string;
    code?: string | number;
  };
};

This event is not just a log. It is a record of a tool execution lifecycle.

The most important fields are:

  • serverId
  • toolName
  • status
  • durationMs
  • attempt
  • error.kind
  • response.mcpIsError

Those fields power most of your dashboard views.

Why this works

A canonical event model lets you compare behavior across servers.

You can answer:

  • Which tools are slow?
  • Which servers produce the most errors?
  • Which tools are frequently retried?
  • Which tools return large responses?
  • Which tools are denied by policy?
  • Which executions belong to the same trace?

Without a canonical model, your dashboard becomes a pile of string searches.

The important part

Keep the event schema stable.

Once multiple clients and servers emit execution events, changing the schema becomes painful. Version it from the beginning:

type ToolExecutionEnvelope = {
  schemaVersion: "1.0";
  event: ToolExecutionEvent;
};

That small decision saves you later.

3. Correlate Tool Executions to Conversations and Requests

Scenario

A user reports:

“The agent gave me the wrong customer summary.”

You find the tool execution for crm.get_customer. But there are 400 executions with that name today. Which one belonged to this user? Which conversation? Which agent turn?

Why correlation matters

Tool executions are rarely isolated. They happen inside a larger interaction.

A single user request may trigger:

search_documents
get_customer
get_recent_orders
summarize_records

If you cannot connect those executions to the same request, you cannot reconstruct what happened.

Solution

Attach correlation identifiers to every execution event.

At minimum, use:

  • sessionId
  • userId, if available
  • agentTurnId
  • traceId

Example:

import { randomUUID } from "node:crypto";

type ExecutionContext = {
  traceId: string;
  sessionId: string;
  userId?: string;
  agentTurnId?: string;
};

function createExecutionContext(
  headers: Record<string, string | undefined>
): ExecutionContext {
  return {
    traceId: headers["x-trace-id"] ?? randomUUID(),
    sessionId: headers["x-session-id"] ?? "unknown",
    userId: headers["x-user-id"],
    agentTurnId: headers["x-agent-turn-id"]
  };
}

Then include that context when emitting tool execution events:

function startToolExecution(
  ctx: ExecutionContext,
  serverId: string,
  toolName: string,
  args: Record<string, unknown>
): ToolExecutionEvent {
  return {
    eventId: randomUUID(),
    traceId: ctx.traceId,
    sessionId: ctx.sessionId,
    clientId: "web-agent",
    serverId,
    transport: "http",
    toolName,
    status: "started",
    startedAt: new Date().toISOString(),
    attempt: 1,
    request: {
      argumentKeys: Object.keys(args),
      argumentPreview: redactObject(args),
      argumentsSha256: sha256(args),
      sizeBytes: Buffer.byteLength(JSON.stringify(args), "utf8")
    }
  };
}

HTTP and trace context

If your MCP servers are remote and use HTTP-based transports, standard trace headers are useful where supported. For example, W3C Trace Context headers can propagate trace identity if your infrastructure already uses them.

But do not make tracing depend on a single header mechanism. Sometimes MCP servers are local stdio processes. Sometimes the transport does not expose custom headers cleanly. In those cases, application-level identifiers such as session ID and agent turn ID are still extremely valuable.

Why this works

Correlation turns isolated events into a story.

Your dashboard can show:

User request
└── Agent turn
    ├── Tool: search_documents
    ├── Tool: get_customer
    ├── Tool: get_recent_orders
    └── Tool: summarize_records

That is the moment observability becomes useful.

⚠️ Gotcha: Do not put sensitive user identifiers into logs without reviewing privacy requirements. Correlation does not require raw personal data. Hashed or pseudonymous identifiers are often enough.

4. Redact Truncate and Hash Payloads Before Storage

Scenario

Your MCP tool accepts a customer support request. The arguments include an API token, a user email, and a long transcript. Your observability pipeline dutifully stores all of it.

Now your dashboard is a data breach waiting for an internal access mistake.

Why raw payloads are dangerous

Tool arguments and results often contain the most sensitive data in your system.

They may include:

  • credentials
  • tokens
  • personally identifiable information
  • internal file paths
  • customer records
  • private documents
  • database identifiers
  • error messages containing secrets

If you store raw payloads indefinitely, your observability system becomes a high-value target.

Solution

Use three payload strategies together:

  1. Redact
  2. Truncate
  3. Hash

Redaction removes obviously sensitive fields. Truncation prevents huge payloads from exploding storage. Hashing lets you compare payloads without storing them fully.

A simple redaction function:

const SECRET_KEY_PATTERN =
  /token|secret|password|authorization|apikey|api_key|credential|cookie|session/i;

function redactObject(input: unknown, depth = 0): unknown {
  if (depth > 4) {
    return "[max-depth]";
  }

  if (Array.isArray(input)) {
    return input.slice(0, 20).map((item) => redactObject(item, depth + 1));
  }

  if (input instanceof Date) {
    return input.toISOString();
  }

  if (input && typeof input === "object") {
    const entries = Object.entries(input as Record<string, unknown>);

    return Object.fromEntries(
      entries.map(([key, value]) => {
        if (SECRET_KEY_PATTERN.test(key)) {
          return [key, "[redacted]"];
        }

        return [key, redactObject(value, depth + 1)];
      })
    );
  }

  if (typeof input === "string" && input.length > 500) {
    return `${input.slice(0, 500)}...`;
  }

  return input;
}

For hashing, use a canonical JSON representation so key order does not change the hash:

import { createHash } from "node:crypto";

function canonicalJson(value: unknown): string {
  if (Array.isArray(value)) {
    return `[${value.map(canonicalJson).join(",")}]`;
  }

  if (value instanceof Date) {
    return JSON.stringify(value.toISOString());
  }

  if (value && typeof value === "object") {
    const keys = Object.keys(value as Record<string, unknown>).sort();

    const body = keys
      .map((key) => {
        const item = (value as Record<string, unknown>)[key];
        return `${JSON.stringify(key)}:${canonicalJson(item)}`;
      })
      .join(",");

    return `{${body}}`;
  }

  return JSON.stringify(value);
}

function sha256(value: unknown): string {
  return createHash("sha256").update(canonicalJson(value)).digest("hex");
}

Then store:

request: {
  argumentKeys: Object.keys(args),
  argumentPreview: redactObject(args),
  argumentsSha256: sha256(args),
  sizeBytes: Buffer.byteLength(JSON.stringify(args), "utf8")
}

Why this works

You keep enough information to debug without turning your dashboard into a sensitive-data warehouse.

The hash is especially useful. It lets you answer:

  • “Has this exact argument shape appeared before?”
  • “Did the same input produce different outputs?”
  • “Which executions used the same payload?”

without storing the full payload.

Production warning

Do not rely only on key-name regexes for redaction. Sensitive data can appear inside strings, nested objects, or tool results. For high-risk environments, add allow-listing, schema-aware redaction, or store only metadata by default.

5. Capture Failure Modes That Hide Inside Success

Scenario

Your dashboard says the tool call succeeded because the HTTP request completed. But the MCP result contains:

{
  "isError": true,
  "content": [
    {
      "type": "text",
      "text": "Permission denied"
    }
  ]
}

The transport succeeded. The tool failed.

Your success-rate chart is now lying to you.

Why this matters

MCP tool execution has multiple failure layers.

A call can fail because:

  • the server process is unavailable
  • the transport timed out
  • the request was malformed
  • the tool name was unknown
  • arguments failed validation
  • policy denied the call
  • the tool executed but returned an error result
  • the tool returned a partial result
  • the client cancelled the call
  • a retry loop gave up

If your dashboard only tracks transport-level success, you will miss the failures users actually experience.

Solution

Classify execution status deliberately.

Use separate statuses:

type ToolExecutionStatus =
  | "completed"
  | "failed"
  | "timeout"
  | "denied"
  | "cancelled";

And classify errors by kind:

type ToolErrorKind =
  | "transport"
  | "protocol"
  | "tool"
  | "policy"
  | "timeout";

A wrapper can make this explicit:

async function withTimeout<T>(promise: Promise<T>, timeoutMs: number): Promise<T> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => {
      reject(new Error("timeout"));
    }, timeoutMs);

    promise
      .then((value) => {
        clearTimeout(timer);
        resolve(value);
      })
      .catch((err) => {
        clearTimeout(timer);
        reject(err);
      });
  });
}

Then handle MCP tool errors separately from thrown transport errors:

type McpToolResultLike = {
  isError?: boolean;
  content?: Array<{ type?: string }>;
};

function classifyToolResult(result: unknown): {
  status: ToolExecutionStatus;
  error?: ToolExecutionEvent["error"];
} {
  const toolResult = result as McpToolResultLike;

  if (toolResult && toolResult.isError === true) {
    return {
      status: "failed",
      error: {
        kind: "tool",
        message: "Tool returned isError=true"
      }
    };
  }

  return {
    status: "completed"
  };
}

This distinction matters because a tool error may be a valid business outcome, while a transport error usually indicates infrastructure trouble.

Why this works

Your dashboard can now separate:

  • infrastructure failures
  • protocol failures
  • tool-level failures
  • policy denials
  • timeouts

That makes alerts and debugging dramatically better.

For example:

  • A spike in transport errors suggests server or network issues.
  • A spike in policy denials suggests permission or configuration drift.
  • A spike in tool errors suggests the downstream system is unhappy.
  • A spike in timeout errors suggests resource or latency issues.

Practical note

If you retry tool calls, record the attempt number. A dashboard that shows only the final result can hide a pattern where every successful call required two retries.

6. Build a Collector That Does Not Block the Agent

Scenario

You add rich observability. Every tool execution now writes synchronously to a database. The dashboard looks great.

Unfortunately, the agent now feels slow. When the database hiccups, tool calls stall. Users complain that the agent became unreliable.

Why observability must be asynchronous

Observability is important, but it should not become the most fragile dependency in your MCP pipeline.

If a tool execution is waiting for an analytics write, you have inverted the priorities.

Solution

Use a buffered collector.

The collector should:

  • accept events quickly
  • batch them
  • flush asynchronously
  • handle backend failures gracefully
  • drop or degrade safely under pressure

A minimal Node.js collector:

type CollectorOptions = {
  endpoint: string;
  maxQueueSize?: number;
  flushIntervalMs?: number;
};

class BufferedExecutionCollector {
  private queue: ToolExecutionEvent[] = [];
  private flushing = false;

  constructor(private options: CollectorOptions) {}

  emit(event: ToolExecutionEvent): void {
    const max = this.options.maxQueueSize ?? 1000;

    if (this.queue.length >= max) {
      // Prefer dropping the oldest event over blocking the agent.
      this.queue.shift();
    }

    this.queue.push(event);

    void this.flush();
  }

  private async flush(): Promise<void> {
    if (this.flushing || this.queue.length === 0) {
      return;
    }

    this.flushing = true;

    const batch = this.queue.splice(0, this.queue.length);

    try {
      await fetch(this.options.endpoint, {
        method: "POST",
        headers: {
          "content-type": "application/json"
        },
        body: JSON.stringify({
          schemaVersion: "1.0",
          events: batch
        })
      });
    } catch {
      // Requeue a limited number of events, but do not grow forever.
      this.queue = [...batch.slice(-100), ...this.queue].slice(-1000);
    } finally {
      this.flushing = false;
    }
  }
}

Use it like this:

const collector = new BufferedExecutionCollector({
  endpoint: "http://localhost:4321/v1/mcp/tool-executions"
});

collector.emit(executionEvent);

Why this works

The agent path stays fast.

The collector can fail, retry, or drop events without directly blocking the MCP tool call.

This is especially important for local MCP servers, where a slow observability backend can make the whole agent feel broken.

Gotcha

Decide your drop policy in advance.

Options include:

  • drop oldest events
  • drop newest events
  • drop payload previews but keep metadata
  • spill to disk
  • fail closed for high-risk tools only

For most systems, preserving lightweight metadata is more important than preserving every payload preview.

7. Store Executions for Debugging Not Only Metrics

Scenario

Your dashboard shows that tool latency increased between 2:00 and 2:15 PM. Great.

Now what?

You need to see actual executions: which tools, which servers, which error messages, which argument shapes, which attempts.

Metrics tell you something happened. Execution records tell you why.

Why raw execution records matter

Aggregate metrics are necessary, but they are not enough.

You need a queryable execution history.

At minimum, store:

  • event ID
  • trace ID
  • session ID
  • server ID
  • tool name
  • status
  • start time
  • end time
  • duration
  • attempt number
  • redacted request preview
  • redacted response preview
  • error details

A simple SQLite schema can go a long way:

CREATE TABLE IF NOT EXISTS tool_executions (
  event_id TEXT PRIMARY KEY,
  trace_id TEXT,
  session_id TEXT,
  client_id TEXT,
  server_id TEXT NOT NULL,
  transport TEXT NOT NULL,
  tool_name TEXT NOT NULL,
  status TEXT NOT NULL,
  started_at TEXT NOT NULL,
  ended_at TEXT,
  duration_ms INTEGER,
  attempt INTEGER DEFAULT 1,
  request_json TEXT,
  response_json TEXT,
  error_json TEXT
);

CREATE INDEX IF NOT EXISTS idx_executions_server_time
ON tool_executions(server_id, started_at);

CREATE INDEX IF NOT EXISTS idx_executions_tool_status
ON tool_executions(tool_name, status);

CREATE INDEX IF NOT EXISTS idx_executions_trace
ON tool_executions(trace_id);

In a Node service using better-sqlite3, insertion can be straightforward:

import Database from "better-sqlite3";

const db = new Database("mcp-observability.db");

const insertExecution = db.prepare(`
  INSERT INTO tool_executions (
    event_id,
    trace_id,
    session_id,
    client_id,
    server_id,
    transport,
    tool_name,
    status,
    started_at,
    ended_at,
    duration_ms,
    attempt,
    request_json,
    response_json,
    error_json
  ) VALUES (
    @event_id,
    @trace_id,
    @session_id,
    @client_id,
    @server_id,
    @transport,
    @tool_name,
    @status,
    @started_at,
    @ended_at,
    @duration_ms,
    @attempt,
    @request_json,
    @response_json,
    @error_json
  )
`);

function storeExecution(event: ToolExecutionEvent): void {
  insertExecution.run({
    event_id: event.eventId,
    trace_id: event.traceId ?? null,
    session_id: event.sessionId ?? null,
    client_id: event.clientId ?? null,
    server_id: event.serverId,
    transport: event.transport,
    tool_name: event.toolName,
    status: event.status,
    started_at: event.startedAt,
    ended_at: event.endedAt ?? null,
    duration_ms: event.durationMs ?? null,
    attempt: event.attempt,
    request_json: JSON.stringify(event.request),
    response_json: event.response ? JSON.stringify(event.response) : null,
    error_json: event.error ? JSON.stringify(event.error) : null
  });
}

For larger deployments, Postgres, ClickHouse, or an OpenTelemetry-compatible pipeline may be more appropriate. But the storage principle is the same: keep queryable execution records, not just counters.

Why this works

Execution records make debugging concrete.

You can ask:

  • Show all failed executions for billing.refund_invoice in the last hour.
  • Show all executions for this trace ordered by start time.
  • Show tool calls where duration exceeded 10 seconds.
  • Show executions with error.kind = 'timeout'.
  • Show all attempts for a specific tool call.

That is the difference between a dashboard that looks impressive and one that actually resolves incidents.

🔍 Why this matters: Metrics answer “is something wrong?” Execution records answer “what exactly went wrong?” You need both.

8. Design Dashboard Views Around Real Investigations

Scenario

Your dashboard has six panels:

  • CPU usage
  • memory usage
  • total MCP connections
  • generic error log stream
  • model latency
  • tool count

None of them helps when someone asks:

“Why did the agent fail to update the customer record?”

Why dashboard design matters

A good MCP observability dashboard is not a collection of generic infrastructure charts. It is an investigation tool.

The main views should map to real questions.

View 1: Execution Explorer

This is the core view.

Columns:

  • time
  • trace/session
  • server
  • tool
  • status
  • duration
  • attempt
  • error kind

Filters:

  • server ID
  • tool name
  • status
  • error kind
  • session ID
  • trace ID
  • time range

SQL example:

SELECT
  started_at,
  trace_id,
  session_id,
  server_id,
  tool_name,
  status,
  duration_ms,
  attempt,
  error_json
FROM tool_executions
WHERE started_at >= datetime('now', '-1 hour')
ORDER BY started_at DESC
LIMIT 100;

View 2: Execution Timeline

For a selected trace or session, show executions in order.

SELECT
  started_at,
  server_id,
  tool_name,
  status,
  duration_ms,
  attempt
FROM tool_executions
WHERE trace_id = ?
ORDER BY started_at ASC;

This view is the MCP equivalent of a distributed trace waterfall.

It should show:

  • overlapping tool calls
  • sequential dependencies
  • retries
  • long gaps
  • failed attempts before success

View 3: Tool Health

Aggregate by tool:

SELECT
  tool_name,
  COUNT(*) AS total_calls,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failed_calls,
  SUM(CASE WHEN status = 'timeout' THEN 1 ELSE 0 END) AS timeout_calls,
  AVG(duration_ms) AS avg_duration_ms,
  MAX(duration_ms) AS max_duration_ms
FROM tool_executions
WHERE started_at >= datetime('now', '-1 hour')
GROUP BY tool_name
ORDER BY failed_calls DESC;

This answers:

  • Which tools are failing?
  • Which tools are slow?
  • Which tools are timing out?

View 4: Server Health

Aggregate by server:

SELECT
  server_id,
  COUNT(*) AS total_calls,
  SUM(CASE WHEN status IN ('failed', 'timeout') THEN 1 ELSE 0 END) AS bad_calls,
  AVG(duration_ms) AS avg_duration_ms
FROM tool_executions
WHERE started_at >= datetime('now', '-1 hour')
GROUP BY server_id
ORDER BY bad_calls DESC;

This is especially useful when multiple MCP servers are owned by different teams.

View 5: Policy and Denials

If your registry or gateway blocks tool calls, surface those separately.

SELECT
  tool_name,
  COUNT(*) AS denied_calls
FROM tool_executions
WHERE status = 'denied'
  AND started_at >= datetime('now', '-24 hours')
GROUP BY tool_name
ORDER BY denied_calls DESC;

A sudden increase in denials can indicate:

  • permission drift
  • broken client configuration
  • an agent trying to use tools it should not see
  • a security issue worth reviewing

View 6: Payload Shape Changes

If you store argument hashes, you can detect unusual changes:

SELECT
  tool_name,
  request_json,
  COUNT(*) AS occurrences
FROM tool_executions
WHERE started_at >= datetime('now', '-1 hour')
GROUP BY tool_name, request_json
ORDER BY occurrences DESC
LIMIT 20;

This is rough, but useful when you want to know whether a new argument shape appeared before a failure spike.

Why this works

These views reflect how engineers actually investigate.

They start broad:

“What failed?”

Then narrow:

“Which server?”
“Which tool?”
“Which session?”
“What happened before it?”
“Was it retried?”
“Did the tool return an error?”

A dashboard that supports that flow becomes part of your incident response muscle memory.

9. Secure the Observability Plane Like an Admin Surface

Scenario

Your dashboard reveals every tool execution. That is excellent until someone realizes it also reveals customer IDs, internal file names, and redaction misses.

Now the observability system needs the same security review as the production systems it watches.

Why this matters

An MCP observability dashboard can become an admin plane.

It may expose:

  • tool names
  • internal server topology
  • user session identifiers
  • argument structures
  • error messages
  • downstream system names
  • partial business data
  • policy decisions
  • destructive operations

If access is too broad, the dashboard becomes a risk.

Solution

Apply ordinary production security controls.

At minimum:

  • require authentication
  • enforce role-based access
  • separate dev, staging, and production data
  • encrypt data at rest
  • encrypt transport
  • audit dashboard access
  • limit payload storage
  • set retention windows
  • redact before ingestion where possible

Retention is especially important.

A reasonable starting point:

DELETE FROM tool_executions
WHERE started_at < datetime('now', '-30 days');

For high-risk environments, keep metadata longer than payload previews. For example:

  • metadata: 90 days
  • payload previews: 7 days
  • full payloads: never, unless explicitly required and protected

Separate debugging data from analytics data

Not every consumer of observability needs the same detail.

Security, support, and engineering may need different views.

For example:

AudienceNeedsShould not usually see
On-call engineererrors, latency, traces, redacted previewsfull customer payloads
Security teamdenials, anomalous tool usage, audit eventsunnecessary PII
Product teamaggregate tool usageraw arguments
Compliance teamretention, access, audit recordslive execution payloads

Why this works

Observability is only useful if people trust it.

If the dashboard leaks sensitive information, teams will stop emitting useful data. Then you lose the very visibility you built.

🚨 Production warning: If your MCP tools can execute destructive operations, the observability dashboard should show who or which session triggered them. Without that, you have visibility without accountability.

Choosing the Right Architecture

There are several ways to build this. The best choice depends on how many MCP servers you run, how sensitive the data is, and how much operational complexity you can tolerate.

Architecture comparison

ApproachBest forStrengthMain trade-off
In-process event loggerSmall apps, prototypesSimple, fast to buildLimited cross-service visibility
Central HTTP collectorMultiple clients or serversShared schema and storageCollector becomes a service to operate
OpenTelemetry pipelineTeams already using OTelStrong tracing, metrics, logs integrationRequires careful MCP semantic mapping
MCP proxyFull protocol visibilityCan capture messages without changing serversComplex, security-sensitive
Vendor APM with custom eventsTeams already using APMFast dashboardingMay not model MCP semantics cleanly

In-process event logger

Start here if you have one client and a few servers.

Your application emits structured execution events into a local store or log file. A small dashboard reads from that store.

This is underrated for early-stage systems. You do not need a distributed observability stack on day one. You need reliable execution records.

Central HTTP collector

This is the natural next step.

Clients and servers send structured execution events to a collector service. The collector validates, redacts, stores, and exposes query APIs.

Use this when:

  • multiple clients call MCP servers
  • multiple teams own servers
  • you need one dashboard
  • you need centralized retention and access control

OpenTelemetry pipeline

OpenTelemetry is a strong fit if you already use it.

You can model each tool execution as a span, attach execution metadata as attributes, and emit errors as span events or logs.

This works well when you want MCP observability alongside HTTP calls, database queries, and model provider calls.

The challenge is semantic modeling. You need to decide what counts as a span:

  • the client tool request?
  • the server-side tool execution?
  • both?
  • retries as child spans?
  • policy checks as separate spans?

Those decisions matter. If you model them poorly, your traces become confusing.

MCP proxy

A proxy can observe MCP messages between client and server.

This is powerful because it can capture tool calls even when you cannot modify the client or server. But it also places a sensitive component directly in the communication path.

If you build a proxy, you must handle:

  • authentication
  • transport compatibility
  • message framing
  • timeouts
  • payload redaction
  • failure isolation
  • protocol version differences

I would not start here unless I had a strong reason.

What I Would Choose

If I were building an MCP observability dashboard from scratch, I would start with a canonical execution event and a small central collector.

The first version would be intentionally boring:

  • structured ToolExecutionEvent
  • client-side wrapper around tool calls
  • server-side wrapper around tool execution
  • redaction and truncation before emission
  • async buffered collector
  • Postgres or SQLite storage
  • simple dashboard with execution explorer and trace timeline

Only after that would I add:

  • OpenTelemetry integration
  • anomaly detection
  • payload diffing
  • policy analytics
  • long-term analytics warehouse

The most common mistake is to build beautiful aggregate charts before you can answer the basic question:

“Show me every tool execution for this user request.”

Once you can answer that, everything else becomes easier.

Production Checklist

Before trusting an MCP observability dashboard in production, I would want these boxes checked.

Event capture

  • Every client-side tool call emits an execution event.
  • Every server-side tool execution emits an execution event.
  • Denied calls are recorded.
  • Timeouts are recorded.
  • Retries include attempt numbers.
  • Tool results with isError: true are marked as failures.
  • Transport errors and tool errors are classified separately.

Correlation

  • Events include session IDs.
  • Events include trace or turn IDs where possible.
  • The dashboard can show executions ordered by time for a single trace.
  • Multiple tool calls from one agent turn are visible together.

Payload safety

  • Sensitive keys are redacted.
  • Long strings are truncated.
  • Large payloads are capped.
  • Full payloads are not stored by default unless required.
  • Argument hashes are stable and canonical.
  • Redaction is tested with realistic data.

Storage

  • Execution records are queryable.
  • Indexes exist for server, tool, status, trace, and time.
  • Retention is explicit.
  • Old payload previews are purged.
  • Storage failure does not block tool execution.

Dashboard

  • Execution explorer exists.
  • Trace/session timeline exists.
  • Tool health summary exists.
  • Server health summary exists.
  • Policy denial view exists.
  • Error details are visible without raw secret exposure.

Security

  • Dashboard requires authentication.
  • Access is role-based.
  • Production data is separated from development data.
  • Dashboard access is audited.
  • Observability data is encrypted at rest.
  • Retention policies are enforced automatically.

Reliability

  • Collector uses buffering.
  • Collector handles backend failures.
  • Event emission does not block MCP tool execution.
  • Drop behavior is explicit.
  • Collector health is monitored.
  • Pipeline backpressure is tested.

If you get those right, your MCP observability dashboard stops being a nice-to-have and becomes the part of the system that tells you what your agent actually did.

Tags:

Post a Comment

Previous Post Next Post