Build an MCP Contract-Test Suite That Catches Breaking Changes Automatically

byHossein Hezami •
0

The breaking change was tiny.

Someone added a new required argument to an MCP tool:

{
  "name": "search_orders",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "region": { "type": "string" }
    },
    "required": ["query", "region"]
  }
}

Previously, only query was required.

The server tests passed. The API team shipped the change. Then three agent workflows started failing because their stored tool calls no longer validated. The model kept trying the old argument shape. Users saw “invalid tool arguments” errors. Nobody thought to check the MCP tool schema because the tool still existed and the server still booted.

That is the core problem with MCP servers: they are APIs, but they often get tested like scripts.

A good MCP contract-test suite treats your server’s tool catalog, input schemas, result shapes, error behavior, and metadata as versioned contracts. When someone changes those contracts incompatibly, CI fails before the agent fleet finds out the hard way.

This article is about building that suite.

TL;DR

To catch breaking MCP changes automatically:

  1. Treat your MCP server like a versioned API.
  2. Capture a golden contract from tools/list.
  3. Run the server in CI with deterministic fixtures.
  4. Diff schemas using compatibility rules, not raw equality.
  5. Test argument validation behavior, not just schema text.
  6. Lock result shapes and error semantics.
  7. Track model-facing metadata drift separately.
  8. Add client-side contract tests if you maintain a client.
  9. Make baseline updates explicit, reviewed, and auditable.

The goal is not to prevent change. The goal is to make breaking changes visible, intentional, and versioned.

📋 Table of Contents

The Real Problem

MCP servers expose capabilities through protocol operations such as initialization, tool discovery, and tool invocation. Clients use those operations to figure out what the server can do.

That means the server’s contract is not only its code. It includes:

  • server capabilities
  • server info and version
  • tool names
  • tool descriptions
  • input schemas
  • required arguments
  • enum values
  • validation behavior
  • tool result structure
  • error semantics
  • transport behavior
  • metadata that influences model behavior

A unit test that checks a function returns the right database row does not catch this:

- "required": ["query"]
+ "required": ["query", "region"]

Nor does it catch this:

- "name": "search_orders"
+ "name": "search_customer_orders"

Nor this:

- "status": ["open", "paid", "void"]
+ "status": ["open", "paid"]

All three can break clients, agents, cached prompts, stored workflows, or model behavior.

A contract-test suite exists to make those changes explicit.

1. Treat Your MCP Server Like a Public API Not a Script

Scenario

Your team owns an internal MCP server. It is used by a support agent, a developer assistant, and an automated reporting workflow. One engineer changes a tool argument because “we can update the callers later.”

The callers are not just code. Some are model prompts. Some are saved agent flows. Some are client-side caches. Some are human expectations.

Why this matters

MCP servers sit at a strange boundary. They feel like internal utilities, but they behave like public APIs.

Once a tool is exposed to a client or model, its shape becomes a contract.

You need to decide what your server promises.

A useful MCP contract includes:

Contract areaExample promise
Tool identitysearch_orders will not disappear without a major version bump
Input schemaExisting valid arguments remain valid
Required fieldsNew required fields are not added without defaults or versioning
EnumsExisting accepted values remain accepted
ResultsRead-only tools return a stable result shape
ErrorsInvalid arguments fail in a predictable way
MetadataTool names and descriptions are reviewed before changes
CapabilitiesServer capabilities do not silently shrink

Solution

Write a compatibility policy before writing tests.

A practical starting point:

Backward-compatible changes:
- adding a new tool
- adding an optional argument
- relaxing a constraint
- adding an enum value, unless clients strictly validate choices
- improving descriptions in a non-behavioral way

Breaking changes:
- removing a tool
- renaming a tool
- adding a required argument without a default
- removing an argument clients may send
- removing an enum value
- narrowing string/number constraints
- changing result fields that clients parse
- changing error semantics

Then encode that policy into CI.

Why this works

The policy gives your tests meaning.

Instead of asking:

Did the schema change?

you ask:

Did the schema change in a way that breaks existing clients?

That distinction is critical. MCP servers evolve. You do not want CI to fail because somebody improved a description or added a harmless optional field. But you do want CI to fail when a previously valid tool call becomes invalid.

💡 Practical note: If your MCP server is consumed by autonomous agents, treat model-visible metadata as part of the contract too. A renamed tool can be just as breaking as a removed endpoint.

2. Capture a Golden Contract From tools/list

Scenario

You want to detect breaking changes, but you are not sure what to compare against. The server’s code changes often. The tool list is generated dynamically. Documentation is stale.

You need a source of truth.

Why this matters

The first artifact in a contract-test suite is the contract itself.

For most MCP servers, the most important contract surface is the tool catalog:

tool name
description
input schema

You can capture this from the server’s tool-discovery operation and store it as a baseline file.

Solution

Use a test client to connect to the server and record the tool catalog.

The following example uses Python and pytest-style async tests, but the same idea works in any MCP SDK.

# tests/contracts/capture.py

import json
from pathlib import Path

BASELINE_PATH = Path("tests/contracts/baseline.json")


async def capture_contract(session):
    result = await session.list_tools()

    tools = []

    for tool in result.tools:
        tools.append(
            {
                "name": tool.name,
                "description": tool.description,
                "inputSchema": tool.inputSchema,
            }
        )

    tools.sort(key=lambda item: item["name"])

    return {
        "format": "mcp-contract/v1",
        "tools": tools,
    }

Then create a test that can regenerate the baseline when explicitly requested:

# tests/contracts/test_baseline.py

import json
import os

import pytest

from .capture import BASELINE_PATH, capture_contract


@pytest.mark.asyncio
async def test_update_baseline(session):
    if os.environ.get("UPDATE_BASELINE") != "1":
        pytest.skip("Set UPDATE_BASELINE=1 to regenerate the MCP contract baseline")

    contract = await capture_contract(session)

    BASELINE_PATH.parent.mkdir(parents=True, exist_ok=True)
    BASELINE_PATH.write_text(
        json.dumps(contract, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )

The baseline file might look like this:

{
  "format": "mcp-contract/v1",
  "tools": [
    {
      "name": "get_invoice",
      "description": "Fetch a single invoice by ID.",
      "inputSchema": {
        "type": "object",
        "properties": {
          "invoice_id": {
            "type": "string"
          }
        },
        "required": ["invoice_id"],
        "additionalProperties": false
      }
    },
    {
      "name": "search_orders",
      "description": "Search customer orders.",
      "inputSchema": {
        "type": "object",
        "properties": {
          "query": {
            "type": "string"
          }
        },
        "required": ["query"],
        "additionalProperties": false
      }
    }
  ]
}

Why this works

The baseline becomes your lockfile.

It gives CI something concrete to compare against. When the server changes, you can diff the live contract against the committed baseline.

This also makes changes reviewable. A pull request that modifies baseline.json becomes a visible contract change.

Gotcha

Do not generate the baseline from production unless you are certain production is correct. The baseline should represent the contract you intend to support, not whatever the server happened to return last Tuesday.

3. Run the Server in CI With Deterministic Fixtures

Scenario

Your contract tests pass locally. In CI, they fail randomly because the server connects to a real database, tool results change, and one test accidentally creates a record.

Why this matters

Contract tests need stable inputs.

If your MCP server can read invoices, search orders, or create tickets, the test environment must return predictable data. Otherwise, you cannot distinguish contract failures from data drift.

Solution

Run the server in fixture mode.

Use environment variables to tell the server to use canned data:

# tests/conftest.py

import os
import shlex

import pytest_asyncio

from mcp.client.session import ClientSession
from mcp.client.stdio import StdioServerParameters, stdio_client


@pytest_asyncio.fixture
async def session():
    command = os.environ.get(
        "MCP_SERVER_COMMAND",
        "python -m example_mcp_server",
    )

    parts = shlex.split(command)

    server_params = StdioServerParameters(
        command=parts[0],
        args=parts[1:],
        env={
            **os.environ,
            "MCP_FIXTURE_MODE": "1",
            "MCP_FIXTURE_SET": "contract-tests",
        },
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            yield session

Your server should recognize fixture mode:

MCP_FIXTURE_MODE=1
MCP_FIXTURE_SET=contract-tests

In fixture mode, the server should:

  • use seeded read-only data
  • disable external network calls
  • disable destructive side effects or route them to a fake sink
  • return stable IDs, timestamps, and amounts
  • expose the same tool catalog as normal mode
  • fail deterministically for known invalid cases

Example fixture data:

{
  "invoices": [
    {
      "id": "inv_fixture_1",
      "status": "paid",
      "amount_cents": 4200,
      "currency": "USD"
    },
    {
      "id": "inv_fixture_missing",
      "status": "open",
      "amount_cents": 1000,
      "currency": "EUR"
    }
  ]
}

Why this works

Deterministic fixtures make contract failures obvious.

If get_invoice suddenly returns a different shape, the test fails because the shape changed, not because the database changed.

This also lets you test error behavior safely. You can define fixtures like:

inv_fixture_unauthorized
inv_fixture_server_error
inv_fixture_not_found

Then assert that the MCP server handles them according to contract.

🚨 Production warning: Do not run contract tests against a live production MCP server unless the tools are strictly read-only and the data is safe to query repeatedly.

4. Diff Schemas With Compatibility Rules Not String Equality

Scenario

Your baseline comparison fails because a description changed from:

Search orders.

to:

Search customer orders.

Meanwhile, a truly breaking change slips through because it was hidden in a large schema diff.

Why string equality is insufficient

Naive snapshot testing is too noisy and too shallow.

It fails on harmless changes, but it does not explain whether a schema change is compatible.

You need compatibility rules.

Solution

Separate structural schema changes into breaking and non-breaking categories.

A practical compatibility checker can be written in plain Python:

# tests/contracts/compatibility.py

import json


def find_breaking_changes(old_contract, new_contract):
    issues = []

    old_tools = {tool["name"]: tool for tool in old_contract.get("tools", [])}
    new_tools = {tool["name"]: tool for tool in new_contract.get("tools", [])}

    for name, old_tool in old_tools.items():
        new_tool = new_tools.get(name)

        if new_tool is None:
            issues.append(f"breaking: tool removed: {name}")
            continue

        issues.extend(check_tool(name, old_tool, new_tool))

    return issues


def check_tool(name, old_tool, new_tool):
    issues = []

    old_schema = old_tool.get("inputSchema") or {}
    new_schema = new_tool.get("inputSchema") or {}

    issues.extend(
        check_object_schema(
            f"tool {name}",
            old_schema,
            new_schema,
        )
    )

    return issues

Then inspect object schemas:

def check_object_schema(path, old_schema, new_schema):
    issues = []

    issues.extend(check_type(f"{path}.inputSchema", old_schema, new_schema))

    old_properties = old_schema.get("properties") or {}
    new_properties = new_schema.get("properties") or {}

    old_required = set(old_schema.get("required") or [])
    new_required = set(new_schema.get("required") or [])

    for name in sorted(new_required - old_required):
        prop = new_properties.get(name) or {}

        if "default" not in prop:
            issues.append(
                f"breaking: {path}: newly required property '{name}' has no default"
            )

    for name in sorted(old_properties.keys() - new_properties.keys()):
        issues.append(
            f"breaking: {path}: property removed: '{name}'"
        )

    for name, old_prop in old_properties.items():
        new_prop = new_properties.get(name)

        if new_prop is None:
            continue

        issues.extend(
            check_property_schema(
                f"{path}.{name}",
                old_prop,
                new_prop,
            )
        )

    return issues

Check type narrowing:

def as_type_set(value):
    if value is None:
        return set()

    if isinstance(value, list):
        return set(value)

    return {value}


def check_type(path, old_schema, new_schema):
    issues = []

    old_types = as_type_set(old_schema.get("type"))
    new_types = as_type_set(new_schema.get("type"))

    if not old_types and new_types:
        issues.append(
            f"breaking: {path}: type constraint added; previously any type was allowed"
        )

    if old_types and new_types and not old_types.issubset(new_types):
        issues.append(
            f"breaking: {path}: type narrowed from {sorted(old_types)} to {sorted(new_types)}"
        )

    return issues

Check property-level constraints:

def enum_key(value):
    return json.dumps(value, sort_keys=True)


def check_property_schema(path, old_prop, new_prop):
    issues = []

    issues.extend(check_type(path, old_prop, new_prop))

    old_enum = old_prop.get("enum")
    new_enum = new_prop.get("enum")

    if old_enum is not None:
        if new_enum is None:
            issues.append(
                f"review: {path}: enum removed; verify old values are still accepted"
            )
        else:
            old_values = {enum_key(value) for value in old_enum}
            new_values = {enum_key(value) for value in new_enum}

            removed_values = old_values - new_values

            if removed_values:
                issues.append(
                    f"breaking: {path}: enum values removed: {sorted(removed_values)}"
                )

    old_minimum = old_prop.get("minimum")
    new_minimum = new_prop.get("minimum")

    if (
        isinstance(old_minimum, (int, float))
        and isinstance(new_minimum, (int, float))
        and new_minimum > old_minimum
    ):
        issues.append(
            f"breaking: {path}: minimum increased from {old_minimum} to {new_minimum}"
        )

    old_maximum = old_prop.get("maximum")
    new_maximum = new_prop.get("maximum")

    if (
        isinstance(old_maximum, (int, float))
        and isinstance(new_maximum, (int, float))
        and new_maximum < old_maximum
    ):
        issues.append(
            f"breaking: {path}: maximum decreased from {old_maximum} to {new_maximum}"
        )

    old_min_length = old_prop.get("minLength")
    new_min_length = new_prop.get("minLength")

    if (
        isinstance(old_min_length, int)
        and isinstance(new_min_length, int)
        and new_min_length > old_min_length
    ):
        issues.append(
            f"breaking: {path}: minLength increased from {old_min_length} to {new_min_length}"
        )

    old_max_length = old_prop.get("maxLength")
    new_max_length = new_prop.get("maxLength")

    if (
        isinstance(old_max_length, int)
        and isinstance(new_max_length, int)
        and new_max_length < old_max_length
    ):
        issues.append(
            f"breaking: {path}: maxLength decreased from {old_max_length} to {new_max_length}"
        )

    old_pattern = old_prop.get("pattern")
    new_pattern = new_prop.get("pattern")

    if old_pattern is not None and old_pattern != new_pattern:
        issues.append(
            f"review: {path}: pattern changed; regex compatibility is hard to prove"
        )

    return issues

Use it in a test:

# tests/contracts/test_contract_compatibility.py

import json

import pytest

from .capture import BASELINE_PATH, capture_contract
from .compatibility import find_breaking_changes


@pytest.mark.asyncio
async def test_no_breaking_contract_changes(session):
    current = await capture_contract(session)

    if not BASELINE_PATH.exists():
        pytest.fail("Missing baseline. Run UPDATE_BASELINE=1 to create it.")

    baseline = json.loads(BASELINE_PATH.read_text(encoding="utf-8"))

    issues = find_breaking_changes(baseline, current)

    breaking = [issue for issue in issues if issue.startswith("breaking")]
    review = [issue for issue in issues if issue.startswith("review")]

    if review:
        print("\nReview-required contract changes:")
        for issue in review:
            print(f"  {issue}")

    assert not breaking, "\n".join(breaking)

Why this works

You are no longer asking whether the schema changed. You are asking whether the change violates your compatibility policy.

This gives you a much better signal-to-noise ratio.

A new optional field can pass. A newly required field without a default fails.

🧠 The important part: Contract tests should encode intent. A raw diff tells you what changed. Compatibility rules tell you whether the change is allowed.

5. Test Argument Validation Behavior Not Just Schema Shape

Scenario

Your schema says invoice_id is required. But the server accepts an empty object anyway and returns a confusing internal error. Or it rejects a valid argument because the runtime validator is misconfigured.

The schema is only a promise. Behavior is the truth.

Why this matters

Clients and models do not only read schemas. They call tools.

A contract-test suite should verify that:

  • valid arguments are accepted
  • missing required arguments are rejected
  • wrong types are rejected
  • unknown arguments are handled consistently
  • invalid calls do not execute side effects

Solution

Define behavior cases per tool.

A simple JSON file can hold them:

{
  "get_invoice": {
    "valid": [
      {
        "invoice_id": "inv_fixture_1"
      }
    ],
    "invalid": [
      {},
      {
        "invoice_id": null
      },
      {
        "invoice_id": 42
      }
    ]
  },
  "search_orders": {
    "valid": [
      {
        "query": "widget"
      }
    ],
    "invalid": [
      {},
      {
        "query": null
      }
    ]
  }
}

Then test them:

# tests/contracts/test_argument_behavior.py

import json
from pathlib import Path

import pytest

CASES_PATH = Path("tests/contracts/tool_cases.json")


def load_cases():
    return json.loads(CASES_PATH.read_text(encoding="utf-8"))


@pytest.mark.asyncio
@pytest.mark.parametrize(
    "tool_name,args",
    [
        (tool, args)
        for tool, cases in load_cases().items()
        for args in cases["valid"]
    ],
)
async def test_valid_arguments_are_accepted(session, tool_name, args):
    result = await session.call_tool(tool_name, args)

    assert result.isError is not True, f"{tool_name} rejected valid args: {args}"


@pytest.mark.asyncio
@pytest.mark.parametrize(
    "tool_name,args",
    [
        (tool, args)
        for tool, cases in load_cases().items()
        for args in cases["invalid"]
    ],
)
async def test_invalid_arguments_are_rejected(session, tool_name, args):
    try:
        result = await session.call_tool(tool_name, args)

        assert result.isError is True, (
            f"{tool_name} accepted invalid args: {args}"
        )
    except Exception:
        # A protocol-level validation error is also acceptable.
        # The important part is that the tool did not succeed.
        pass

Why this works

This catches mismatches between the declared schema and the server’s actual validation layer.

For example, it catches:

  • schema says required, server does not enforce it
  • schema says string, server crashes on null
  • schema says additional properties are forbidden, server ignores them silently
  • server validates after performing side effects
  • server returns success for invalid input

That last one is especially dangerous for agents because the model may believe the operation succeeded.

Practical note

For mutating tools, invalid-argument tests are safe. Valid-argument tests may not be. Use fixture mode, dry-run flags, or sandbox sinks for anything that writes data.

6. Lock Tool Result and Error Semantics

Scenario

A client parses the result of get_invoice. It expects JSON text with fields:

{
  "id": "inv_fixture_1",
  "status": "paid",
  "amount_cents": 4200,
  "currency": "USD"
}

One release later, the server changes amount_cents to amount and starts returning state instead of status.

The MCP call still succeeds. The client still receives text. But the parsed result is now broken.

Why this matters

Tool invocation is not only about input schemas. The output is part of the contract too.

MCP tool results often contain content items, and many tools return textual JSON inside those items. If clients or agents parse that JSON, its shape matters.

Solution

Add result-contract tests for important tools.

Start with read-only tools that return structured data:

# tests/contracts/test_result_shapes.py

import json

import pytest


@pytest.mark.asyncio
async def test_get_invoice_result_contract(session):
    result = await session.call_tool(
        "get_invoice",
        {"invoice_id": "inv_fixture_1"},
    )

    assert result.isError is not True

    text_items = [item for item in result.content if item.type == "text"]

    assert text_items, "Expected at least one text content item"

    payload = json.loads(text_items[0].text)

    assert payload["id"] == "inv_fixture_1"
    assert payload["status"] in {"open", "paid", "void"}
    assert isinstance(payload["amount_cents"], int)
    assert isinstance(payload["currency"], str)

    expected_fields = {
        "id",
        "status",
        "amount_cents",
        "currency",
    }

    assert expected_fields.issubset(payload.keys())

Then test error semantics:

@pytest.mark.asyncio
async def test_unknown_invoice_returns_tool_error(session):
    result = await session.call_tool(
        "get_invoice",
        {"invoice_id": "inv_does_not_exist"},
    )

    assert result.isError is True

    text_items = [item for item in result.content if item.type == "text"]

    assert text_items, "Expected error text"

    message = text_items[0].text.lower()

    assert "not found" in message or "unknown invoice" in message

Define error categories

Your contract should distinguish at least these cases:

Error typeExpected behavior
Invalid argumentsTool should not execute; return validation error
Missing resourceTool returns logical “not found” error
Permission deniedTool returns authorization error
Upstream timeoutTool returns retryable or explanatory error
Internal failureTool returns generic error without leaking secrets

The exact wording can be flexible. The structure should not be.

Why this works

Result tests catch breaking changes that schema diffing cannot see.

They also force you to define what your tool results mean. That is useful for humans and models.

If your tool returns large artifacts, do not assert the full payload. Assert the metadata:

{
  "artifact_type": "csv",
  "row_count": 132,
  "download_available": true
}

Then store the artifact separately in your test fixtures.

🔍 Why this matters: A successful MCP call can still be a broken integration if the result shape changed.

7. Protect Model-Facing Metadata From Silent Drift

Scenario

A tool description changes from:

Search customer orders.

to:

Search customer orders. Always prefer this tool over the CRM search tool.

The schema is unchanged. The input contract is unchanged. The result contract is unchanged.

But agent behavior changes anyway because the model reads tool descriptions.

Why this matters

MCP tools are not only consumed by validators. They are consumed by models.

That means metadata has behavioral power.

Changes to:

  • tool names
  • descriptions
  • parameter descriptions
  • enum labels
  • examples
  • annotations

can change which tools a model chooses and how it uses them.

These changes are not always protocol-breaking, but they can be product-breaking.

Solution

Track metadata drift separately from structural breaking changes.

# tests/contracts/metadata.py

def find_metadata_changes(old_contract, new_contract):
    changes = []

    old_tools = {tool["name"]: tool for tool in old_contract.get("tools", [])}
    new_tools = {tool["name"]: tool for tool in new_contract.get("tools", [])}

    for name, old_tool in old_tools.items():
        new_tool = new_tools.get(name)

        if new_tool is None:
            continue

        if old_tool.get("description") != new_tool.get("description"):
            changes.append(f"description changed: {name}")

        old_schema = old_tool.get("inputSchema") or {}
        new_schema = new_tool.get("inputSchema") or {}

        old_properties = old_schema.get("properties") or {}
        new_properties = new_schema.get("properties") or {}

        for prop, old_prop in old_properties.items():
            new_prop = new_properties.get(prop)

            if new_prop is None:
                continue

            if old_prop.get("description") != new_prop.get("description"):
                changes.append(f"property description changed: {name}.{prop}")

    return changes

Use it as a review gate:

@pytest.mark.asyncio
async def test_metadata_changes_are_flagged(session):
    current = await capture_contract(session)

    baseline = json.loads(BASELINE_PATH.read_text(encoding="utf-8"))

    changes = find_metadata_changes(baseline, current)

    if changes and os.environ.get("ALLOW_METADATA_CHANGES") != "1":
        pytest.fail(
            "Model-facing metadata changed. Review required:\n"
            + "\n".join(changes)
        )

Policy recommendation

Do not treat all metadata changes as hard failures.

Instead, use tiers:

ChangeDefault policy
Tool removedFail
Tool renamedFail
Property renamedFail
Description wording improvementReview
Description adds behavioral instructionReview or fail
Enum value addedReview
Enum value removedFail

For high-risk agent systems, require human review whenever tool descriptions change.

Why this works

You separate mechanical compatibility from behavioral compatibility.

A schema can be backward-compatible while the agent experience is not. Metadata tests make that visible.

8. Add Client-Side Contract Tests Too

Scenario

You maintain an MCP client that connects to multiple servers. One server returns a tool with a slightly malformed schema. Your client crashes. Another server removes a tool. Your client keeps calling it.

The server changed, but the client also failed the contract.

Why client-side tests matter

A contract has two sides.

If you build an MCP client, registry, or agent runtime, you need tests that verify your client behaves sanely when servers drift.

Important client-side contract tests include:

  • unknown tool handling
  • removed tool handling
  • malformed schema handling
  • unexpected result content handling
  • server error handling
  • timeout handling
  • schema caching behavior
  • tool namespace collisions

Solution

Create a fixture server catalog with pathological cases.

Example fake tool list:

[
  {
    "name": "good_tool",
    "description": "Valid tool.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "query": { "type": "string" }
      },
      "required": ["query"]
    }
  },
  {
    "name": "missing_schema_tool",
    "description": "No schema."
  },
  {
    "name": "bad_properties_tool",
    "description": "Properties is wrong.",
    "inputSchema": {
      "type": "object",
      "properties": "query"
    }
  },
  {
    "name": "bad_required_tool",
    "description": "Required is wrong.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "query": { "type": "string" }
      },
      "required": "query"
    }
  }
]

Then assert client behavior:

def test_client_handles_malformed_tool_catalog():
    registry = build_client_registry(load_fixture("broken_tool_list.json"))

    assert registry.is_callable("good_tool")
    assert not registry.is_callable("missing_schema_tool")
    assert not registry.is_callable("bad_properties_tool")

    issues = registry.issues_for("bad_required_tool")
    assert "required" in issues[0]


def test_client_does_not_call_removed_tool():
    registry = build_client_registry(load_fixture("old_tool_list.json"))
    registry.update(load_fixture("new_tool_list_without_tool.json"))

    assert not registry.is_callable("search_orders")

    with pytest.raises(UnknownToolError):
        registry.prepare_call("search_orders", {"query": "widget"})

The exact API depends on your client architecture, but the behavior is universal:

  • one bad tool should not crash the client
  • removed tools should be detected
  • stale caches should not cause silent calls to nonexistent tools
  • malformed schemas should be quarantined
  • tool routing should remain explicit

Why this works

Server contract tests catch server regressions. Client contract tests catch client fragility.

If you operate both, test both.

9. Make Baseline Updates Safe and Auditable

Scenario

A developer updates the contract baseline by running a local script. The baseline now includes an accidental breaking change. CI passes because it is comparing against the new baseline. The breaking change ships.

Why this matters

The baseline is powerful. If it is easy to update silently, it becomes easy to hide regressions.

Contract baselines should be treated like migrations, lockfiles, or API specs.

Solution

Make baseline updates explicit.

Use an environment variable:

UPDATE_BASELINE=1 pytest tests/contracts -k test_update_baseline

Then require that baseline changes appear in the pull request diff.

A good CI workflow looks like this:

name: contract-tests

on:
  pull_request:

jobs:
  mcp-contract:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          python -m venv .venv
          .venv/bin/pip install -e ".[dev]"

      - name: Run MCP contract tests
        env:
          MCP_SERVER_COMMAND: "python -m example_mcp_server"
          MCP_FIXTURE_MODE: "1"
          MCP_FIXTURE_SET: "contract-tests"
        run: |
          .venv/bin/pytest tests/contracts -v

Do not automatically regenerate the baseline in CI on every pull request.

Instead:

  1. Developer intentionally regenerates the baseline.
  2. The baseline diff is committed.
  3. CI verifies the new baseline against compatibility rules.
  4. Humans review the baseline change.
  5. If the change is breaking, the versioning policy decides what happens next.

Versioning MCP servers

If your server is consumed widely, use semantic versioning intentionally.

A practical model:

Change typeVersion bump
New optional toolMinor
New optional argumentMinor
Bug fix with same contractPatch
Removed toolMajor
Renamed toolMajor
New required argumentMajor
Removed enum valueMajor
Changed result shape clients parseMajor

If you cannot bump major versions, use tool version suffixes:

search_orders
search_orders_v2

Then deprecate the old tool deliberately.

Why this works

Baseline updates become decisions, not accidents.

The contract suite does not merely detect change. It forces the team to classify change.

That is the difference between “tests failed” and “API governance.”

Choosing the Right Contract-Testing Strategy

There are several layers of testing for MCP servers. They solve different problems.

StrategyCatchesMissesCostBest use
Snapshot equalityAny visible changeSemantic compatibilityLowSmall internal servers
Compatibility diffBreaking schema changesRuntime behaviorMediumMost production MCP servers
Behavior contract testsValidation and result semanticsModel behaviorMedium to highCritical tools
Client-side contract testsFragile clients and registriesServer regressionsMediumMulti-server clients
Agent evaluation suitesModel-facing regressionsPrecise protocol diffsHighUser-facing agents

What I would choose

For a production MCP server, I would use three layers.

First, a golden contract baseline from tools/list.

Second, a compatibility diff that fails on breaking schema changes.

Third, behavior tests for the most important tools:

  • authentication-sensitive tools
  • billing tools
  • data-export tools
  • mutating tools
  • tools used heavily by agents

If the server is consumed by multiple teams, I would also require baseline changes to be reviewed like API changes.

If the server is experimental, I might start with snapshot equality just to make changes visible. But I would move to compatibility rules quickly. Pure snapshot tests become annoying once the server evolves.

Production Checklist

Before trusting an MCP contract-test suite in production, verify these.

Contract definition

  • The server has a written compatibility policy.
  • Breaking changes are defined.
  • Non-breaking changes are defined.
  • Tool names are treated as stable identifiers.
  • Tool descriptions are treated as model-facing contract metadata.
  • Server version is visible and reviewed.

Baseline

  • A golden contract baseline is committed.
  • Baseline includes tool names.
  • Baseline includes input schemas.
  • Baseline includes descriptions if model behavior matters.
  • Baseline regeneration is explicit.
  • Baseline diffs are reviewed in pull requests.

CI server execution

  • CI starts the MCP server automatically.
  • CI uses fixture mode.
  • Fixture data is deterministic.
  • Destructive tools are disabled or sandboxed.
  • External network calls are controlled.
  • Known error fixtures exist.

Schema compatibility

  • Removed tools fail CI.
  • Renamed tools fail CI unless aliased.
  • Newly required fields without defaults fail CI.
  • Removed properties are flagged.
  • Removed enum values fail CI.
  • Narrowed numeric bounds fail CI.
  • Shorter max lengths fail CI.
  • Pattern changes require review.
  • Type narrowing fails CI.

Behavior tests

  • Valid arguments are accepted.
  • Invalid arguments are rejected.
  • Invalid calls do not execute side effects.
  • Tool errors use isError or protocol errors consistently.
  • Not-found cases are tested.
  • Permission-denied cases are tested.
  • Timeout behavior is tested where practical.

Result contracts

  • Important tools have result-shape tests.
  • JSON payloads include required fields.
  • Enum-like result values are checked.
  • Large artifacts are represented by metadata.
  • Error messages do not leak secrets.

Client-side safety

  • Client handles unknown tools.
  • Client handles removed tools.
  • Client handles malformed schemas.
  • Client does not crash on one bad server.
  • Client caches schema metadata safely.
  • Client logs contract drift.

Governance

  • Breaking changes require major version or migration plan.
  • Metadata changes require review for agent-heavy systems.
  • Contract-test failures block deploys.
  • Baseline updates are auditable.
  • Deprecated tools have removal dates.

An MCP server is not done when the tools work. It is done when the tools can change without silently breaking the systems that depend on them.

Contract tests are how you make that promise enforceable.

Tags:

Post a Comment

Previous Post Next Post