Build a RAG System That Understands OpenAPI Specs and Generates Verified API Calls

The generated request passed JSON Schema validation. It still created the wrong order.

The model selected POST /orders, filled every required field, and produced a payload that was structurally perfect. But it set status to "active" when the only permitted values were "pending" and "draft", and it sent customer_id as an integer when the API expected a UUID string. The schema validator saw types and required fields. It didn't see meaning.

That's the gap between retrieving OpenAPI documentation and generating verified API calls. A RAG system that finds the right endpoint but lets the model invent argument values is only half a system. The retrieval problem is real—OpenAPI specs are deeply context-dependent, and naive chunking destroys that context. But the generation problem is equally real: the model must be constrained to produce calls that are not just well-formed, but semantically faithful to the contract.

This is what it takes to close both gaps.

📋 Table of Contents

• Why Naive RAG Fails on OpenAPI Specs
• 1. Chunk by Endpoint, Not by Token
• 2. Embed Each Endpoint Multiple Times
• 3. Serialize for the Model, Not for the Spec
• 4. Generate Structured Tool Calls, Not Free-Text Code
• 5. Constrained Decoding When You Need Executable Code
• 6. The Four-Gate Execution Loop
• 7. Handle 3.0, 3.1, and 3.2 Without Lying to the Model
• 8. Evaluate Endpoint Retrieval and Argument Fidelity Separately
• What I'd Build and What I'd Skip

Why Naive RAG Fails on OpenAPI Specs

OpenAPI documents look like structured data, so it's tempting to treat them like any other JSON or YAML file. Split by character count, embed, retrieve, done. That approach fails for a specific structural reason: every piece of an OpenAPI operation is meaningless without the operation it belongs to.

A parameter description like “The customer's unique identifier” is useless if the retrieved chunk doesn't make clear that it belongs to POST /orders rather than GET /customers/{id}. A security scheme defined at the document root applies to the endpoint, but a naive chunker leaves it behind. A $ref to components/schemas/OrderStatus might resolve to an enum, but if the chunk only contains the reference pointer, the model sees $ref: '#/components/schemas/OrderStatus' and has no idea what values are legal.

Research on RAG for service discovery has been consistent about this. Endpoint-based chunking outperforms naive splitting for OpenAPI preprocessing, because an endpoint is the smallest self-contained unit of API meaning. The same body of work found that format-specific preprocessing—not generic text splitting—preserves the information the model needs to select and call the right operation.

The second failure is equally important. Even when retrieval returns the correct endpoint, the model can generate argument values that pass type checks but violate the API's semantic contract. A 2026 benchmark for web API integration code generation, WAPIIBench, was built precisely because generating API invocations produces hallucinated endpoints, incorrect argument usage, and other errors at rates that matter in production. Schema validation alone doesn't catch an enum violation if the retrieved schema was truncated. It doesn't catch a semantically wrong ID if the type is still string.

The rest of this article is a pipeline that addresses both failures, in order: retrieval that preserves endpoint context, serialization that keeps the model's input compact, generation constrained by the contract, and an execution loop that refuses to send anything it hasn't verified.

🔍 Why this matters: An API call that returns a 400 is annoying. An API call that succeeds but does the wrong thing—creates a duplicate, charges the wrong amount, sends data to the wrong tenant—is a production incident. The verification layer exists to make the second failure mode structurally impossible, not just less likely.

1. Chunk by Endpoint, Not by Token

Scenario: You index a spec with 120 operations. A user asks, “How do I cancel an order?” Naive retrieval returns a chunk containing the cancel path parameter and half of the description from DELETE /orders/{id}, plus a fragment of a completely unrelated schema definition. The model sees enough to guess. It doesn't see the required reason field or the fact that cancellation is only allowed for orders in processing status.

Why the obvious solution fails: Token-based splitting has no concept of an operation. It splits at character boundaries, which means it splits through the middle of parameter lists, across $ref boundaries, and between an endpoint and its security requirements.

Solution: Parse the spec, dereference internal references, and make each operation a single chunk. Every chunk must be self-contained: the HTTP method, the path, all parameters (path, query, header, cookie), the request body schema, the response schemas, the security requirements, and any operation-level metadata like deprecated or operationId.

The most important preprocessing step is inlining $ref values before chunking. If a parameter references components.parameters.OrderId, resolve it into the chunk. If a schema references components.schemas.OrderStatus, resolve the enum into the chunk. If a security scheme is defined at the document root, copy the applicable scheme into every operation that uses it. This is what the WAPIIBench methodology does: it preprocesses OpenAPI specifications before chunking, inlining all references and copying API-wide security schemes and path-level parameters into each endpoint's parameter list.

Here's a Python sketch using the openapi-core and jsonref libraries to produce self-contained endpoint chunks:

import jsonref
from openapi_core import Spec

def build_endpoint_chunks(spec_path: str) -> list[dict]:
    with open(spec_path) as f:
        raw = json.load(f)

    # Resolve every internal $ref into the document before chunking.
    # This is the step that prevents chunks from containing unresolved pointers.
    resolved = jsonref.replace_refs(raw, proxies=False)

    spec = Spec.from_dict(resolved)
    chunks = []

    for path, path_item in resolved.get("paths", {}).items():
        for method, operation in path_item.items():
            if method not in {"get", "post", "put", "patch", "delete", "head", "options"}:
                continue

            # Path-level parameters apply to every operation under the path.
            path_params = path_item.get("parameters", [])
            op_params = operation.get("parameters", [])
            merged_params = _merge_parameters(path_params, op_params)

            chunks.append({
                "operation_id": operation.get("operationId", f"{method}_{path}"),
                "method": method.upper(),
                "path": path,
                "summary": operation.get("summary", ""),
                "description": operation.get("description", ""),
                "parameters": merged_params,
                "request_body": operation.get("requestBody", {}),
                "responses": operation.get("responses", {}),
                "security": operation.get("security", resolved.get("security", [])),
                "tags": operation.get("tags", []),
                "deprecated": operation.get("deprecated", False),
            })

    return chunks

def _merge_parameters(path_params, op_params):
    by_name = {(p["name"], p["in"]): p for p in path_params}
    by_name.update({(p["name"], p["in"]): p for p in op_params})
    return list(by_name.values())

Why this works: The chunk boundary is the operation, which is the smallest unit that can be invoked independently. Every parameter carries its location (in: query, in: path, etc.), so the model doesn't have to guess whether an ID belongs in the URL or the query string. Security schemes are attached to the operation that requires them. And because references are resolved, the model sees the actual enum values, not a pointer to a schema it cannot access.

One subtlety: operationId is not guaranteed to be present. Fall back to method + path when it's missing, and use the summary field as the primary natural-language label. Summaries are often the only human-written description of what an endpoint does, and they're crucial for retrieval.

💡 Practical note: Do not strip deprecated operations during chunking. Include them but mark them explicitly. If a user asks about a deprecated endpoint, the model should know it exists and should avoid recommending it—not be forced to hallucinate an alternative because the documentation was silently removed.

2. Embed Each Endpoint Multiple Times

Scenario: A user asks, “How do I filter orders by status?” The endpoint chunk for GET /orders contains a long description, a list of ten query parameters, a response schema with nested objects, and error responses. A single embedding of this entire chunk is dominated by the response schema, which is large but irrelevant to the question. The chunk ranks poorly, and retrieval returns a different endpoint that happens to mention “status” in its description.

Why single embeddings fail: An OpenAPI operation is not a coherent piece of prose. It's a bundle of different information types—path, description, parameters, request body, responses—each of which answers a different kind of question. Compressing them into one vector means the largest section dominates the embedding.

Solution: Generate multiple embeddings per endpoint, each covering a different view of the operation. The WAPIIBench retriever stores up to five embeddings per chunk, created by including and excluding different parts of the endpoint documentation: one variant contains only the path and description, another adds the parameter list, another adds the response schema, and so on. At query time, the system retrieves across all variants and deduplicates by endpoint. This approach reports 75.7% top-1 accuracy and 95.2% top-5 accuracy across their task set.

Here's the embedding strategy in practice:

def endpoint_views(chunk: dict) -> list[str]:
    base = f"{chunk['method']} {chunk['path']}\n{chunk['summary']}\n{chunk['description']}"

    views = [base]

    if chunk["parameters"]:
        views.append(
            base + "\n\nParameters:\n" + "\n".join(
                f"- {p['name']} ({p['in']}): {p.get('description', '')}"
                for p in chunk["parameters"]
            )
        )

    if chunk["request_body"]:
        views.append(base + "\n\nRequest body schema:\n" + _compact_schema(chunk["request_body"]))

    if chunk["responses"]:
        views.append(base + "\n\nResponses:\n" + _compact_schema(chunk["responses"]))

    return views

Store each view as a separate row in the vector index, with the same endpoint_id but a different view_type. At retrieval time, search all views, group by endpoint_id, and take the maximum similarity score per endpoint. This prevents one large, irrelevant section from dragging down the entire endpoint's relevance.

A hybrid lexical index should operate on the operation-level text—operationId, path, summary, parameter names, and enum values—not on the individual views. Lexical search is where exact identifier matches happen, and those identifiers belong to the endpoint as a whole. Splitting the lexical index across views dilutes exact-match signal.

🧠 The important part: Multiple embeddings are not a substitute for good chunking. If the endpoint chunk is incomplete—if it's missing a parameter or a security scheme—multiple views of an incomplete chunk just produce multiple incomplete retrievals. Inline first, then embed.

3. Serialize for the Model, Not for the Spec

Scenario: You retrieve the correct endpoint, and now you need to put it in the prompt. You paste the OpenAPI YAML fragment. It's 180 lines, full of indentation, description: | block scalars, and verbose constructs like name: order_id where the model only needs order_id. The prompt consumes 4,000 tokens for one endpoint. You have room for two endpoints at most.

Why YAML and JSON are token-inefficient for this: OpenAPI is designed for machines and documentation generators, not for LLM consumption. YAML's indentation and key repetition are expensive. JSON's braces and quotes add tokens without adding meaning. A parameter that needs one line—order_id: string (path, required)—takes five lines in YAML.

Solution: Design a compact serialization format specifically for the model. The WAPIIBench authors explored JSON and YAML but found them token-inefficient, so they designed a custom format inspired by TypeScript type declarations. The goal is not to be valid TypeScript. The goal is to express the same contract in fewer tokens while remaining unambiguous.

Here's what that looks like in practice:

POST /orders
Summary: Create a new order
Auth: bearer

Body (application/json):
{
  customer_id: string (uuid, required)
  items: Array<{
    sku: string (required)
    quantity: integer (min: 1, required)
  }> (required)
  status?: "pending" | "draft"
  notes?: string
}

Responses:
  201: Order { id: string (uuid), status: "pending" | "draft", created_at: string (date-time) }
  400: Error { code: string, message: string }
  409: Error { code: "DUPLICATE_ORDER", message: string }

This format does several things at once. It uses ? for optional fields, | for enums, and <> for arrays, borrowing TypeScript conventions that models have seen extensively in training. It collapses required into a suffix rather than a separate JSON array. It flattens response schemas into one line per response. The result is typically 2–3× more compact than equivalent YAML, which directly increases the number of endpoints you can fit in the context window.

For a production system, generate this format from the resolved endpoint chunk, not by hand. A serializer that walks the schema and emits the compact syntax is more maintainable than prompting the model to “read YAML” and hoping it ignores the irrelevant parts.

⚠️ Gotcha: Compaction must not drop semantic constraints. If the original schema has minLength, pattern, minimum, or maxItems, those constraints need to survive compaction—otherwise the model has no way to know them, and the validator becomes the only line of defense. Include numeric and string constraints inline, even if it makes the format slightly less compact.

4. Generate Structured Tool Calls, Not Free-Text Code

Scenario: The model is asked to “write code that creates an order.” It produces an Axios call with a URL, a method, a headers object, and a body. The body is almost correct, but the model uses customerId instead of customer_id, and it includes a status field that wasn't in the schema. The code looks plausible. It fails at runtime.

Why free-text code generation is the wrong default: When the model writes code, it controls syntax, structure, and semantics all at once. Each of those is a separate failure surface. A missing parenthesis, a wrong parameter name, and a semantically invalid enum value are all “code generation errors,” but they require completely different fixes.

Solution: When the goal is to execute an API call—not to produce a code snippet for a human to review—generate a structured tool call instead of free-text code. Define each operation as a tool with a JSON Schema input, and let the model emit arguments against that schema. The application validates the arguments, executes the request, and returns the result.

The langchain-openapi-tools library does exactly this. It converts OpenAPI 3.0, 3.1, and Swagger 2.0 specifications into LangChain structured tools with Pydantic input schemas, so the model emits arguments that are validated before any network transport. The Pydantic schemas enforce types, required fields, and enums before the request leaves the application.

Here's the core pattern without a framework, using Pydantic to mirror the OpenAPI schema:

from enum import Enum
from pydantic import BaseModel, Field
from uuid import UUID

class OrderStatus(str, Enum):
    pending = "pending"
    draft = "draft"

class OrderItem(BaseModel):
    sku: str
    quantity: int = Field(ge=1)

class CreateOrderArgs(BaseModel):
    customer_id: UUID
    items: list[OrderItem] = Field(min_length=1)
    status: OrderStatus = OrderStatus.pending
    notes: str | None = None

def dispatch_create_order(raw_args: dict, tenant_id: str):
    try:
        args = CreateOrderArgs(**raw_args)
    except Exception as e:
        return {"error": "invalid_arguments", "detail": str(e)}

    # Authorization is separate from schema validation.
    if not can_create_order(tenant_id, args.customer_id):
        return {"error": "unauthorized"}

    return execute_create_order(args)

Why this works: The model never emits a URL, an HTTP method, or a headers object. It selects a tool by name and fills arguments against a schema. The application owns the transport, the base URL, the authentication headers, and the retry policy. The only thing the model controls is the operation selection and the argument values—and both of those are validated before execution.

This also fixes the “arbitrary URL” problem. If the model can generate a raw HTTP request, it can be prompt-injected into sending data to an attacker-controlled endpoint. If it can only select from a fixed catalog of tools, the set of reachable destinations is bounded by the catalog.

🚨 Production warning: Do not expose a generic http_request tool that accepts arbitrary URLs, methods, and headers when a narrow operation will do. A generic HTTP tool expands the attack surface to every reachable host and every method the runtime supports. Define tools at the operation level, not the protocol level.

5. Constrained Decoding When You Need Executable Code

Scenario: You need the model to generate a runnable Axios snippet, not just a tool call—perhaps for a code assistant, a test generator, or a migration tool. The model produces code that is syntactically valid JavaScript and calls the right endpoint, but it passes a query parameter in the request body because it confused GET and POST conventions. Schema validation happens after generation, so you catch it, but you've already spent the tokens.

Why post-hoc validation isn't enough for code generation: Validation catches errors, but it doesn't prevent them. In a retry loop, the model can produce the same class of error repeatedly. Constrained decoding prevents invalid tokens from being generated in the first place, which reduces both latency and cost.

Solution: Translate the OpenAPI operation into a set of regex-based generation rules and enforce them during decoding. The WAPIIBench authors implemented this: constraints are automatically derived from OpenAPI specifications and represented as regular expressions, organized as generation rules with a start condition, stop condition, and body. The constraints are hierarchical—first the HTTP method, then the URL, then the parameters, then the value types—and they use capture groups and conditionals to prevent duplicate parameters and enforce required ones.

A simplified constraint for a single endpoint might look like this:

import re

# Conceptual constraint for POST /orders with a required body.
# Real implementations compose these hierarchically across endpoints.
METHOD_PATTERN = r"axios\.post\("
URL_PATTERN = r"['\"]https://api\.example\.com/orders['\"]"
BODY_PATTERN = r",\s*\{.*?customer_id\s*:\s*.*?items\s*:\s*\[.*?\].*?\}"

CONSTRAINT = re.compile(
    METHOD_PATTERN + URL_PATTERN + BODY_PATTERN,
    re.DOTALL,
)

In practice, you don't hand-write these regexes. You generate them from the OpenAPI spec at build time. The spec tells you which methods are valid for a path, which parameters are required, and which data types each parameter accepts. The regex generator translates that into a grammar that the decoding engine enforces.

For teams using hosted models rather than self-hosted inference, constrained decoding is less directly available. OpenAI's structured outputs and strict tool calling, and Anthropic's strict tool use, compile a JSON Schema into a grammar and guarantee that generated arguments conform to it. That handles the arguments but not arbitrary code syntax. If you need executable code with guaranteed syntax, self-hosted constrained decoding with Outlines, XGrammar, or a serving engine that supports guided decoding is the practical path.

Approach	Guarantees	Best for	Trade-off
Strict JSON Schema / structured outputs	Arguments conform to schema	Tool calls, function calling	Does not guarantee code syntax
Regex-based constrained decoding	Code matches the derived grammar	Executable snippets, migrations	Requires self-hosted or guided decoding support
Post-hoc validation + retry	Invalid calls are rejected	Any provider, any model	Higher latency and token cost on failures
Pydantic / JSON Schema validation only	Shape and types are correct	Baseline defense	Does not catch semantic errors

🔍 Why this matters: Constrained decoding doesn't make the model smarter. It makes the output space smaller. A model that would have generated an invalid call is forced to generate a valid one—or fail to generate at all. Both outcomes are better than a confident, well-formed, wrong request.

6. The Four-Gate Execution Loop

Scenario: The model emits a tool call. It passes Pydantic validation. The types are correct, the required fields are present, and the enum values are legal. But the customer_id belongs to a different tenant. Or the operation is a write that requires a human approval step. Or the endpoint is idempotent and the model has already called it once in this conversation.

Why schema validation is not enough: Schema validation answers one question: does this argument object match the shape the tool declared? It does not answer whether the caller is allowed to perform this operation, whether the resource belongs to the caller, whether the side effect is safe, or whether this exact call has already happened.

The production execution loop has four gates, in order:

Gate 1 — Schema validation. Reject unknown tools, missing required fields, wrong types, and values outside enums. Use the same schema that was shown to the model, so the prompt and the validator can't drift apart. Log every rejection with the raw arguments and the validation error—that log is the training set for future evaluations.

Gate 2 — Semantic validation. Check constraints that JSON Schema can't express. Does the customer_id exist? Is the order in a state that permits cancellation? Does the quantity exceed the account's limit? This gate is where API-specific business logic lives. Pydantic custom validators or a separate validation function can handle it.

Gate 3 — Authorization and policy. Bind the action to the authenticated user and tenant. The Dasha tool-calling guide puts it plainly: a model proposes an action; application code decides whether it may execute, and a valid tool call does not establish user identity, permission, business correctness, or successful execution. For write operations, require an explicit approval bound to the normalized arguments. If the arguments change, the approval is invalid.

Gate 4 — Dry run or sandbox. For high-risk writes, execute the call in a dry-run mode first, or run it against a sandbox environment. The model sees the preview and the application checks for unintended side effects. Only after the preview is approved does the real request leave the system.

def execute_tool_call(tool_name: str, raw_args: dict, context: dict):
    tool = TOOL_REGISTRY.get(tool_name)
    if not tool:
        return {"state": "rejected", "code": "UNKNOWN_TOOL"}

    # Gate 1: shape and type validation.
    try:
        args = tool.input_model(**raw_args)
    except ValidationError as e:
        log_rejection(tool_name, raw_args, e)
        return {"state": "rejected", "code": "INVALID_ARGUMENTS", "detail": e.errors()}

    # Gate 2: semantic validation.
    semantic_error = tool.semantic_check(args, context)
    if semantic_error:
        return {"state": "rejected", "code": semantic_error}

    # Gate 3: authorization.
    if not tool.authorize(args, context["tenant_id"], context["user_id"]):
        return {"state": "rejected", "code": "UNAUTHORIZED"}

    # Gate 4: dry run for consequential writes.
    if tool.requires_dry_run:
        preview = tool.dry_run(args, context)
        if not preview["safe"]:
            return {"state": "preview_failed", "detail": preview["reason"]}

    return tool.execute(args, context)

The loop also needs a budget: maximum tool calls per conversation, maximum cost, maximum wall-clock time. A model that gets stuck in a retry cycle after repeated validation failures should be stopped, not allowed to burn tokens indefinitely.

💡 Practical note: Return structured errors to the model, not stack traces. {"error": "invalid_arguments", "field": "status", "allowed": ["pending", "draft"]} gives the model enough information to correct itself. A raw Pydantic traceback does not.

7. Handle 3.0, 3.1, and 3.2 Without Lying to the Model

Scenario: You build the pipeline against OpenAPI 3.0, then a user points it at a 3.1 spec. The schema for a nullable field says type: ["string", "null"]. Your serializer, which only knows about type: string and nullable: true, emits nothing useful. The model sees a field with no type and guesses. The generated call sends a number where a string or null was expected.

Why version differences matter: OpenAPI 3.1 aligned the Schema Object with JSON Schema Draft 2020-12. The nullable keyword was removed in favor of type arrays like ["string", "null"]. OpenAPI 3.2, finalized in 2026, builds on 3.1 with first-class support for streaming media types like text/event-stream and application/jsonl, structured tag hierarchies, and additional HTTP methods beyond the standard set. A pipeline that treats all specs as 3.0 will silently drop or misinterpret these constructs.

Solution: Normalize before you serialize. Parse the spec version, and run a version-aware normalizer that converts 3.0 constructs into a canonical internal representation aligned with 3.1's JSON Schema semantics. Then serialize the canonical representation for the model.

def normalize_schema(schema: dict, spec_version: str) -> dict:
    """Convert version-specific schema constructs to canonical JSON Schema."""
    if "type" in schema and isinstance(schema["type"], str):
        if spec_version.startswith("3.0") and schema.get("nullable"):
            schema["type"] = [schema["type"], "null"]
            schema.pop("nullable", None)
        elif spec_version.startswith("3.0"):
            schema["type"] = [schema["type"]]
    for key in ("items", "additionalProperties"):
        if isinstance(schema.get(key), dict):
            schema[key] = normalize_schema(schema[key], spec_version)
    if "properties" in schema:
        schema["properties"] = {
            k: normalize_schema(v, spec_version)
            for k, v in schema["properties"].items()
        }
    return schema

For 3.2 features, the normalizer needs two additional decisions. Streaming responses should not be represented as a single response schema; the model needs to know that the endpoint returns a sequence of typed items, not one object. The 3.2 itemSchema field describes the shape of each event. If your serializer collapses that into a generic object, the model has no way to generate a correct streaming call. Include the item schema explicitly, with a marker that the response is a stream.

For the new HTTP methods and additionalOperations, the tool catalog must include them. If your tool generator only recognizes get, post, put, patch, and delete, a QUERY operation will be invisible to the model, and the model will either fail to answer or hallucinate a POST endpoint that doesn't exist.

⚠️ Gotcha: $ref resolution is not the same across versions. OpenAPI 3.1 uses JSON Schema 2020-12, which supports $dynamicRef and $dynamicAnchor. Many OpenAPI parsers still don't fully support these keywords. If your resolver silently leaves a $dynamicRef unresolved, the model sees a pointer instead of a schema. Test your resolver against real 3.1 specs, not just toy examples.

8. Evaluate Endpoint Retrieval and Argument Fidelity Separately

Scenario: The system answers most questions correctly in manual testing. Then a user asks for an operation that exists but is rarely used, and retrieval returns a similar but wrong endpoint. The model confidently generates a call to the wrong operation. Because the endpoint exists and the schema is valid, the call executes. The user gets a plausible but incorrect result.

Why end-to-end evaluation hides failures: A single “correct/incorrect” metric conflates two different systems. Retrieval can fail—the right endpoint isn't in the candidate set. Generation can fail—the right endpoint is retrieved, but the arguments are wrong. If you only measure the final answer, you can't tell which stage to fix.

Solution: Evaluate retrieval and generation against separate golden sets.

For retrieval, measure endpoint-level hit rate: for each task description, is the correct operation in the top-k retrieved endpoints? Report Hit@1 and Hit@5 separately. The WAPIIBench retriever reports 75.7% top-1 and 95.2% top-5 accuracy on its endpoint chunks, which gives you a realistic range to compare against. If your top-5 hit rate is below 90%, retrieval is the bottleneck; no amount of generation tuning will fix it.

For generation, measure argument fidelity across four dimensions: operation correctness, parameter presence, parameter values, and absence of extra parameters. A generated call can be wrong in any of these dimensions independently. The WAPIIBench evaluation pipeline checks the generated invocation against a ground-truth request configuration, which allows it to attribute errors to specific arguments rather than marking the whole call as wrong.

def evaluate_call(generated: dict, expected: dict) -> dict:
    return {
        "operation_match": generated["operation_id"] == expected["operation_id"],
        "missing_params": [
            p for p in expected["required_params"]
            if p not in generated["params"]
        ],
        "extra_params": [
            p for p in generated["params"]
            if p not in expected["allowed_params"]
        ],
        "wrong_values": {
            k: {"expected": v, "generated": generated["params"].get(k)}
            for k, v in expected["params"].items()
            if generated["params"].get(k) != v
        },
    }

Run this against a JSONL golden set on every pipeline change. If endpoint hit rate drops or argument fidelity regresses, block the merge. A small set of 30–50 tasks covering the operations your users ask about most is enough to catch the regressions that manual testing misses.

The WAPIIBench dataset itself is useful as a reference, but it's built around Axios and JavaScript. If your system generates Python, Go, or tool calls, build your own task set from the API you actually integrate with. The evaluation pattern transfers; the exact tasks don't.

🧠 The important part: Retrieval quality and generation quality are not independent. A model that receives the wrong endpoint will generate wrong arguments no matter how good the prompt is. Fix retrieval first, then generation. Evaluating them together makes it impossible to know which one is broken.

What I'd Build and What I'd Skip

If I were starting this system again, here's the stack and the decisions I'd make, based on what actually holds up in production.

The stack I'd choose

Layer	Choice	Reasoning
Spec parsing	A parser that fully resolves $ref and supports 3.0, 3.1, and 3.2	Incomplete resolution is the most common source of silent retrieval failures.
Chunking	One chunk per endpoint, with path-level params and security schemes merged in	Endpoint-based chunking outperforms naive splitting for OpenAPI RAG.
Embedding	Multiple views per endpoint, deduplicated at retrieval	Single embeddings are dominated by large, irrelevant sections. Multi-view retrieval improves top-k hit rate.
Serialization	Compact TypeScript-like format, not YAML or JSON	Token efficiency directly increases the number of endpoints that fit in context.
Generation	Structured tool calls with strict JSON Schema for execution; constrained decoding only for executable snippets	Tool calls eliminate syntax errors and bound the reachable destinations. Constrained decoding is for code generation, not execution.
Validation	Pydantic or JSON Schema at the boundary, plus semantic checks and authorization	Schema validation catches shape errors; it does not catch business-logic errors or unauthorized access.
Execution	Dry-run for writes, structured errors for retries, hard budgets for calls and cost	Prevents duplicate side effects and runaway retry loops.
Evaluation	Separate retrieval hit rate and argument fidelity golden sets, gated in CI	End-to-end evaluation hides which stage is failing.

What I'd skip

Embedding the raw YAML. It's tempting because it's the source of truth. It's also the least token-efficient representation, and it forces the model to parse a format that was never designed for generation.

A single “search endpoints” tool with no schema. If the model can only search by natural language and then call a generic HTTP tool, you've replaced API documentation with a search engine and a loaded gun. Define operation-level tools with validated schemas.

Post-hoc validation as the only defense. Validation is necessary. It is not sufficient. A call that passes schema validation can still violate tenant boundaries, business rules, or idempotency requirements. The execution loop needs semantic checks and authorization, not just shape checks.

Trusting the model to produce line-level citations to spec sections. The model should cite the operation ID, not a line number in a YAML file. Resolve the operation ID to the spec location in your application, the same way you resolve a tool call to an endpoint. Model-generated coordinates drift; application-resolved coordinates don't.

The decision that matters most

The difference between an OpenAPI RAG that generates plausible calls and one that generates verified calls is not the model. It's whether the pipeline treats the spec as a contract to be enforced rather than a document to be summarized.

A contract has required fields, allowed values, version-specific semantics, and authorization boundaries. A document has prose and structure. If you retrieve the document and hope the model respects the contract, you get intermittent failures that are hard to reproduce. If you enforce the contract at every gate—retrieval, serialization, validation, authorization, execution—you get a system where a wrong call is rejected before it becomes a wrong outcome.

Build the enforcement layer first. The retrieval and generation layers are easier to tune once the system refuses to send anything it can't verify.