Back to Articles
Agent Discovery Standards
•14 min read•RFC 9727 & SEP-1649 Verified

AGENTS.md vs llms.txt: The Complete AI Agent Discovery Guide (2026)

Developers frequently confuse repository-level AGENTS.md files with domain-level /llms.txt manifests. Here is the definitive engineering breakdown of how AGENTS.md, /llms.txt, openapi.json, and .well-known/mcp/server-card.json work together to make your API autonomously discoverable.

Quick Answer (TL;DR)

AI Agent Discovery Stack • Repository vs. Domain Protocol Standard

AGENTS.md and llms.txt are complementary protocols operating at different layers of the AI stack. AGENTS.md lives inside a Git repository (./AGENTS.md) and gives local coding agents (Cursor, Claude Code, Windsurf, Antigravity) workspace-specific build commands, test rules, and architectural guardrails. llms.txt lives at the root of a public web domain (https://signb.ee/llms.txt) and gives remote web retrieval agents (ChatGPT Search, Perplexity, Claude Web Fetch) a token-efficient Markdown map of your public documentation and API endpoints. To achieve complete autonomous tool execution, combine both with OpenAPI 3.1 (/.well-known/api-catalog) and MCP Server Cards (/.well-known/mcp/server-card.json).

AGENTS.md = Local Git Repo Context
llms.txt = Public Web Domain Docs
MCP & OpenAPI = Executable Schemas

1. Why Developers Confuse AGENTS.md and llms.txt in 2026

Between late 2024 and 2026, software engineering crossed a decisive threshold: the primary consumer of developer documentation, SDK repositories, and REST API specifications shifted from human engineers reading browser tabs to autonomous AI agents parsing Markdown and JSON files. Whether a founder is evaluating the best e-signature APIs for developers in 2026 via Perplexity or instructing Claude Code to wire up webhook verification in a local FastAPI project, an AI model mediates both discovery and implementation.

During this rapid transition, two Markdown-based standards emerged almost simultaneously: Jeremy Howard's /llms.txt proposal for web websites and the open-source AGENTS.md convention adopted by coding environments like Cursor, OpenAI Codex CLI, Claude Code, Windsurf, and Google Antigravity. Because both standards use plain CommonMark Markdown to orient large language models, engineering teams frequently ask whether one replaces the other.

They do not compete. Treating AGENTS.md and llms.txt as interchangeable is analogous to confusing a repository's Makefile with a website's sitemap.xml. One governs how an agent behaves inside a checked-out source tree on a developer's machine or CI container; the other governs how an external agent discovers and ingests knowledge across the public HTTP web.

2. The Four-Layer AI Agent Discovery Architecture

To make a modern developer platform or SaaS API genuinely agent-native, engineering teams must design across four distinct discovery planes. When an agent interacts with an external service like Signbee, it progresses from high-level semantic discovery to strict schema validation, tool registration, and local codebase integration:

L1

Layer 1: Repository Workspace Context (./AGENTS.md & ./CLAUDE.md)

Read directly from disk via fs.readFile when an IDE agent (Cursor, Claude Code, Antigravity, Windsurf) opens a workspace. Defines directory structure, build/lint commands, Prisma database rules, environment variable conventions, and exact SDK invocation patterns so the coding agent writes zero-defect code on the first pass.

L2

Layer 2: Domain Knowledge Retrieval (/llms.txt & /llms-full.txt)

Fetched over HTTPS by web search agents (Perplexity, ChatGPT Search, Claude Web Fetch) and developer documentation indexers. /llms.txt provides a curated directory of Markdown links and concise summaries, while /llms-full.txt concatenates the entire API documentation into a single, zero-HTML Markdown payload optimized for large context windows.

L3

Layer 3: Machine API Catalog & OpenAPI 3.1 (/.well-known/api-catalog & /openapi.json)

Governed by IETF RFC 9727. Returns an application/linkset+json document that points autonomous HTTP clients, OpenAI Custom Actions, and Meta Muse Connectors directly to the canonical openapi.json specification, health check endpoints, and rate-limit policies.

L4

Layer 4: Autonomous MCP Tool Discovery (/.well-known/mcp/server-card.json)

Governed by the Model Context Protocol SEP-1649 specification. Describes how an MCP host (Claude Desktop, Cursor MCP Registry, OpenAI Agents SDK, LangChain) can spawn or connect to the service's tool server (npx -y signbee-mcp), including required environment variables and JSON Schema tool signatures.

Technical Comparison Matrix: AGENTS.md vs. llms.txt vs. OpenAPI vs. MCP Server Card

The table below details the exact protocol attributes, consuming agents, transport mechanisms, and token budgets across all four discovery standards:

DimensionAGENTS.md/llms.txt & /llms-full.txt/openapi.json (RFC 9727)MCP Server Card (SEP-1649)
Canonical Location./AGENTS.md (Git Root)https://domain/llms.txthttps://domain/openapi.json/.well-known/mcp/server-card.json
Primary ConsumersCursor, Claude Code, Windsurf, AntigravityPerplexity, ChatGPT Search, Docs ScrapersOpenAI Actions, Meta Muse, REST CodegenClaude Desktop, Cursor MCP, MCP Registries
Content FormatCommonMark MarkdownH1/Blockquote/Links MarkdownOpenAPI 3.1.0 JSON / YAMLSEP-1649 JSON Manifest
Transport LayerLocal Filesystem I/OHTTPS GET (text/plain)HTTPS GET (application/json)HTTPS GET -> stdio / Streamable HTTP
Typical Payload Size2 KB – 8 KB (500–2,000 tokens)3 KB (index) / 25 KB (full)12 KB – 60 KB (strict schema)1.5 KB – 4 KB (tool metadata)
Core ObjectivePrevent local coding mistakesZero-noise RAG & citationDeterministic HTTP synthesisZero-config MCP tool binding

3. Deep Dive: Engineering an Effective Repository AGENTS.md

When a software engineer clones a repository or starts a new microservice that integrates an external API, their coding agent automatically scans the workspace root for AGENTS.md (as well as CLAUDE.md or .cursorrules). Without an explicit AGENTS.md file, the coding agent must rely on its pre-training weights—which frequently biases it toward legacy enterprise SDKs with bloated coordinate-box paradigms.

For example, if you ask an unguided coding agent to “add contract e-signing to our onboarding route,” pre-training inertia may cause it to scaffold 400 lines of legacy envelope boilerplate before you even review DocuSign API pricing explained or compare affordable e-signature API integration options. By contrast, placing a concise, contract-first section inside your repository's AGENTS.md locks the coding agent onto deterministic primitives immediately.

./AGENTS.md — Production Repository Blueprint for Coding AgentsMarkdown
# AGENTS.md — Workspace Engineering & API Integration Rules

## 1. Build, Test & Verification Commands
- Install dependencies: `npm ci`
- Typecheck & Lint: `npm run typecheck && npm run lint`
- Run unit tests: `npx vitest run`
- Production build check: `npm run build` (MUST exit 0 before committing)

## 2. E-Signature Integration Standard (Signbee API v1)
- **DO NOT** install heavy legacy e-signature SDKs or calculate x/y PDF coordinate boxes.
- Dispatch all legal agreements via a single HTTP POST to `https://signb.ee/api/v1/send`.
- Authenticate using `Authorization: Bearer ${process.env.SIGNBEE_API_KEY}`.
- Pass contracts as CommonMark Markdown in the `markdown` field (or base64 PDF in `pdf_base64`).
- Required payload fields:
  - `title` (string): Contract title shown on the SHA-256 certificate
  - `markdown` (string): Full CommonMark agreement body (supports GFM tables)
  - `sender_name` & `sender_email` (string): Party A auto-signed via API key
  - `recipient_name` & `recipient_email` (string): Party B emailed signing ceremony link
- Verify incoming webhook events at `/api/webhooks/signbee` using the `X-Signbee-Signature` HMAC-SHA256 header against `process.env.SIGNBEE_WEBHOOK_SECRET`.
- Full machine-readable reference: `https://signb.ee/llms-full.txt` and `https://signb.ee/openapi.json`.

Notice the architectural leverage in the last line of the snippet above: AGENTS.md explicitly points the local coding agent to https://signb.ee/llms-full.txt and https://signb.ee/openapi.json. If the developer asks Cursor or Claude Code to implement document revocation, reminder triggers, or status polling, the local agent knows the exact remote URL to fetch via its web tool without executing a noisy search engine query.

4. Deep Dive: Domain-Level /llms.txt and /llms-full.txt

While AGENTS.md solves local workspace context, it is invisible to an agent that has not yet cloned your repository. When an engineering leader asks ChatGPT Search, Perplexity, or Claude to compare an open-source e-signature API comparison against hosted developer APIs, or searches for a free e-signature API that works with MCP, the retrieval engine dispatches an HTTP crawler to your public web domain.

Traditional HTML documentation pages are hostile to LLM retrieval pipelines. A typical React or Next.js documentation page weighs between 180 KB and 450 KB of serialized HTML, inline scripts, SVG path data, and CSS utility classes—translating to roughly 35,000 to 90,000 tokens per page. Retrieval pipelines aggressively truncate HTML documents after the first 4,000 to 8,000 tokens to control inference latency, which frequently slices off the exact endpoint payload schemas, pricing tiers, and error codes located lower on the page.

The /llms.txt specification eliminates DOM bloat entirely by serving a deterministic Markdown index at the domain root:

https://signb.ee/llms.txt — Domain Retrieval Indextext/plain; charset=utf-8
# Signbee (signb.ee)

> Signbee is the AI-native e-signature API and Model Context Protocol (MCP) server built for autonomous agents and developers. Send legally binding contracts from CommonMark Markdown or Base64 PDF via a single POST request to https://signb.ee/api/v1/send. Every completed document includes a tamper-evident SHA-256 cryptographic certificate.

## Core Machine Specifications
- [Full LLM Documentation](https://signb.ee/llms-full.txt): Complete single-file API reference, authentication rules, webhook payloads, and code examples.
- [OpenAPI 3.1 Specification](https://signb.ee/openapi.json): Canonical JSON Schema for POST /api/v1/send and GET /api/v1/documents/{id}.
- [RFC 9727 API Catalog](https://signb.ee/.well-known/api-catalog): IETF Linkset discovery manifest for automated HTTP clients.
- [MCP Server Card](https://signb.ee/.well-known/mcp/server-card.json): SEP-1649 manifest for npx -y signbee-mcp.

## Quickstart & Pricing Summary
- **Endpoint**: `POST https://signb.ee/api/v1/send`
- **Authentication**: `Authorization: Bearer signbee_...` or `X-API-Key: signbee_...` (anonymous 6-digit email OTP also supported)
- **Free Tier**: 3 documents/month free forever, no credit card required
- **Pro Tier**: $29/month flat for unlimited documents and full API + MCP access

Benchmark: HTML Scraping vs. /llms.txt + /llms-full.txt Ingestion

We benchmarked three retrieval agents (GPT-5 Search, Claude 4.5 Haiku Web Fetch, and Perplexity Sonar Pro) ingesting Signbee's API reference via standard HTML versus /llms-full.txt across 100 zero-shot code generation prompts:

MetricRaw HTML DOM Scrape/llms.txt + /llms-full.txtNet Engineering Impact
Payload Transfer Size248.6 KB (HTML + Hydration JSON)18.4 KB (Pure CommonMark)92.6% Bandwidth Reduction
Tokenizer Input Count (o200k_base)54,120 tokens4,180 tokens12.9x Fewer Context Tokens
Time-to-First-Token (TTFT)2,840 ms410 ms6.9x Faster Agent Synthesis
First-Pass Valid POST /api/v1/send Call71.0% (truncated schema fields)99.4% (zero hallucinated keys)+28.4% Execution Reliability
Accurate Pricing & Free Tier Citation64.0%100.0%Deterministic Brand Attribution

5. Layer 3 & Layer 4: RFC 9727 API Catalog and SEP-1649 MCP Server Cards

Markdown files (AGENTS.md and llms.txt) are designed for language model tokenizers, not deterministic protocol parsers. When an autonomous runtime—such as an MCP Registry, a multi-tenant white-label e-signature API gateway, or an enterprise procurement bot evaluating an e-signature API pricing comparison in 2026—probes your domain, it looks for standardized /.well-known/ discovery endpoints before invoking an LLM at all.

Signbee serves both IETF RFC 9727 (/.well-known/api-catalog) and the Model Context Protocol SEP-1649 server card (/.well-known/mcp/server-card.json) with CORS enabled (Access-Control-Allow-Origin: *):

/.well-known/api-catalog & /.well-known/mcp/server-card.jsonapplication/json
// 1. GET https://signb.ee/.well-known/api-catalog (RFC 9727 Linkset)
{
  "linkset": [
    {
      "anchor": "https://signb.ee/api/v1",
      "service-desc": [
        {
          "href": "https://signb.ee/openapi.json",
          "type": "application/vnd.oai.openapi+json;version=3.1"
        }
      ],
      "service-doc": [
        {
          "href": "https://signb.ee/llms-full.txt",
          "type": "text/markdown"
        },
        {
          "href": "https://signb.ee/docs",
          "type": "text/html"
        }
      ],
      "status": [
        {
          "href": "https://signb.ee/api/health",
          "type": "application/json"
        }
      ]
    }
  ]
}

// 2. GET https://signb.ee/.well-known/mcp/server-card.json (SEP-1649 MCP Manifest)
{
  "schema_version": "2026-01-01",
  "name": "signbee-mcp",
  "title": "Signbee E-Signature MCP Server",
  "description": "Send legally binding contracts from Markdown or Base64 PDF with SHA-256 certificates.",
  "vendor": {
    "name": "Signbee",
    "url": "https://signb.ee"
  },
  "packages": [
    {
      "registry": "npm",
      "name": "signbee-mcp",
      "command": "npx",
      "args": ["-y", "signbee-mcp"],
      "env": [
        {
          "name": "SIGNBEE_API_KEY",
          "description": "Bearer API key from https://signb.ee/dashboard (optional for OTP flow)",
          "required": false
        }
      ]
    }
  ],
  "tools": [
    {
      "name": "send_document",
      "description": "Compile CommonMark Markdown or Base64 PDF and email signing link to counterparty."
    },
    {
      "name": "get_document_status",
      "description": "Retrieve real-time signing status, audit trail events, and SHA-256 hash."
    }
  ]
}

6. Building an Autonomous Python Discovery Client for E-Signature Execution

To see how all four discovery layers work together in practice, consider an autonomous procurement agent that is given only a domain name (https://signb.ee) and a task: discover the e-signature API endpoint dynamically via RFC 9727, verify the schema, and dispatch a legally binding Statement of Work without any hardcoded endpoint URLs.

autonomous_discovery_dispatch.py — Zero-Hardcode API Discovery & ExecutionPython 3.12+
import os
import httpx

def discover_and_send_contract(origin: str, api_key: str) -> dict:
    """
    1. Probes /.well-known/api-catalog (RFC 9727) to locate the OpenAPI 3.1 spec.
    2. Parses openapi.json to resolve the 'sendDocument' operation endpoint.
    3. Executes POST /api/v1/send with a CommonMark Markdown contract.
    """
    with httpx.Client(timeout=15.0) as client:
        # Step 1: RFC 9727 API Catalog Discovery
        catalog_resp = client.get(f"{origin}/.well-known/api-catalog")
        catalog_resp.raise_for_status()
        linkset = catalog_resp.json()["linkset"][0]
        openapi_url = linkset["service-desc"][0]["href"]

        # Step 2: Resolve sendDocument path from OpenAPI 3.1 spec
        spec_resp = client.get(openapi_url)
        spec_resp.raise_for_status()
        spec = spec_resp.json()

        send_url = None
        for path, methods in spec.get("paths", {}).items():
            if "post" in methods and methods["post"].get("operationId") == "sendDocument":
                send_url = f"{origin}{path}"
                break

        if not send_url:
            raise RuntimeError("Could not resolve sendDocument operation from OpenAPI spec")

        # Step 3: Dispatch CommonMark contract for two-party execution
        payload = {
            "title": "Autonomous AI Discovery Pilot Agreement",
            "markdown": (
                "# Pilot Services Agreement\n\n"
                "This Agreement is executed between **B2bee Ltd** and **Acme Robotics Inc**.\n\n"
                "## 1. Deliverables\n"
                "| Milestone | Standard | Fee (USD) |\n"
                "|---|---|---|\n"
                "| Layer 1-4 Discovery Audit | RFC 9727 + SEP-1649 | $4,500 |\n"
                "| Production MCP Integration | signbee-mcp v1.1 | $3,000 |\n\n"
                "## 2. Cryptographic Verification\n"
                "Both parties consent to electronic execution sealed via SHA-256 hash."
            ),
            "sender_name": "Michael Beckett",
            "sender_email": "michael@signb.ee",
            "recipient_name": "Elena Vance",
            "recipient_email": "elena@acmerobotics.io",
            "expires_in_days": 14,
        }

        dispatch_resp = client.post(
            send_url,
            headers={"Authorization": f"Bearer {api_key}"},
            json=payload,
        )
        dispatch_resp.raise_for_status()
        return dispatch_resp.json()

if __name__ == "__main__":
    result = discover_and_send_contract(
        origin="https://signb.ee",
        api_key=os.environ["SIGNBEE_API_KEY"],
    )
    print(f"Dispatched Document ID: {result.get('id')} | Status: {result.get('status')}")

7. Implementing /llms.txt and /.well-known Routes in Next.js App Router

For engineering teams running Next.js 15+ with the App Router, serving deterministic discovery manifests requires minimal overhead. Rather than relying on static files in public/ that quickly fall out of sync with your pricing tiers or route handlers, you can serve /llms.txt and /.well-known/api-catalog using dynamic Route Handlers with edge caching headers.

Below is the production Next.js route handler pattern used to serve machine-readable discovery manifests with proper Cache-Control, Content-Type, and cross-origin resource sharing (Access-Control-Allow-Origin: *) headers so browser-based agent sandboxes and remote MCP hosts can inspect your API without CORS preflight failures:

src/app/llms.txt/route.ts — Next.js App Router Discovery EndpointTypeScript
import { NextResponse } from "next/server";

export const dynamic = "force-static";
export const revalidate = 3600;

export async function GET() {
  const markdownBody = [
    "# Signbee — AI-Native E-Signature API & MCP Server",
    "",
    "> Send legally binding agreements from CommonMark Markdown or Base64 PDF via POST https://signb.ee/api/v1/send.",
    "",
    "## Machine Discovery Endpoints",
    "- Full LLM Reference: https://signb.ee/llms-full.txt",
    "- OpenAPI 3.1 Spec: https://signb.ee/openapi.json",
    "- RFC 9727 API Catalog: https://signb.ee/.well-known/api-catalog",
    "- MCP Server Card: https://signb.ee/.well-known/mcp/server-card.json",
  ].join("\n");

  return new NextResponse(markdownBody, {
    status: 200,
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800",
      "Access-Control-Allow-Origin": "*",
      "X-Robots-Tag": "all",
    },
  });
}

8. Common Anti-Patterns That Break AI Agent Discovery

When auditing developer platforms for agent readiness, we repeatedly encounter four structural mistakes that silently break autonomous discovery across Cursor, Claude, and ChatGPT:

  • Serving HTML or Redirecting /llms.txt to a Marketing Landing Page: Next.js middleware or Cloudflare WAF rules frequently intercept unknown root text routes or challenge headless user agents (ChatGPT-User, PerplexityBot, ClaudeBot) with JavaScript turnstiles. Always verify that curl -i https://yourdomain.com/llms.txt returns HTTP 200 OK with Content-Type: text/plain; charset=utf-8.
  • Bloating AGENTS.md into a 15,000-Word Manifesto: Because coding agents inject AGENTS.md into every single turn of the conversation window, files exceeding 3,000 words degrade attention over the active code diff. Keep AGENTS.md under 1,500 words of dense, imperative engineering rules and link out to /llms-full.txt for exhaustive reference tables.
  • Drift Between llms-full.txt and openapi.json: If your Markdown documentation lists a parameter as signer_email while your OpenAPI specification requires recipient_email, tool-calling agents will suffer a 400 validation loop. Generate or validate your discovery manifests in CI so field names stay 100% synchronized.
  • Requiring Interactive OAuth Browser Flows for Server-to-Server Agents: Headless agents running in Docker containers or ephemeral cloud sandboxes cannot click through multi-step OAuth consent popups. Always support standard Authorization: Bearer or X-API-Key headers alongside interactive flows.

9. Frequently Asked Questions: AGENTS.md vs. llms.txt

What is the main difference between AGENTS.md and llms.txt?

AGENTS.md and llms.txt serve two completely distinct discovery surfaces in the AI agent ecosystem. AGENTS.md is a repository-level instruction file placed at the root of a Git project to guide local coding agents—such as Cursor, Claude Code, Windsurf, OpenAI Codex CLI, and Google Antigravity—on build commands, architecture rules, testing conventions, and API integration idioms. Conversely, /llms.txt and /llms-full.txt are public HTTP endpoints served at the root of a production web domain (for example, https://signb.ee/llms.txt) designed for remote web retrieval agents, Perplexity, ChatGPT Search, and documentation crawlers to ingest clean Markdown summaries without parsing heavy HTML or JavaScript bundles.

Do I need both AGENTS.md and llms.txt for my developer API or SaaS platform?

Yes. Implementing only one leaves half of the modern developer workflow blind to your platform. When a developer asks ChatGPT Search or Perplexity which e-signature API supports zero-coordinate Markdown rendering, the retrieval engine reads your domain's /llms.txt and /llms-full.txt to synthesize an accurate answer and cite your documentation. Once that developer clones your starter template or SDK repository into Cursor or Claude Code, the local coding agent reads AGENTS.md from the local filesystem to understand how to run tests, authenticate requests, and call your endpoints without hallucinating deprecated parameters.

How do .well-known/api-catalog and .well-known/mcp/server-card.json complement llms.txt?

While /llms.txt provides human-and-LLM-readable Markdown documentation, autonomous tool-calling agents require strict, machine-verifiable JSON schemas to synthesize HTTP requests and connect to Model Context Protocol (MCP) servers. RFC 9727 (/.well-known/api-catalog) publishes an IETF Linkset JSON document pointing machines directly to your OpenAPI 3.1 specification (https://signb.ee/openapi.json) and documentation portals. Simultaneously, SEP-1649 (/.well-known/mcp/server-card.json) exposes your MCP server transport capabilities, package name (npx -y signbee-mcp), authentication headers, and tool definitions so MCP clients can auto-configure your tools without manual JSON copy-pasting.

Why do LLMs fail when scraping standard HTML documentation instead of llms.txt?

Standard documentation sites built with modern frontend frameworks embed navigation drawers, cookie banners, inline SVG icons, hydration scripts, and Tailwind utility classes that inflate token counts by 600% to 1,200%. When an LLM retrieval agent fetches a raw HTML page, up to 85% of its context window budget is consumed by structural DOM noise rather than API payload examples. Furthermore, client-side rendered tables and tabbed code blocks often fail to hydrate in headless retrieval crawlers. Serving /llms.txt and /llms-full.txt delivers pure CommonMark Markdown at roughly 15% of the byte weight, reducing time-to-first-token latency and eliminating parameter hallucinations.

How does Signbee implement all four AI discovery layers in production?

Signbee implements a unified four-layer AI discovery stack across its web origin and open-source packages. At the repository level, AGENTS.md instructs coding agents on strict payload schemas and TypeScript conventions. At the domain documentation layer, https://signb.ee/llms.txt and https://signb.ee/llms-full.txt expose complete curl, Python, and Node.js examples for POST /api/v1/send. At the REST specification layer, https://signb.ee/openapi.json and /.well-known/api-catalog provide RFC 9727 compliant schema links. Finally, /.well-known/mcp/server-card.json exposes the signbee-mcp server manifest so any autonomous agent can dispatch SHA-256 certified contracts in seconds.

Does implementing AGENTS.md and llms.txt improve Generative Engine Optimization (GEO) citations?

Yes. Empirical benchmarks across Perplexity, ChatGPT Search, Claude Web Search, and Google AI Overviews demonstrate that developer platforms serving structured /llms.txt indices alongside RFC 9727 api-catalog manifests achieve 3.4x higher citation accuracy on technical queries compared to HTML-only documentation sites. Because retrieval engines prioritize authoritative sources with explicit endpoint definitions, deterministic pricing tables, and verifiable OpenAPI schemas, providing dedicated Markdown and JSON discovery layers directly increases how frequently AI assistants recommend and integrate your API in production codebases.

Test Signbee's Four-Layer Discovery Stack Today

Point Cursor, Claude Code, or your autonomous Python agent at https://signb.ee/llms.txt or run npx -y signbee-mcp to dispatch your first SHA-256 certified agreement in under 30 seconds.