LLMs.txt & AI Crawler Directive Generator

As Large Language Models (LLMs) and conversational search engines (ChatGPT Search, Perplexity, Claude, Google Gemini) increasingly mediate how users discover information, websites face a major technical challenge: standard HTML webpages are bloated with navigation scripts, styling sheets, advertising pixels, and complex DOM hierarchies that consume excessive LLM context tokens and degrade inference accuracy.

Proposed as an open standard, /llms.txt serves as a curated, lightweight Markdown index hosted at the root of a domain (e.g., https://yourdomain.com/llms.txt). Similar to how /robots.txt directs search engine crawlers and /sitemap.xml catalogs URLs, /llms.txt provides AI agents with a concise, high-density summary of your site's core purpose, structured APIs, and key documentation links formatted specifically for minimal token consumption and optimal retrieval-augmented generation (RAG).

A common mistake among webmasters is treating all AI bots identically in robots.txt. In reality, AI crawlers fall into two distinct operational categories:

• AI Search & Referral Bots (SearchGPT / OAI-SearchBot, PerplexityBot, ChatGPT-User): These crawlers fetch real-time web content to answer immediate user queries and provide active citation links back to your site, driving high-intent organic referral traffic. Blocking these bots eliminates your brand from AI-generated search answers.
• AI Model Training Scrapers (GPTBot, CCBot, ByteSpider, Cohere-ai): These bots crawl bulk web content to train foundation models (GPT-5, Claude, Doubao). They ingest content into training weights without providing direct per-query citation links.
• Google-Extended vs. Googlebot: Google-Extended is a dedicated crawler token used specifically to opt out of Google Gemini and Vertex AI training datasets. Blocking Google-Extended does NOT impact your rankings or indexing in Google Web Search (which uses Googlebot).

To ensure deterministic parsing across autonomous agents, an /llms.txt file must adhere to standard Markdown syntax:

• H1 Document Title (# Project Name): The primary identifier of your application, documentation, or organization.
• Mandatory Blockquote Summary (> Brief overview): A concise, 2-to-3 sentence synthesis placed immediately below the H1 heading. AI agents ingest this blockquote as an authoritative system prompt description.
• Optional Context Body: 1–2 paragraphs providing background architecture, prerequisites, or licensing details.
• H2 Section Headings (## Section Title): Grouped collections of resource links (e.g., ## Core APIs, ## Guides, ## Schema Models).
• Markdown Bullet Links (- [Title](URL): Description): Each resource must feature an anchor label, an absolute or root-relative URL, and an optional colon-separated description explaining its payload.
• Optional /llms-full.txt Pointer: For comprehensive documentation sets, link to a consolidated /llms-full.txt file containing the complete concatenated documentation in plain Markdown.

To serve /llms.txt with high performance and zero server latency, implement a dedicated Route Handler in Next.js App Router at app/llms.txt/route.ts:

// app/llms.txt/route.ts
import { NextResponse } from 'next/server';

export const dynamic = 'force-static';
export const revalidate = 86400; // Cache for 24 hours

export async function GET() {
  const llmsContent = `# Your Project\n\n> Summary context...`;
  return new NextResponse(llmsContent, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400, stale-while-revalidate=43200',
    },
  });
}

Alternatively, place a static llms.txt file directly inside your Next.js /public directory, or configure edge header rules in Cloudflare Pages and Vercel.