mdorigin

Markdown-first publishing for humans and agents.

Search and agent discovery

Use this guide to make one published content tree readable by browsers, search engines, and agents. These capabilities improve output correctness and navigation; they do not guarantee ranking, rich results, or AI citations.

Prerequisites

Configure discovery

{
  "siteUrl": "https://example.com",
  "llmsTxt": true,
  "structuredData": true
}

The generated /llms.txt lists Markdown pages by locale and section, plus the OpenAPI/search entry points. It excludes drafts, noindex pages, and duplicate canonical URLs. The directory is limited to 200 pages and 32 KB, not a full-text export. A content-root llms.txt always takes precedence, even with generation disabled; use it to curate a larger site. There is no automatic llms-full.txt.

HTML and Markdown responses advertise the optional directory using HTTP Link: ...; rel="describedby". Explicit .md responses and negotiated Markdown also carry a canonical Link to the HTML page. Markdown is not automatically marked noindex.

Choose indexing and crawler policy separately

---
title: An accessible but unlisted page
robots:
  index: false
  follow: true
  maxSnippet: 120
search: false
---

robots produces matching HTML meta and X-Robots-Tag directives on every representation. noindex: true is a shorthand for robots.index: false. Such pages are omitted from sitemap, feeds, and generated agent discovery, but remain accessible. Site search is independent: only search: false excludes the document from its next search build. Neither setting is authentication.

Use robots.rules to choose crawler-specific policy explicitly:

{
  "robots": {
    "rules": [
      { "userAgent": "OAI-SearchBot", "allow": ["/"] },
      { "userAgent": "GPTBot", "disallow": ["/"] }
    ]
  }
}

This is an example policy, not a default. Without rules, the generated robots.txt only declares the configured Sitemap URL. A user-owned robots.txt always wins. Do not disallow crawling of a noindex page if a cooperating crawler needs to read its noindex directive.

Describe content honestly

Use date for publication, updated for the last substantive content update, and author for a real person or array of names. Quote date values if precision matters. Valid formats are ISO calendar dates or timestamps with explicit timezones. Invalid values are omitted, never replaced with the build time.

With structuredData: true, explicit type: post produces BlogPosting; other pages produce WebPage. Author and publication/modification dates come from the same normalized page model; missing values are not invented. showSummary and showDate do not erase search or social metadata.

Search and cite a section

mdorigin build search --root docs/site --out dist/search
curl 'https://example.com/api/search?q=cloudflare&topK=5'
curl -H 'Accept: text/markdown' 'https://example.com/guides/cloudflare'

Search results keep page-level canonicalUrl and may add citationUrl with a rendered heading fragment. Heading IDs are deterministic across repeated headings, inline formatting, and non-Latin text; explicit HTML IDs are respected. Rebuild indexes created before this capability to include content revisions.

Citations are omitted for stale content, ambiguous repeated excerpts, missing source pages, and legacy indexes. charStart/charEnd are indexbind's search content coordinates (UTF-8 byte offsets in 0.6.x), not stable offsets in the raw Markdown file. Do not implement raw-file range reads using them.

Verify before and after deployment