AlexandrAI AGENTS.md
agents-md

AlexandrAI Tools Robots Txt Generator Agent Guide

Agent guide for building robots.txt user-agent, allow, disallow, and sitemap lines.

AGENTS.md

Guidance for coding agents working on AlexandrAI Tools Robots Txt Generator in the alexandrai-tools project.

Project overview

This guide covers src/tools/robots-txt-generator/index.ts. The tool always starts with User-agent star, optionally adds Allow slash, trims disallow entries and skips blank paths, optionally appends a blank line followed by a sitemap line, and joins output with newline characters.

Primary source files

  • src/tools/robots-txt-generator/index.ts defines meta, RobotsOptions, and buildRobots.
  • src/tools/robots-txt-generator/index.test.ts covers allow-all output, disallow lines, sitemap inclusion, blank disallow skipping, and omitted sitemap behavior.
  • src/tools/robots-txt-generator/Body.astro owns browser controls.
  • src/tools/robots-txt-generator/i18n.ts owns localized copy.
  • src/lib/tools.ts registers metadata.

Setup commands

npm install npm test npm run typecheck npm run build npm run dev

Change rules

  • Always start output with User-agent: *.
  • Add Allow: / only when allowAll is true.
  • Trim disallow path entries before use.
  • Skip blank disallow entries.
  • Append sitemap only when a trimmed sitemap value is present.
  • Insert a blank line before sitemap output.
  • Do not validate crawler policy semantics in pure string builder unless tests are added.

Testing

Test allow-all, block-only, multiple disallow paths, blank disallow values, sitemap inclusion, no sitemap, sitemap trimming, newline ordering, and UI examples without embedding real URLs in generated docs.

PR checklist

  • npm test passes for robots-txt-generator.
  • Generated output shape, escaping, and invalid-input behavior remain documented.
  • Browser-local generation stays free of hidden network calls.
  • UI labels, metadata, and tests update together for behavior changes.