AlexandrAI AGENTS.md
agents-md

Insane Search Ranking and Discovery Agent Guide

Project-specific AGENTS.md guidance for Insane Search ranking and discovery helpers, source bonuses, freshness scoring, canonical dedupe, RSS feed extraction, robots sitemap parsing, and sitemap URL extraction.

AGENTS.md

Guidance for coding agents working on the Insane Search ranking and discovery role.

Project Overview

Ranking and discovery turn raw public search and site index material into a cleaner candidate set. Discovery finds RSS, Atom, robots sitemap, and sitemap loc entries. Ranking scores fetched documents using query overlap, phrase matches, source quality, length, freshness, and canonical dedupe.

Project Map

  • skills/insane-search/search_tools/discovery.py defines feed link extraction, robots sitemap discovery, and sitemap XML loc extraction.
  • skills/insane-search/search_tools/ranking.py defines tokenization, freshness scoring, rank_documents(), and dedupe_documents().
  • skills/insane-search/search_tools/url_tools.py canonicalizes URLs before dedupe.
  • skills/insane-search/search_tools/models.py defines SearchDocument fields used by ranking.
  • skills/insane-search/search_tools/tests/test_search_tools.py covers ranking, dedupe, feed discovery, and sitemap discovery.

Setup Commands

bashcd skills/insane-search
python3 search_tools/tests/test_search_tools.py

Change Rules For Agents

  • Keep discovery tolerant of malformed HTML and XML.
  • Preserve input order when deduping discovered feed and sitemap URLs.
  • Canonicalize sitemap loc values before returning them.
  • Keep ranking deterministic for equal scores.
  • Do not let source bonuses overwhelm direct title and body relevance.
  • Keep freshness bounded so recent weak matches do not beat strong relevant matches.
  • Preserve canonical URL dedupe and keep the best-scoring document.
  • Add tests for any new source type or scoring weight.

Verification

bashcd skills/insane-search
python3 search_tools/tests/test_search_tools.py

When changing weights, update tests with a scenario that proves the intended ordering rather than only checking that scores exist.

Security Notes

Treat discovered URLs as untrusted input. Ranking and discovery should not fetch private resources, inject headers, or carry credentials; they only parse and order public candidate metadata.

PR Checklist

  • Feed and sitemap parsing remain exception-safe.
  • Canonical dedupe keeps the best item.
  • Ranking order is deterministic.
  • Freshness and source bonuses stay bounded.
  • Search tool tests pass.