AlexandrAI AGENTS.md
agents-md

Insane Search Public Fallback Planner Agent Guide

Project-specific AGENTS.md guidance for Insane Search public fallback planning, access-state classification, same-origin variants, public indexes, reader/cache/archive candidates, and terminal access boundaries.

AGENTS.md

Guidance for coding agents working on the Insane Search public fallback planner role.

Project Overview

The public fallback planner builds a retrieval worklist when a public page is blocked, JS-rendered, rate-limited, or uncertain. It may suggest engine retry, browser inspection, embedded public data links, same-origin variants, reader/cache/archive routes, public indexes, and public search result pages. It must stop when auth or paywall boundaries are confirmed.

Project Map

  • skills/insane-search/search_tools/fallbacks.py defines RetrievalCandidate, RetrievalPlan, build_public_fallback_urls(), and plan_public_retrieval().
  • skills/insane-search/search_tools/access.py classifies public, JS challenge, blocked, rate limited, auth required, paywall, and unknown states.
  • skills/insane-search/search_tools/url_tools.py canonicalizes candidate URLs.
  • skills/insane-search/search_tools/discovery.py extracts feed and sitemap candidates.
  • skills/insane-search/search_tools/tests/test_search_tools.py covers fallback planning, terminal boundaries, embedded public URLs, and JS escalation.
  • skills/insane-search/SKILL.md defines the public retrieval escalation rules.

Setup Commands

bashcd skills/insane-search
python3 search_tools/tests/test_search_tools.py

Change Rules For Agents

  • Always classify access state before adding fallback candidates.
  • Return terminal=True for auth-required and paywall states.
  • Keep the generic engine candidate first for non-terminal states.
  • Add browser inspection when public access is blocked, challenged, rate-limited, or unknown.
  • Extract only public embedded resource URLs.
  • Filter sensitive paths such as login, session, auth, password, and token-bearing links.
  • Keep same-origin variants and public indexes bounded and deterministic.
  • Do not create fallbacks that log in, sign up, scrape private sessions, or evade paywalls.

Verification

bashcd skills/insane-search
python3 search_tools/tests/test_search_tools.py
python3 engine/bias_check.py

For every new candidate kind, add a test that proves ordering, deduplication, terminal-boundary behavior, and sensitive URL filtering.

Security Notes

This module maximizes public retrieval only. Confirmed auth and paywall controls are hard stops unless a user separately provides an authorized browser session for their own access.

PR Checklist

  • Access classifier remains the first decision.
  • Terminal states produce no public bypass worklist.
  • Sensitive embedded URLs are filtered.
  • Candidate order is stable.
  • Search tool tests pass.