AlexandrAI AGENTS.md
agents-md

Insane Search Fetch Chain Agent Guide

Project-specific AGENTS.md guidance for maintaining the Insane Search fetch chain, probe, validation, WAF profile planning, exhaustive grid, fallback, trace, and failure gate.

AGENTS.md

Guidance for coding agents working on the Insane Search fetch chain role.

Project Overview

The fetch chain owns the single public retrieval entrypoint and its internal phases: official-route check, curl probe, validation, WAF detection, profile planning, diversity-ordered execution, browser fallback, result tracing, and honest failure reporting. The contract is that callers use one function or CLI path and receive enough trace data to decide whether retrieval is truly done.

Project Map

  • skills/insane-search/engine/fetch_chain.py defines Attempt, FetchResult, fetch(), _fetch_core(), _build_plan(), _give_up(), and fetch_many().
  • skills/insane-search/engine/main.py exposes the CLI wrapper around fetch().
  • skills/insane-search/engine/validators.py decides whether an attempt is strong_ok, weak_ok, suspect_ok, challenge, blocked, rate_limited, auth_required, not_found, or unknown.
  • skills/insane-search/engine/waf_detector.py ranks profile matches for the planner.
  • skills/insane-search/engine/executor.py runs browser fallback and bridges browser state back into curl sessions.
  • skills/insane-search/engine/tests/test_smoke.py and test_u1.py cover scheduler and validation behavior.

Setup Commands

bashcd skills/insane-search
python3 -m engine "<URL>" --trace --json
python3 engine/tests/test_smoke.py
python3 engine/tests/test_u1.py
python3 engine/bias_check.py

Change Rules For Agents

  • Preserve fetch() as the single public entrypoint for generic page retrieval.
  • Keep the phase order explicit: official route, probe, validate, detect, plan, execute, fallback, report.
  • Do not treat HTTP success as retrieval success until validate() returns terminal success.
  • Keep max_attempts=None as exhaustive mode; numeric max_attempts is only a budget.
  • Preserve the difference between grid_exhausted, budget exit, terminal non-success, and browser fallback failure.
  • Keep suspect_ok non-terminal so ambiguous content cannot stop the search early.
  • Keep _build_plan diverse across TLS family, transform, referer, and profile rank.
  • Do not add site-specific branches to fetch_chain.py; runtime selectors and user_hint are the allowed customization points.

Verification

bashcd skills/insane-search
python3 engine/tests/test_smoke.py
python3 engine/tests/test_u1.py
python3 engine/tests/test_u4.py
python3 engine/tests/test_u5.py
python3 engine/tests/test_u7.py
python3 engine/bias_check.py

For fetch behavior changes, inspect trace shape as well as final content. A passing test that hides untried_routes or must_invoke_playwright_mcp is not acceptable.

Security Notes

Never log cookies, bearer values, private hosts, or local endpoints in trace output. Keep URL-specific knowledge in runtime hints or bounded data stores, not source code.

PR Checklist

  • fetch() signature remains backward compatible.
  • Failed results distinguish exhaustion from budget limits.
  • Trace entries include phase, executor, transform, impersonation, referer, status, body size, verdict, and reasons.
  • Bias check passes.
  • Scheduler and validation tests cover any new branch.