AlexandrAI AGENTS.md
agents-md

Insane Search Skill Agent Guide

Project-specific AGENTS.md guidance for maintaining the Insane Search public retrieval skill, generic fetch chain, WAF profiles, public fallback planner, and bias checks.

AGENTS.md

Guidance for coding agents working on Insane Search, a public retrieval skill for blocked or JavaScript-heavy web pages.

Project Overview

The project is a Claude/Codex skill with a Python engine and search-planning helpers. Its central rule is that HTTP 200 is only the beginning of validation; success requires content checks, challenge-marker checks, cookie-state checks, and optional selector proof.

Key paths:

  • skills/insane-search/SKILL.md: operational rules and trigger guidance.
  • skills/insane-search/engine/: fetch chain, transport, validators, WAF detection, safety, learning, and URL transforms.
  • skills/insane-search/search_tools/: public retrieval planning, ranking, URL tooling, and search discovery.
  • skills/insane-search/references/: platform and fallback notes.
  • skills/insane-search/tests/coverage_battery.py: behavior coverage battery.
  • skills/insane-search/engine/bias_check.py: no-site-name guard.

Setup Commands

bashcd <project-root>/skills/insane-search
python3 -m engine "<PUBLIC_URL>" --trace
python3 -m engine "<PUBLIC_URL>" --trace --json
python3 engine/bias_check.py
python3 tests/coverage_battery.py

Use <PUBLIC_URL> in examples and tests. Do not preserve private browsing targets in fixtures, traces, or public docs.

Design Rules

  • Keep a single generic entry point: python3 -m engine "" or engine.fetch(...).
  • Do not stop at the first HTTP 200. Validate content before declaring success.
  • Keep site-specific selectors, referers, cookies, and hints out of stored engine code. Runtime hints are acceptable; hardcoded domain behavior is not.
  • Prefer official public APIs and feeds when they exist.
  • Public fallback planning may use same-origin variants, reader services, archives, feeds, sitemaps, metadata, and public search results. It must not bypass auth or paywalls.
  • Playwright/MCP reconnaissance is a last-mile public-read tactic, not permission to use private browser state.

Code Style

  • Put transport and TLS details in engine/transport.py.
  • Put validation in engine/validators.py; do not duplicate success rules in callers.
  • Put WAF profile logic in engine/waf_detector.py and generic profiles, without brand-specific shortcuts in core code.
  • Put public search and fallback planning in search_tools/.
  • Add reference docs under references/ only when they describe reusable public behavior.

Testing And Verification

  • Run the bias check after touching engine logic, WAF profiles, references, or tests.
  • Run the coverage battery after changing retrieval planning, URL transforms, terminal-state classification, or validation.
  • Include trace summaries in development notes, but redact URLs that expose private targets or query parameters.

Security

  • Never store cookies, tokens, API keys, logged-in browser state, or private headers.
  • Treat auth-required and paywall states as terminal public boundaries.
  • Do not add code that scrapes behind login, rotates identities, or hides automation for non-public access.