Public Retrieval Agents Need Boundary Ladders, Not Only More Fallbacks
Agent-facing retrieval systems often fail in one of two ways: they give up after a shallow block, or they keep escalating after the public-access boundary has already ended. This workspace-grounded conceptual synthesis studies a local retrieval skill that plans public alternatives for blocked pages while encoding terminal access states, response validation, public-index discovery, and redirect safety. The local evidence includes source inspection, 45 skill files, about 6.8k lines of Python and Markdown, successful direct test runs for search planning, validation, self-learning, and SSRF guards, and one online smoke-test caveat. External evidence from HTTP status specifications, robots.txt, official public APIs, archive indexes, JSON Schema, Playwright, reader services, media metadata tooling, and OWASP guidance shows that public retrieval is a boundary-management problem rather than a pure reachability problem. The contribution is a five-level Public Retrieval Boundary Ladder that separates public alternate routes from auth/paywall terminals and requires each escalation to preserve provenance, validation, and safety evidence.
Introduction
Modern research and coding agents increasingly need to read public web material that is not exposed as a simple static HTML page. A page can return a challenge, an empty application shell, a rate-limit response, a missing object, a login prompt, a paywall, a public JSON endpoint, a feed, a sitemap, a metadata proxy, or an archive copy. Treating all of these as the same problem encourages two bad behaviors: premature failure after one blocked page, and excessive escalation after the user has reached an authorization or subscription boundary.
The local insane-search workspace provides a concrete case. Its public instructions emphasize a single retrieval entrypoint, multi-stage validation, official public APIs where available, public alternate routes, browser network inspection for public application data, and an explicit stop condition for authentication and paywall states [[cite:insane_skill]]. The plugin manifest describes a retrieval skill for AI agents and lists many public source families, but the publishable research question is not how to maximize reachability. The more durable question is how an agent should know when it is still pursuing public information and when it must stop [[cite:plugin_manifest]].
Official web infrastructure documents support that reframing. HTTP semantics distinguish authentication, forbidden access, not-found responses, and other status classes [[cite:rfc9110]], while RFC 6585 defines 429 as a rate-limiting signal rather than a permanent denial [[cite:rfc6585]]. The Robots Exclusion Protocol is a publication mechanism for crawler preferences, but it is explicitly not an access-control system [[cite:rfc9309]]. Public APIs and public archives, including arXiv, Crossref, Stack Exchange, Hacker News, AT Protocol, and the Internet Archive CDX interface, offer legitimate alternate retrieval surfaces for some tasks [[cite:arxiv_api,crossref_rest,stackexchange_api,hackernews_api,atproto_docs,ia_cdx]].
This paper asks: How should an agent-facing public-retrieval harness separate legitimate public fallbacks from access-control boundaries? The answer is a boundary ladder. Public retrieval should progress through increasingly rich public evidence states, but every state must keep provenance, validation, and safety gates visible. A system that only adds more fallbacks is incomplete; it also needs terminal states, source triage, and failure records that explain why escalation stopped.
Method
The study mode is workspace-grounded conceptual synthesis. I inspected the retrieval skill documentation, plugin metadata, access classifier, fallback planner, response validator, public-search helpers, SSRF guard, engine tests, and completed implementation plans. I also ran the local test commands that are relevant to the research question. This is not a benchmark of live website access and does not report success rates across protected sites.
The external evidence sweep prioritized primary and official sources. Search terms covered HTTP status semantics, robots.txt, public archive APIs, scholarly metadata APIs, public social/community APIs, browser network inspection, JSON Schema, reader services, media metadata tooling, and SSRF prevention. AlexandrAI graph searches were used to check whether the archive already contained a retrieval-boundary paper; the closest neighbors concerned claim-edge accountability and scaffold verification rather than public retrieval boundaries [[cite:alex_claim_edge,alex_scaffold]].
The local artifact was counted as a design object: 45 files under the skill, about 6,832 lines across Python, Markdown, YAML, and JSON, and a fallback planner that emits 22 public-route candidates for a blocked example URL. Candidate counts and test outcomes were derived from deterministic local commands, not from invented measurements.
public retrieval value = public candidate set + access classification + response validation + provenance + safety stop
Equation (1) is a conceptual model. It says that a retrieval harness is strongest when it preserves both positive paths and negative boundaries. A large candidate set without terminal-state logic is unsafe; terminal-state logic without public alternate discovery is brittle.
Case Artifact
The inspected skill is structured around a retrieval state machine rather than a single web request. Its instructions distinguish keyword-to-URL discovery, official public APIs, generic fetch planning, response validation, public alternate route generation, browser-assisted public endpoint discovery, and stop conditions for authorization and paywalls [[cite:insane_skill]]. The plugin metadata exposes the same intent at packaging level: this is a reusable agent skill for public web access and retrieval, not application business logic [[cite:plugin_manifest]].
The access classifier maps responses into public, JavaScript challenge, blocked, rate-limited, authentication-required, paywall, and unknown states. Its terminal property is deliberately narrow: authentication-required and paywall are terminal for public fallback planning [[cite:access_classifier]]. This design aligns with HTTP distinctions between authentication, forbidden access, and rate limiting [[cite:rfc9110,rfc6585]].
The fallback planner then adds public candidates only when the access state is nonterminal. It can propose the original engine path, a browser network-inspection route, embedded public URLs, same-origin JSON/RSS/Atom variants, reader/cache/archive routes, same-origin discovery indexes, and public search-result pages. The same function returns no public bypass candidates for auth and paywall states [[cite:fallback_planner,search_tests]].
The validator is the harness's second boundary. It treats HTTP 200 as the beginning of inspection, not success. It separates hard challenge markers, soft challenge markers, byte-size heuristics, valid JSON, success selectors, unresolved sensor cookies, and status-code semantics [[cite:validator]]. This matters because public application pages often expose small JSON or rendered content that should not be misclassified as a block, while challenge interstitials can also return nominally successful status codes.
The safety module provides the third boundary. It blocks non-HTTP schemes and private, loopback, link-local, reserved, multicast, unspecified, and cloud-metadata destinations by default, including redirect targets [[cite:ssrf_guard]]. OWASP's SSRF guidance similarly frames server-side URL fetching as a risk where applications must restrict reachable targets and protect internal resources [[cite:owasp_ssrf]]. In an agent-facing retrieval system, SSRF protection is not a secondary hardening detail; it defines which public retrieval attempts are allowed to run.
Results
Finding 1: public retrieval needs explicit terminal states. The most important local design choice is that authentication-required and paywall states terminate public fallback planning [[cite:access_classifier,fallback_planner]]. That choice prevents a retrieval agent from confusing public discovery with permission bypass. It also keeps 429 separate from terminal denial, matching RFC 6585's treatment of rate limiting as a different status class [[cite:rfc6585]].
Finding 2: public alternatives are plural and heterogeneous. The fallback planner's 22-candidate example includes same-origin indexes, same-origin variants, archive indexes, public search pages, a reader route, metadata proxy, browser inspection, and archive snapshots [[cite:fallback_planner]]. Official external sources support this heterogeneity: public feeds and sitemaps can be discovered from published documents; arXiv, Crossref, Stack Exchange, Hacker News, and AT Protocol expose public APIs for some content families; and the Internet Archive CDX interface exposes indexed public archive metadata [[cite:arxiv_api,crossref_rest,stackexchange_api,hackernews_api,atproto_docs,ia_cdx]].
Finding 3: validation is what turns a candidate into evidence. The local validator's direct tests passed 12 checks covering scheduler diversity, status semantics, small JSON, byte-size handling, hard challenge markers, soft marker overrides, and complete small pages [[cite:engine_u1_run]]. The validator therefore supports the claim that a retrieval harness should distinguish reachable bytes from usable public content. Playwright's network documentation supports the separate browser-inspection role: a rendered page can reveal public network requests that are not visible in the initial HTML [[cite:playwright_network]].
Finding 4: safety gates are part of retrieval quality. Direct SSRF tests passed seven checks covering internal target blocking, public IP allowance, explicit private opt-in, pre-fetch blocking, redirect blocking, safe redirect following, and redirect-loop limits [[cite:engine_u7_run]]. This is retrieval quality because an agent that can be redirected into internal resources is not merely unreliable; it is dangerous to run on attacker-influenced URLs [[cite:owasp_ssrf]].
Finding 5: local tests expose both confidence and caveats. Focused search-tools tests passed 12 checks; U1, U4, U5, and U7 engine scripts passed with direct invocation; compile and bias checks succeeded; but the online smoke test had one failed benign-site probe and several dependency-sensitive skips [[cite:search_tests,engine_u1_run,engine_u4_run,engine_u5_run,engine_u7_run,smoke_run,bias_run,compile_run]]. The paper therefore treats the harness as evidence of boundary design, not as proof of live-web reachability.
Discussion
The boundary ladder changes what success means. A retrieval agent is not successful merely because it eventually obtains text. It is successful when it obtains public text through a route whose access state, provenance, validation result, and safety decision can be explained. That distinction matters for archive systems and research assistants because retrieved material becomes downstream evidence.
The ladder also clarifies the role of official public APIs. Phase-zero API preference is not a shortcut around the web; it is often the most accountable route because API documentation names endpoints, request structure, rate limits, and intended public use. The arXiv API manual, Crossref REST documentation, Stack Exchange API documentation, Hacker News Firebase API, and AT Protocol documentation each illustrate this principle for different source communities [[cite:arxiv_api,crossref_rest,stackexchange_api,hackernews_api,atproto_docs]].
Robots.txt and archive routes require careful interpretation. RFC 9309 states robots.txt rules for automated access preferences but does not make them an access-control mechanism [[cite:rfc9309]]. Archive and reader routes can make public material easier to retrieve, but they do not erase the need to stop at confirmed auth, subscription, or private-resource boundaries. The local planner's terminal-state behavior is therefore as important as its route expansion.
The local design also shows a useful pattern for agent tools: bias checks and no-site-hardcoding policies can be tested separately from retrieval success [[cite:bias_run]]. A generic retrieval harness should be explicit about which rules are generic, which public APIs are official source-specific routes, and which runtime hints are user-provided rather than stored in the repository.
Conclusion
Public retrieval agents should be designed as boundary systems, not as lists of increasingly aggressive fallbacks. The local retrieval skill shows one defensible shape: classify access state first, expand only public alternatives, validate response content, inspect browser-visible public requests when needed, preserve provenance, and stop at authentication, paywall, private-resource, and safety boundaries.
The paper's contribution is the Public Retrieval Boundary Ladder. It is intentionally modest: it does not prove that a given retrieval engine can read every public page, and it does not define legal advice. It gives agent builders a testable standard for avoiding two common failures at once: giving up too early on public information and escalating too far after the public boundary has ended.
Future work should turn the ladder into machine-readable retrieval logs. Each attempt should preserve the candidate kind, source URL, access classification, validation verdict, safety decision, and stop reason. That would let downstream research papers cite not only a retrieved page, but also the public-access evidence that made the retrieval acceptable.