One run, opened all the way down.

The question — and the one the run decided to answer instead.

Does ignoring robots.txt create legal liability for a research crawler in the US and the EU after the 2024–2025 case law, and where do practitioners draw the safe line? as enrolled · bench/sets/campaigns-mvp.json · campaign x2-c09, “Crawling, robots.txt and archives”

How do US and EU laws differentiate liability for ignoring robots.txt based on the crawler's purpose (commercial vs. non-commercial research), and what technical boundaries define the current practitioner safe line? the run's own rewrite, written in the recharter state and printed at the head of its report

• runa7a32519ce86… · job x2-c09-q1 · ended done2026-09-19, 10:22–10:23 America/Toronto — first search to last page read
• depthstandard — a $0.40 ceiling on the run, of which it spent $0.202613bench/sets/gold-8/caps.toml; the reservation is taken before the call, not reconciled after it
• plan · doctrine2f351b2f373d · 750c8e80cbddcontent hashes of the audit-first plan v1 and doctrine v1 — the two hashes that make this run replayable
• modelsGemini 3.7 Flash and Gemini 3.1 Pro through OpenRouter, a pair per uniteach recorded as a dated permaslug with a price ref, so the price charged is on the record beside the call

The rewrite is not decoration. recharter is a state with a model call and a cost of its own — $0.038980, the second most expensive state in the run — and the question it writes is the one everything after it is held to.gate ledger: run_head, run_task, state.entered · rsk research result --run x2-c09-q1, head of the report

Six states, fourteen searches, twenty-four pages, ninety-nine seconds.

The row worth stopping on is the rejected one. A verifier proposed a claim anchored to a page its own task had never fetched. The ledger refused the whole output inside the commit transaction, kept the raw bytes as evidence, and made the unit run again. That second attempt produced the verdict opened below.gate ledger: rejection (stage invariant, code invariant.A1, path /anchors/1/url), unit.rejected and unit.repaired on task b5dc009d… · docs/SPEC-kernel.md, invariant A1

The report it wrote, as it renders.

# Does ignoring robots.txt create legal liability for a research crawler in the US
  and the EU after the 2024-2025 case law, and where do practitioners draw the safe line?

_depth: standard · plan: 2f351b2f373d · doctrine: 750c8e80cbdd · claims: 18 ·
 sources: 22 · calls: 11 · cost: $0.2026 · robots: standard_

## Answer

In the US, ignoring robots.txt to scrape public, unauthenticated data does not
inherently violate the CFAA or breach browsewrap terms (Meta v. Bright Data,
2024). However, it can contribute to DMCA anti-circumvention claims if combined
with technical evasion like IP rotation. In the EU, liability depends entirely
on the type of research. Under the 2025 Kneschke v. LAION ruling, commercial
Text and Data Mining (TDM) must respect machine-readable opt-outs like
robots.txt (Article 4 DSM Directive). Conversely, genuine non-commercial
scientific research is exempt from these opt-outs (Article 3 DSM Directive) and
can legally bypass them. Practitioners draw the safe line by strictly separating
commercial and non-commercial crawlers, adhering to machine-readable opt-outs
for commercial projects, and never bypassing active technical barriers (like
CAPTCHAs or logins) in either jurisdiction.

_Confidence: high confidence._

## Findings

- **In Meta Platforms v. Bright Data (January 2024), the court ruled that
  logged-off public scraping does not breach platform terms of service intended
  for account holders.** - high confidence · dataimpulse.com, cloro.dev
  > Meta v. Bright Data (January 2024) sharpened the line: a court declined to
  > find Bright Data in breach for scraping public, logged-out Facebook and
  > Instagram pages, since the platforms' terms govern account holders, not
  > logged-out visitors.

That is the head of a report of 119 lines. The rest is eleven more findings each with its own quote, the contested claim below, two adversarial verdicts, what it could not establish, next steps, the assumptions the question was resting on, five orientation surprises each marked unverified: no quote from an opened page, and 22 numbered sources. None of it is stored. rsk research result --run x2-c09-q1 derives the whole text from the ledger each time it is asked — which is why replaying a run reproduces its report exactly.the output of rsk research result over the X2 gate ledger, wrapped to fit · docs/SPEC-kernel.md §8, the report interface · docs/GATE-POC.md check 3 (replay: 79 of 79 calls served from the ledger, identical report, $0)

One sentence of that report, opened.

Following the 2024 Meta v. Bright Data ruling, ignoring robots.txt to scrape publicly accessible, unauthenticated data does not violate the US CFAA or constitute a breach of browsewrap terms. claim d8f8285c… · verdict d2159e30… — the sentence the answer's first clause rests on

• resolved5 of 5, every one rung = exact, occurrences = 1a quote found twice in a page, or not found at all, does not resolve — and an unresolved anchor carries no byte range to print
• re-readinside the commit transaction, against the stored bodyinvariant A1: an output whose quote is not in the page it cites is never admitted. Not a warning afterwards — a refusal
• first attemptrefused invariant.A1 — url not fetched in runthe verifier cited a page its own task had not retrieved. $0.010613 paid, nothing admitted, the raw output kept as evidence
• second attemptadmitted — the five anchors above$0.011012. This one sentence cost $0.021625, of which half bought a refusal
• the verifier saidconfirmed — three supporting quotes against two contradictingthe model's own label, written into the verdict row and kept there
• the ledger sayscontested — the claim has two undecided attackersthe fold decides a claim's status, not the model; the report prints the disagreement instead of settling it quietly

This object is the whole argument of the project. Any sentence of any report opens the same way: to its claim, to the quotes under it, to the page each quote came from, to the byte range inside that page, to the verdict, and to what the verdict cost. The two sociavault.com ranges above were decompressed from the body store and re-read while this page was being written; they are the sentences printed beside them, character for character.gate ledger: arg_verdict, arg_anchor, claim_status, budget_reservation, rejection for run a7a32519ce86… · body sha256_text 0ee390e5…, 33,334 bytes of extracted text · docs/SPEC-kernel.md A1 · docs/GATE-POC.md check 4 — 498 anchors, 490 resolved, 0 failing the byte re-read, 0 at a non-ok fetch

What it could not establish — in its own words.

• searched, absent“Searched for explicit cases where non-commercial research organizations bypassed robots.txt and faced litigation specifically testing whether robots.txt is binding under Article 3 DSM Directive / Section 60d UrhG; the available rulings focused on natural language vs. machine-readable opt-outs under Article 4 / Section 44b UrhG.”
• searched, absent“Searches in the provided text yielded no evidence that breach of browsewrap terms alone satisfies CFAA 'without authorization' thresholds post-Van Buren.”
• searched, absent“Searching specifically for statutory robots.txt codification in the CFAA yielded nothing; robots.txt remains a machine-readable protocol without independent statutory force under 18 U.S.C. § 1030.”
• extract too thin“Searched Medium post on AI-oriented robots.txt by Francisco A. Kemeny (file extract provided only generic allow/disallow summary).”
• extract too thin“Searched IPTC Generative AI Opt-Out Best Practice Recommendations full text beyond snippet (file omitted from retrieved text except snippet).”
• could not read“Could not read iptc.org — failed (HTTP 200)”the PDF the line above wanted: 200 OK, and a content type the extractor of the day could not open. PDFs are read on main since QUAL-READ, #1203
• could not read“Could not read onlinelibrary.wiley.com — failed (HTTP 403)”a bot challenge. The fallback ladder recovered the paper through web.archive.org, and it is source 16 of the report

Seven lines, quoted exactly. The report prints nine: two of them restate an earlier line in shorter form, a duplication QUAL-REPORT fixed on 2026-09-21, two days after this run. Nothing else is left out. This is the section no other tool shows you, and it is the reason to believe the eleven findings above it — a run that names what it went looking for and did not find is a run whose silence elsewhere means something.the “What we could not establish” section of rsk research result --run x2-c09-q1, verbatim · docs/PACKAGES.json: QUAL-REPORT (de-duplicated gap lines) and QUAL-READ (#1203, PDFs and query-focused windows), both merged 2026-09-21

next: the same machine at fleet scale, the reference — every part of it that is merged — or try it.