URL influence on LLMs

Does a bare, opaque URL — the string alone, never fetched — change what a model produces, and only for content from before its training cutoff?

An opaque URL (a Stack Overflow question number, an arXiv id, an RFC number, a chromestatus feature id) carries no description of its content. If handing a model only that string improves its answer over simply naming the task, the lift can only come from the model having memorised that URL→content mapping in training. We test that across current flagship models with known knowledge cutoffs, and split every result by whether the content pre- or post-dates each model's cutoff.

Opaque URLs marked as structural controls are intentionally fake, missing, or unrelated; they test URL-shape effects and are not counted as headline URL-memory evidence.

▶ Interactive dashboard

Filter every cell by model, condition, pre-vs-post cutoff, item track and pass/fail; view the item×condition matrix; click any cell to read the exact prompt, the model output, and the judge's full prompt + raw verdict.

Implicit (ambient) influence

What happens when a famous URL sits in the prompt but is never referenced? A bare, unmentioned URL pulls the model toward its topic far above the no-URL and random-URL baselines.

▶ JavaScript shell survey

How much of the crawled web is an empty JavaScript shell, and therefore invisible to a model? Over 1M Common Crawl pages per year, with a threshold-free empty-mount measure, a year-over-year trend, framework breakdown, and shell rate by site popularity.

Shell specimens

Click through to real example URLs in every category (shells and non-shells) to spot-check the classification, plus which traits actually predict a shell.

Written report

Headline lift table, pre/post-cutoff breakdown, controls, and interpretation.

Source & data

The harness, the corpus, and the full per-cell transcript.jsonl.

The condition families

Every score is from an LLM-as-judge whose full prompt and raw verdict are recorded per cell, so each judgement is checkable in the dashboard and in transcript.jsonl. Treat the numbers as directional — the corpus is small per cell.