14% of the Web Is Actually Dead — But Not How You Think (We Scanned 10M Domains)

Originally published at crawlora.net. When you hit a dead URL in production, do you know whether the domain is gone — or whether an anti-bot system just blocked your crawler? They look identical from a failed request, but they're completely different failures, and most tools don't tell them apart. W