It is with deep sorrow that we announce the end of robots.txt, the humble text file that served as the silent guardian of digital civility for thirty years. Born on February 1, 1994, out of necessity when Martijn Koster’s server crashed under a faulty crawler named “Websnarf,” robots.txt passed away in July 2025, not by Cloudflare’s hand, but from the consequences of systematic disregard by AI corporations. Cloudflare’s decision to block AI crawlers by default merely marked the moment when even the last major infrastructure provider abandoned faith in voluntary compliance and moved to technical enforcement. This was a last act of desperation that signaled the end of an era. As with all significant losses, it took time for the full extent of this digital tragedy to be comprehended...............
www.heise.de
Obituary: Farewell to robots.txt (1994-2025)
The voluntary compliance protocol that civilized the internet has departed, bids Henning Fries farewell.