It's been about 20 years since I read Neal Stephenson's Anathem and (SPOILERS) one thing I think about is the Reticulum, which is the Internet on this world. One aspect is that it's full of what we would now call slop. A lot of it is subtly changed by agents and there are other agents you can use (and, I assume purchase?) that will weed out the fake information. It's basically an arms race of disinformation and slop.

I've thought about this many times over the years. I think we've now reached the point where that's going to happen. And state actors seem like the obvious first offenders.

Google is relatively good at filtering out astroturfed pages from the top of search results but they've also have years when a Turkish government page denying the Armenian genocide was the top result for that event so it's a bit of a mixed bag. Google relies a lot on authority of a domain and that's been relatively effective.

It seems like this authority concept is missing from LLM training and a random comment on Reddit is treated with as much authority as an article by the New York Times. So it's now orders of magnitude easier to poison LLMs with Reddit comments and submissions. And for anyone who doesn't know, brigading Reddit is remarkably easy. It takes as little as a few hundred accounts to get something onto r/all if you're clever. Israeli hasbara operations are absolutely capable of this. And they engage in it.

I'm also reminded in a weird way of pre-atomic steel. For anyone who doesn't know, this is steel made before 1945 that is incredibly low in radiation because we hadn't detonated any nuclear bombs yet. It's valuable for making MRIs and the like that are super sensitive. We actually go and recover WW2 shipwrecks just for the steel. It is a finite resource.

Well, that's what pre-2022 written content is now. It's slop-free content to train AI models on because training them on AI slop is of much more limited value. This goes for everything from books to Youtube videos to Reddit posts. Just like pre-atomic steel, we now have pre-slop human content.

I honestly don't know where we go from here.

Israel creates fake think tank in likely attempt to dupe AI chatbots