> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.

It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.

> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.

It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.

Wiki:

> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11] In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]

As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")

> That's pretty clearly a hype article

It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.

> I'm asking for empirical measurements of timelines, not hypotheticals.

Go and find it then? You haven't bothered looking.

> Lol that excuse is getting really old

You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.

You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.

You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.

Everyone should slow down AI development except for me