> I find the most passionate advocates of LLMs right now ...

Whether or not that's true, you have to keep in mind that it's not true of everyone.

I'm a passionate advocate of LLMs. I implemented my first neural net, for image recognition, in the late 1980s (based on Bart Kosko's work, I'm not claiming to have invented anything.) Now, I use LLMs daily for coding and other tasks. I don't usually "vibe code" unless it's a simple enough project that the code quality really doesn't matter - a simple example would be data integration, where as long as the data in the target system matches the data in the source system, I don't care much about how the sausage in between is made.

But otherwise, I'm approving or denying every tool call, like a caveman, and when they mess up, I explain to them why. I interrupt them and redirect them when they need it. I change code myself and let them know. With all that, they're still incredibly useful - more useful than most "junior programmers", realistically. No human software developer can hope to match LLMs on many significant dimensions of development. But that doesn't mean experienced and competent humans have no role, at least currently.

The disconnect here is that your characterization of "the low quality of their output" is too simplistic. The quality of their natural language writing is different, for the kinds of reasons stated in the OP. Some of the accusations of "lower quality" really boil down to "too compressed for humans to easily understand", again as the OP points out. And the quality of their code has a lot to do with how they're prompted.

One of the biggest barriers to good LLM output - both writing and code generation - is that they're trained on data generated by humans who fit George Carlin's observation that half of them are stupider than average. When it comes to coding, on average they don't have enough education not to think that imperative code with minimally-restricted mutation all over the place is a good idea. They think dynamic typing, so that code can't be statically verified, is a good idea. They reflect our own collective limitations. If you recognize that and redirect them to sources of good practices, they can be much better.

In short, they're much more of a mirror than most people realize or admit.

With all this noise, it should not be forgotten that we appear to have figured out roughly how a significant component of intelligence works, and have real life, useful proofs of concept. All the technical criticism, to me, is just coming from people who don't seem to understand how intellectual advancement works. The political criticism is another matter.

Why is AI generated language so hard to understand?