I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident "probabilities" that are complete bullshit. You can't see the underlying reasoning... but it's incredibly tempting for people who want to place faith in them. Rather than even trying to understand the complex system at play, it's easy to give up on trying to find reason, and just accept the second- or third-tier outputs of massively complex things that, on the first tier, are not necessarily reliable sources of truth.

Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.

Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.

Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?

Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.

Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.

Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.

Here's your Jev. Note the preponderance of nouls ;)

https://www.ebay.com/itm/146783842141

If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.

Contrastive Language Models