I find this perspective surprising coming from someone who's done ML for a decade.

How much of a good machine learning system in production is really "the model"? How many lines of code are dedicated to constructing features and preprocessing them? How many lines are needed to process the decision emitted by the model in context of whatever else is going on? How often does the customer interact directly with the inference/prediction from a machine learning model?

My experience has been that model-fitting and model inference very often ends up being part of a bigger system anyway. And even if you do end up with some kind of super-duper all-purpose predictor model, it's rarely the thing you start with, instead you work up to it over weeks or months of iteration. So why should we put so much privilege and esteem on the matrix transformatinos or the decision tree, when they are really just part of a bigger system that receives some input from somewhere and produces useful output for someone?

The same holds for AI in my opinion. An AI system is a system that accepts inputs and produces useful outputs. LLMs are borderline miraculous in that an AI system built around an LLM can do so much more than anything that came before, and you can push a lot of amazing things right down into the tensors inside the LLM. But how far can that go? I don't think we even have an answer yet, because it's all so new and every time it looks like it's going to plateau, the big labs come out with another big step forward.

But there are other questions too, which should look familiar to anyone in industry. Do we need to do it that way? Should we do it that way? If our goal is to build a useful AI system that balances power and cost, then maybe we shouldn't. What if we can make models 10x bigger and spend 10x on RLHF and achieve the behavior we're talking about entirely in tensors. OK, so what if we can? We can also run a collection of little models in a carefully designed harness and achieve that behavior right now, today, at a much lower cost both in terms of up-front investment and marginal cost per unit of work completed.

I get it from an elegance perspective, but I guess I have completely the opposite mindset. Industry has bled me dry of any desire for elegance. The stuff I build now is stupid and crude but it makes our customers happy and costs peanuts to run and that's exactly what my job wants from me. So I say let's commoditize LLMs and start being clever about how we use them. It's not that we shouldn't also push the frontier in size and power, but unless things change w.r.t. the actual costs involved, it doesn't really seem like the best way forward in the short term for achieving smarter AI systems without also making costs go up and up forever.

The bitter lesson still holds of course. But eventually you have to pay for all that power. And you might not want to shell out $20/1M just so you can be sure your LLM can detect uncertainty in its own context and be more honest about when it doesn't know anything about some obscure record label.

Human mathematicians are being outcounterexampled