I agree that AI needs alignment and validation, and this is already been worked on [*1]. I'll assume below that this is a given.

For the rest, AI itself may be able to handle better than most humans. It already has good idea about what being human is [*2], understands that this isn't a one-dimensional problem, can do balancing like humans would, find more direct/efficient paths than humans. For many things I discuss with AI, I find that it already reasons much better than most humans (not even considering that AI knows so much more than any human).

I think we over-index on "the same prompt will provide different plans". Humans would do the same too. Humans, working in isolation or with collaboration can self-correct, but the same applies to AI.

>> On what ground would you give the machine a pass?

The benchmark I hold is humans themselves. Humans currently have the pass, and have had it since history. Yet, there are many irrational decisions everywhere around.

I hear complaints that AI hallucinates. Yes, it does. Humans do too, and more often than they are willing to admit. The concept of 'god' may entirely be a hallucination (i.e., something not supported by facts). There is no good scientific evidence of prayers working, yet many humans believe in the same.

The real issue is that AI seems to learn and hallucinate in a different way than humans -- it sometimes makes some very silly mistakes. No disagreement, we need to make it better. The pace at which AI can become better however could easily surpass the speed at which humans learn or change. I am not suggesting we give AI the pass till it becomes good enough, there's enough progress on alignment, etc.

>> Do you really think our ancestors chose to cut off their legs in abandonment when they figured that some animals could outpace them?

Great! Here lies an important point. Between humans and animals, we have nature's evolutionary processes, survival of the fittest, ... Depending on how one sees it (and this is a real debate in my mind), we could use AI to enhance our survival and progress faster, or we could see AI as an enemy (like it is another species) and compete.

For some people, the goal is exactly advancements of humans. I, so far, see it as evolutionary progress, whatever form it takes.

[*1] Whether we are doing enough for that or not, is a valid debate.

[*2] AI cannot experience it, it cannot 'know' it in the human sense of knowing if that means something different, but it could emulate well enough.

What's the future for pure math research in the age of AI?