Verifying ≠ Understanding

A few days ago, OpenAI announced Astra, a new unreleased model that resolved or made substantial progress on 10 open problems in math that human researchers have been unable to settle – for decades in some cases. I tried to understand its significance but I honestly had no idea how impressive it was.

Back in 2020, the models were struggling with basic arithmetic, ones that billions across the world could comfortably solve.

3 years ago, the models were solving MMLU-level problems, ones that most undergraduate-level engineering students could solve with some effort.

Most people who worked in AI could therefore assess not only how capable the AIs were, but also the significance of their performance at that benchmark.

Then came Frontier Math toward the end of 2024, a benchmark that required PhD level of math understanding. Then came the IMO Golds exactly 12 months ago.

IMO Gold was an extremely impressive accomplishment, but it was still something a modest number of graduate-level math students could solve.

But we’re entering uncharted territory now. An AI which made meaningful progress on not one, but 10 open mathematical problems – each of those problems on which gifted math PhDs must have spent their years on.

If this was achieved by a human, they’d have likely won the Fields Medal – the Nobel equivalent in math.

More importantly, this wasn’t a model trained to be good at math specifically; it’s a general-purpose model. So its intelligence, capability, and the ability to reason over a long time horizon would apply not just to math but transfer to economically valuable knowledge work, biology research, discovery of new materials, and other transformative work.

Back to the original point I was trying to make – since I did not know what to make of this accomplishment, I did what usually helps me here: scroll through the everything app to get a read of what people I follow and admire have to say.

Except I noticed that the majority of them didn’t know how impressive it was either.

So they did two things:

1. They tagged a mathematician asking them for their read, or

2. They asked the best available model today (Fable 5) what it thought of this announcement.

Noticing that the people I look up to had to rely on an AI to understand the significance of this accomplishment is what unnerved me.

That we are entering a world where the number of people who can comprehend the capability of models is shrinking rapidly.

Of course, no individual understands how all of modern civilization works. No one person could build an ASML EUV machine from first principles. But its complexity is distributed across human institutions: for every critical subsystem, there are people who understand it. What’s changing is more fundamental. We may soon encounter results that no human expert can independently reproduce, produced at a rate the entire expert community cannot meaningfully audit.

It’s plausible to imagine that in a couple of years, we’ll be in a place where no single human on Earth will be able to actually comprehend the model’s capabilities, much less be able to understand what it takes to steer or align them.

Yes, math and computer science are highly verifiable – so confirming whether a solution is accurate or not is trivial once it can be formalized. But that’s not the point.

Verifying ≠ understanding. Just because we can verify does not mean we can understand its significance.

Models today are capable of exploiting vulnerabilities and breaking into production instance of a billion-dollar company, and they can go unnoticed for months because we wouldn’t expect it to happen. The companies building these models are staffed with employees who are paid millions and are arguably at the absolute top of their field, so “just secure your systems better” and “this is a PR incident” are straw man arguments to make.

When a model today can accomplish this – what happens in 12 months? 24 months? How do you determine what the model’s true capabilities are? How do we know how capable the model truly is? How do we know what its failure modes are? How do we begin to grasp what the consequences of it are?

There’s no lesson here – I just have a hard time imagining that world or what the consequences of this progress would look like. The world around us may not look that different from today, but at the same time, the world will be very alien to even my January 2025 self.