Misled by Euphoria
The need for greater transparency in AI development
I stumbled into this document (which I wrote in 11/2022 shortly after the FTX debacle) when I was going through my Google Docs. history. I’m publishing it now because it rings truer with the release of chatGPT and the widespread euphoria of AI capabilities. This was an exercise to solidify my half-formed thoughts, and this is the first time I’m publishing my thoughts on a public medium. Any feedback welcome!
Back in late 2021, if someone rang warning bells about FTX and made negative claims about SBF, or mocked the Sequoia’s glowing spotlight article, the person who made those negative remarks would have been considered to be a buzzkill, a pessimist, or something worse. Almost no one would have paid heed to an FTX Cassandra, and rightly so, because the evidence in public light overwhelmingly showed how FTX was a huge success, how it acquired so many companies, how its founders are prodigies, how they’ve raised from the best VCs, how they’re significantly contributing to EA. But all of this was a red herring while SBF and his team were defrauding investors by using investor funds to win big by risking it all, which collapsed just an year later.
This reads like a parody, but it’s on the Sequoia website. What a joke sequoiacap.com/article/sam-ba…
It is heartbreaking how many people lost their life savings, and all this because they’ve believed in a trusted company, one that was too-big-to-fail (they were valued at $32 billion before they crashed to $1 and filed for bankruptcy). We’ve seen this happen time and again, with 2008, with Theranos, with the dot-com bubble, and a large part of this arises because we humans as we are, can be gullible, and euphoric when it comes to optimistic ideas. This is good when the outcomes in reality match the euphoria, but that’s not the case many times, and we then end up driving off the cliff.
But although this causes major heart-burn and misery for many, we always find our way out of it because these are not catastrophic (at least not for the entire civilization), and because we are extremely resilient. But we may not get a second chance every single time, AI being one of those cases. If we get zealous about reaching AGI as soon as possible, and the euphoria caused by this potential incentive dislikes the idea of prioritizing AI alignment since it can constrain progress, we may keep making faster and bigger forward strides, until the AI pulls the rug from beneath us.
Till today, although I’ve seen many people credit Eliezer Yudkowsky’s ideas, I never really got his constant doom about how we’re driving straight into the ditch because we’re building something without sufficient guardrails (can we ever build sufficient guard rails?) and that may someday end us. But today, I’ve realized how much strength it takes to switch your view from being euphoric to becoming Cassandra, and continuously talking about the hazard, while people may hate you for it because the general perception in the field is euphoric. But he’s right, and we need to figure alignment if we want to survive.
To state one instance: OpenAI, the current leader in applied AI, continually reiterates that its mission is to ensure AGI significantly benefits humanity (and I believe that to a certain extent). But it begs the question – how can we actually ensure organizations and entities are aligned without possessing sufficient information about them, because most organizations never share (and generally rightly so) important information in public.
Can there be something along the lines of of zero-knowledge proofs, where the general public, despite not obtaining any revealing information about the organizations, would still be able to definitively confirm if these entities are on the right path, or if they’re en route to drive off the cliff? I realize that it’s easy to imagine how a system like this could also be abused by competitors/ adversaries to circuitously obtain revealing information.But is there a way we can build a system like this whose net effect would be positive? Pagerank is a way to quantify reputation. How can we quantify trust?