Where is AI headed? 2024 and beyond

15 months ago, in October 2022, there was barely any talk in the mainstream about large language models, or ‘generative AI’. But then chatGPT released the next month, and we’re all aware of what happened after. chatGPT gained 100million users in ~2 months, and suddenly everyone was talking about AI, resulting in the subsequent Cambrian explosion of ‘generative AI’ products and companies.

But that’s only half the story – regardless of how we’ve all used chatGPT for work or fun, we’ve seen AI for X, where X was customer support, design, software engineering, and I won’t even get started on all the AI chatbots. Some claim that AI is going to change everything, and some claim that AI is just another fad.

Regardless of what your stance is, as you think about this technology in its current state, it’s important to emphasize that this is the worst it’ll ever be. If we go back 5 years in time to when GPT-1 was released, few if any would’ve predicted that the technology would come this far in such a short period of time. Now think about the same, but extrapolate this to 5 years in the future.

The scaling laws in 2020 have shown us that the recipe to improving AI capabilities boils down three main ingredients: data, model size, and compute. As each of these parameters are scaled, the model’s capabilities correspondingly improve.

Now consider the above laws in the practical context. The training budget of GPT-4 was $100 million. While that seems like a humongous amount of money, consider that Meta and Alphabet generally spend over $30 billion annually on R&D. Even if we assume a future model’s training will cost $10 billion, that’s a 100x increase in costs from the present. Additionally, with the amount of spending from chip companies like NVIDIA, we are likely to see atleast 1–2 orders of magnitude in compute efficiency improvements. If we just take both of these into account and assume no further architectural breakthroughs (which is unlikely), we’re still looking at 1000–10000x increase in compute scale compared to GPT-4.

To imagine the improvement in capabilities that this scale can bring, let’s conduct a simple thought experiment. Compare GPT-4’s capabilities to GPT-2 (which could barely produce semantically correct paragraphs), and extrapolate that amount of capability increase to GPT-4. While this isn’t definitive, there hasn’t been strong evidence that proves that this improvement is not possible, and the scaling laws continue to hold.

Hence, as you think about the current AI paradigm of large language models, I urge that you see beyond the chatbots, question-answering machines and the co-pilots, to instead notice what they actually are: systems that can reason like us. As dramatic as it may sound, the fact is that we’re building systems that have been approaching (and will soon go beyond) human-level capabilities, and are not showing signs of slowing down.

Some of the key missing pieces with these models today are long-term memory, ability to reliably use tools, and the ability to plan and improvise. But all of these are problems that can and will be solved, and we’re going to see AI capabilities continue to improve exponentially.

To put this exponential improvement into perspective, OpenAI believes that Superintelligence, which refers to AI systems that are dramatically more capable than us, could arrive this decade. Ponder about this for a few seconds, putting aside the possibility of it, to instead think about what the consequences of such a system would be. The world’s systems are nowhere near prepared, and we do not have much time to prepare for this.