In 2019 Rich Sutton wrote the small essay "The Bitter Lesson". He points out that the most effective strategy in AI the past 70 years is to simply use more computation. This is a bitter lesson because for decades AI researchers thought that if we just found the write algorithm we'd have human level intelligence with the computation we had.
Looking back, that was extremely optimistic, especially for people working on AI in the 60s and 70s. By the end of the first decade of the 2000s, we finally had consumer computers that were 64-bit, with memory, storage, and processors that could begin to effectively implement some of the neural networks that had been developed decades earlier. This led to a revival in artificial intelligence research, and neural network topologies developed quickly.
In 2017, the transformer architecture was discovered and led to a revolution in AI technoology. Almost all the research and development and money that has been spent has been spent on scaling LLMs (Large Language Models). This scaling has been incredibly effective, and has essentially proven the Bitter Lesson correct.
However, there is an even more bitter lesson, the Bitterest Lesson. And that is this:
Efficient algorithms always beat brute force at scale.
While we will get better returns by scaling our LLMs, there will be ever diminishing returns. The only way forward is research into breakthroughs on the level that transformers were. We absolutely know there are more efficient algorithms how there to develop intelligence. How do we know? From the human brain.
The human brain consumes about 20 Watts continuously. Over 22 years the average brain has consumed about 3,857 KWh. That's a good amount of time for the average person to grow up and get a bachelor's degree. Of course, a person is only awake for about 2/3 of their life, and a significant amount of energy is spent on things other than conscious thought. Let's be generous and say that 100% of brain energy is used for concious thought and learning. It's estimated that OpenAI used somewhere between 52,812,500 KWh and 62,318,750 KWh of energy.
This means OpenAI used around 13,693 and 16,157 times more energy than the human brain to become competent. Now this is not a perfectly equivalent comparison. The AI is better at some things, and a 22 year old human is better at other things. However, the fact that we have to spend thousands of times more energy to train an AI compared to a human means that the algorithms we're using for intelligence are not efficient.
There are definitely other algorithms out there that are more efficient, and more effective that the current transformers we're using right now. When those are found there will be a huge explosion in intelligence, and it will have enormous impact on society; far greater than what we currently have.