OpenAI’s chatbot, ChatGPT, has garnered a lot of attention and investment due to its impressive abilities. However, the CEO of OpenAI recently cautioned that the research strategy that led to the development of ChatGPT has reached its limits. It is uncertain where future advances in AI will come from. OpenAI has achieved remarkable progress in language-based AI by scaling up existing machine-learning algorithms to unprecedented sizes.
The latest project, GPT-4, was likely trained using trillions of words of text and thousands of powerful computer chips, costing over $100 million. Despite these advancements, the CEO’s warning suggests that a new approach is needed to achieve further progress in AI beyond simply increasing the size of models.Sam Altman, the CEO of OpenAI, has stated that larger models are not the future of AI progress.
He believes that the era of using massive models to improve AI performance has come to an end, and that advancements will come from other approaches. Altman made these comments at an event held at MIT, stating that future AI models will be improved in other ways beyond size. This suggests that OpenAI will shift its focus towards developing more efficient and practical models, rather than simply increasing their size.
Altman’s statement presents a surprising development in the competition to create and utilize new AI algorithms. Since OpenAI introduced ChatGPT in November, Microsoft has incorporated the technology into a chatbot for its Bing search engine, while Google has released a competing chatbot called Bard. This has led many individuals to explore the potential of these chatbots for various work and personal tasks. However, Altman’s assertion that bigger models are no longer the solution to advancing AI technology suggests that future developments may take a different approach.
Several well-funded startups such as AI21, Cohere, Character.AI, and Anthropic are dedicating significant resources towards building larger algorithms to compete with OpenAI’s technology. These companies are trying to catch up with OpenAI’s impressive advancements in AI that work with language, such as the initial version of ChatGPT that was based on an upgraded version of GPT-3. However, users now have the option to use ChatGPT powered by the more advanced GPT-4.
According to Altman’s statement, it seems that the enlargement of the models and feeding them more data to achieve better results is no longer OpenAI’s focus. This may indicate that GPT-4 might be the last significant achievement to emerge from this research strategy. However, Altman did not mention any alternative research strategies or techniques that OpenAI might adopt instead. In the paper that explains GPT-4, OpenAI states that the returns from scaling up the model size are decreasing. Additionally, Altman pointed out that there are physical constraints to how many data centers OpenAI can build and how fast they can construct them.
Nick Frosst, who previously worked on AI at Google and is now a co-founder at Cohere, agrees with Altman’s view that continuing to scale up AI models may not lead to further progress. Frosst suggests that advancements in transformers, the type of machine learning model used in GPT-4 and its competitors, may not depend on increasing the size of the models. He believes that improving transformers can come from exploring new AI model designs or architectures and tuning them based on human feedback. Frosst thinks these are promising areas that many researchers are already exploring.
OpenAI’s family of language algorithms comprises an artificial neural network, a type of software inspired by how neurons work together, that is trained to predict the words that follow a given text string.
