Are Foundational Model Companies Overvalued?

Random thoughts on the valuation of foundational model companies

Companies like OpenAI, Anthropic, and Google are currently leading the development of foundational models, which are highly valued for their potential to unlock fast value across a range of industries. OpenAI’s ChatGPT, Github co-pilot, Bing’s search assistant, and Jasper’s writing assistant are examples of the value of foundational models in action. However, this moment may be short-lived, as new technologies and developments are likely to drastically reduce the cost of building and deploying foundational models.

Firstly, the cost of training, deploying, and running distilled foundational models is expected to be significantly lower than that of traditional models. Distillation involves using a large, complex base model to train a smaller, more efficient model that performs just as well. This process is much cheaper than training a new model from scratch and is becoming increasingly common. As distillation becomes more widespread, the cost of training models will continue to decrease, making it possible for companies of all sizes to build and deploy sophisticated deep learning models.

Secondly, while research costs will remain a significant expense for foundational model companies, it is expected that training and inference costs will eventually go to zero. Building a foundational model requires extensive research to develop a deep understanding of the data and how it can be effectively modeled, which will continue to require significant investment in research. However, as the field of deep learning matures, more and more people will train their models on the same publicly available datasets, such as Wikipedia. This will reduce the cost of acquiring and preparing training data, making it more affordable for companies to train their models. Additionally, advancements in hardware and software engineering will continue to drive down the cost of inference, and code optimizations and specialized hardware will enable more efficient training. As a result, the cost of training and inference will eventually go to zero, making it possible for companies of all sizes to build and deploy sophisticated deep learning models at a lower cost.

Finally, foundational models are expected to become increasingly vertically integrated, without accruing value to foundational model companies like OpenAI. As advancements in hardware and software engineering continue to drive down the cost of inference and training, the value of foundational models will increasingly be captured by companies that integrate them into their products and services. For example, Replit trained Ghostwriter (their version of copilot) while Neeva trained their own search assistant, without relying on or integrating with OpenAI. This means that foundational model companies will need to adapt and innovate in new ways to continue to drive progress in the field of deep learning.

In conclusion, while foundational models are here to stay, cost structures will evolve, making it harder for Foundational Model companies to exist without providing value added services on top, as others rush to clone these models and vertically integrate them into their own products.

Written with ChatGPT, obviously.