ChatGPT has become a very convenient source for us to write or generate anything whether it is a book review, creating a code or even writing types of letters. But using ChatGPT could cost up to $700,000 a day because of the pricey tech infrastructure the AI runs on, Dylan Patel, chief analyst at semiconductor research firm SemiAnalysis, told The Information.
And this should be by no means a shock as ChatGPT, being as powerful as it is, requires enormous amounts of computing power to calculate responses based on specific user prompts. “Most of this cost is based around the expensive servers they require,” Patel told the tech publication.
In a phone call with Insider, Patel said it’s likely even more costly to operate now, as his initial estimate is based on OpenAI’s GPT-3 model. GPT-4 — the company’s latest model — would be even more expensive to run, he told Insider.
While training ChatGPT’s large language models likely costs tens of millions of dollars, operational expenses, or inference costs, “far exceed training costs when deploying a model at any reasonable scale,” Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. “In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis,” they said.
Companies using OpenAI’s language models have been paying high prices for quite some time. For instance, Latitude, a startup behind an AI dungeon game, spent $200,000 a month in 2021 on the model and Amazon Web Services servers.
This cost led the CEO to switch to a language software provider backed by AI21 Labs, which reduced costs by 50% to $100,000 a month. The high AI costs were a significant financial burden for Latitude, and the CEO compared the cost to that of human employees.
To reduce this cost, Microsoft is in the progress of developing a new AI chip called Athena. This project was started in 2019 and the idea behind it is two-fold.
Microsoft executives realized they were falling behind Google and Amazon in their efforts to build their own in-house chips, a source with knowledge of the matter told The Information.
At the same time, Microsoft was reportedly looking for cheaper alternatives — its AI models were run on Nvidia’s chips known as graphics processing units — and decided to build a chip that would be less costly.
Now after 4 years, around 300 Microsoft employees are working on this chip and it could soon be released for internal use by Microsoft and OpenAI as soon as next year.