According to Bryan Caplan, an economics professor at George Mason University, ChatGPT has made remarkable progress in just three months by improving its score from a D to an A on his economics test.
Caplan was surprised by ChatGPT’s improvement, which resulted in him potentially losing the first big bet of his life. In January, Caplan gave ChatGPT questions from his fall midterms, which test students’ understanding of economics rather than requiring them to memorize textbook information.
The old version of ChatGPT scored a 31 out of 100, which was equivalent to a D and well below Caplan’s 50% median. Its responses showed a lack of understanding of basic concepts, such as comparative and absolute advantage, and were more political than economic.
ChatGPT’s lackluster performance disappointed other academics as well. Although it passed a Wharton Business School exam in January, its professor said it made “surprising mistakes” on simple calculations. Caplan was so underwhelmed by ChatGPT’s responses that he bet an AI model wouldn’t score an A on six out of seven of his exams before 2029.
However, ChatGPT-4 has since made remarkable progress. Caplan was stunned by its score of 73%, which was equivalent to an A and among the best scores in his class. ChatGPT’s paywalled upgrade aimed to address some of the early issues with the beta version, GPT-3.5, making it 40% more likely to return accurate responses and better able to handle more nuanced instructions.
ChatGPT’s improvements were obvious to Caplan, as it was able to give clear answers to his questions, understand principles it previously struggled with, and score perfectly on explaining and evaluating concepts championed by economists such as Paul Krugman.
“The only thing I can say is it just seems a lot better,” Caplan said.
Caplan has now given the bot new tests it hasn’t seen before, and it has performed even better than its previous 73% score.
As a result, Caplan is more confident that he will win his next AI-related bet. He currently has a bet with Eliezer Yudkowsky, an AI doomer who has clashed with Sam Altman, the creator of ChatGPT. Their wager is that AI will cause the end of the world before January 1, 2030.
“I’m probably going to lose this AI bet, but I am totally on board to do a bunch more end-of-the-world AI bets because I think these people are out of their minds,” he said.
