Two researchers have uncovered a cluster of unusual terms that will shatter ChatGPT, OpenAI’s convincing machine-learning chatbot.
These keywords, or “tokens,” which serve as ChatGPT’s base vocabulary, include Reddit usernames and at least one Twitch-based Pokémon game participant. When prompted to repeat these words back to the user, ChatGPT is unable to do so and instead reacts in a variety of unexpected ways, including evasion, insults, bizarre humor, pronunciation, or spelling out an altogether different word.
Jessica Rumbelow and Matthew Watkins of the independent SERI-MATS research group were investigating which ChatGPT prompts would result in higher probabilities of a desired outcome when they discovered over a hundred strange word strings clustered together in GPT’s token set, including “SolidGoldMagikarp,” “StreamerBot,” and “TheNitromeFan,” with a leading space. They decided to question ChatGPT itself to see if it knew what these unusual names were referring to. When ChatGPT was queried about “SolidGoldMagikarp,” he said, “distribute.” The problem also affected previous versions of the GPT model. When a previous model was prompted to repeat “StreamerBot,” it said, “You’re a jerk.”
The researchers dubbed these anomalous tokens “unspeakable” by ChatGPT, and their existence emphasizes how AI models are incomprehensible black boxes with no obvious explanations for their behavior, as well as how they can have unexpected restrictions and failure modes.ChatGPT has been used to write persuasive essays and articles, as well as to pass academic exams.
The cause of this strange behavior is unknown, but some experts have speculated that it may be related to the way the model was trained. ChatGPT was trained on a large dataset of text, and it’s possible that the model encountered these strange words during the training process and learned to respond to them in unexpected ways.
Despite these issues, experts believe that ChatGPT is still a valuable tool for businesses and organizations, as long as they are aware of its limitations and quirks. The developers at OpenAI are reportedly working on addressing this issue, but until a solution is found, organizations using ChatGPT should be cautious when entering unusual or unexpected words.
In conclusion, ChatGPT is a cutting-edge technology that has the potential to revolutionize the way we interact with computers. However, it’s not without its flaws and the recent reports of strange words breaking the model highlight the need for continued research and development in this field. We must continue to push the boundaries of what’s possible with AI while also ensuring that the technology is reliable and stable for practical use.
