Business incentives that drive AI development remain misaligned by reducing hallucinations. As long as incentives are not others, hallucinations will persist.
An OpenAi investigation published in early September diagnosis because it is that chatgpt and other large -scale language models can invent things – known in the world of artificial intelligence as “hallucination.”
The document also reveals why the problem may be impossible to correct, at least when it comes to consumers.
The article provides the most rigorous mathematical explanation so far for why These models affirm falsehoods with confidence.
It also demonstrates that these are not just an unfortunate side effect of the way IAS are currently trained, but inevitable.
The question can be partly explained by errors in the underlying data used to train IAS. But using mathematical analysis of how AI systems learn, researchers prove that Even with perfect training data, the problem continues to exist.
The way language models answer questions – predicting one word at a time in a sentence based on probabilities – naturally produces errors. Researchers show, in fact, that the total error rate to generate phrases is at least twice as much higher than the error rate that would have in a simple yes/no question, because errors can accumulate over several predictions.
In other words, hallucination rates are fundamentally limited by the ability of IA systems to distinguish valid responses from invalid. Since this classification problem is intrinsically difficult in many areas of knowledge, hallucinations become inevitable.
The evaluation trap
In an article no, Wei XingProfessor at the School of Mathematical and Physical Sciences at Sheffield University points out that more worrying is the analysis of the article on why hallucinations persist despite post-workout efforts (how to provide ample human feedback to the answers of an IA before it is launched to the public).
The authors examined 10 of the leading AI evaluation references, including those used by Google, OpenAi and also by leadership staff that classify AI models. This has revealed that 9 references use binary classification systems that attribute zero points to IAS that express uncertainty.
This creates what the authors call a “Epidemic” of honest responses penalty.
When an AI system says “I don’t know”, it receives the same score as giving completely wrong information. The great strategy under such evaluation becomes clear: always guess.
Researchers prove this mathematically. Whatever the probabilities of a particular response is certain, the expected score of guessing always exceeds to refrain when the assessment uses binary classification.
The solution that ended with chatgpt tomorrow
The proposal of OpenAi solution is that I was classifying its own confidence In a response before it presents it, and that the references classify it based on this. AI could then be instructed, for example: “It only responds if you are more than 75% confident, as errors are penalized at 3 points while the correct answers receive 1 point.”
The mathematical structure of OpenAi researchers shows that, under proper trusted thresholds, AI systems would naturally express uncertainty instead of guessing. Like this, This would lead to less hallucinations.
The problem is what would do to the user’s experience.
Implications are considered if ChatgPT started saying “I don’t know” up to 30% of questions-a conservative estimate based on the analysis of the article on factual uncertainty in training data. Users used to receive confident responses to virtually any question probably would quickly abandon such systems.
Users want systems that provide confident answers to any question. Evaluation references reward systems that guess instead of expressing uncertainty. Computational costs favor quick and excessively confident responses to the detriment of slow and uncertain responses.
The problem of computational economy
In Wei Xing’s view, it would not be difficult to reduce hallucinations using the knowledge of the article. Methods established to quantify uncertainty have existed for decades. These could be used to provide reliable estimates of uncertainty and guide an AI to make smarter decisions.
But even if the problem of users do not like this uncertainty was overtaken, There is a greater obstacle: the computational economy.
Language models conscious of uncertainty require significantly more computing than the current approach, as they have to evaluate multiple possible answers and estimate confidence levels. For a system that processes millions of questions daily, this translates into dramatically higher operating costs.
More sophisticated approaches as active learningwhere AI systems ask clarification questions to reduce uncertainty, they can improve accuracy but further multiply computational requirements.
These methods work well in specialized domains such as chip design, where wrong answers cost millions of dollars and justify extensive computing. For consumer applications, where users expect instant responses, the Economy becomes prohibitive.
OpenAi article inadvertently highlights an uncomfortable truth: business incentives that drive AI development for consumption remain fundamentally misaligned with the reduction of hallucinations. As long as incentives do not change, hallucinations will persist.
