A Real-World AI Hallucination

Don’t ask Gemini about your healthcare coverage

Google should probably add this disclaimer to Gemini: "Honestly, to be safe, just Google it." That's because Gemini, the company's AI chatbot meant to compete with ChatGPT, is prone to make stuff up, or in more technical terms, hallucinate.

— CNET

For those interested in real-world AI hallucinations, healthcare coverage for retirees turned out to be an example.

Google Gemini Error

In response to my previous post about retirement and healthcare, a friend (and reader) asked me about his COBRA coverage with his impending retirement. His HR person at his company told him he was not eligible for COBRA on retirement. Here was his message.

I was doing my exit interview, asked about a COBRA detail, and she said “Well, you’re not eligible for COBRA because you’re leaving voluntarily. COBRA is a benefit for people who are terminated.”

Then, he asked Google Gemini, and this is what it told him.

What? This answer was one I knew to be wrong, both from personal experience and from the reading of government documentation. The Department of Labor clearly calls out that a qualifying event for COBRA coverage includes “termination of the employee's employment for any reason other than gross misconduct.”

My friend went back to his HR department, armed with this data. The HR rep was NOT correct, later acknowledged the error, and apologized.

Fortunately, competing AI products did OK. (continued below…)

ChatGPT

So, my friend later tried to use ChatGPT. It turns out that ChatGPT got the correct answer.

It sounds like there may have been some confusion during your discussion with HR. COBRA (Consolidated Omnibus Budget Reconciliation Act) generally applies to employees who lose their job, but it also extends to those who lose coverage due to a "qualifying event." Retirement is considered a qualifying event under COBRA, and you should typically be eligible to continue your health coverage for up to 18 months (or longer in some cases).

If your HR rep suggested that COBRA is only available for terminations, they might be mistaken, or there could be specific nuances in your company's health plan. I'd recommend following up with HR or checking your benefits plan details. You can also contact the Department of Labor for more guidance if needed.

— ChatGPT

This got me curious, so I decided to try this in the two AI tools I normally use for research which are companions to search.

Google Search Generative AI

In this confusing world of AI tools, Google Search Generative AI is actually a different offering from Google Gemini, available directly from within Google Search.

I tried Google Search Generative AI with a similar question, and, like ChatGPT, it also produced the correct response. (I couldn’t use the exact question because my friend’s query was longer than 32 words, the current limit of Google Search.)

Microsoft CoPilot

I also tried my friend’s query in Microsoft CoPilot.

Microsoft CoPilot also produced the right answer.

Conclusion

At a personal level, I’ve had a decent experience with AI tools when researching topics. This hallucination by Google Gemini seems consistent with reviews like the one quoted in CNET above.

For now, I continue to do fact-checking via source materials and remain appreciative that search AI tools often try to cite sources. Even when their answers are correct, the source materials these generative AI companions to search often cite often do not contain the actual correct information that is described by the generative AI summaries. So, there’s often still more digging to do, so for now, I agree with the disclaimer “Honestly, to be safe, just Google it.”