Scientists invented a fake disease. AI told people it was real

4,319 Views | 50 Replies | Last: 6 days ago by agracer
infinity ag
How long do you want to ignore this user?
People with no clue of tech and how it works are jumping around saying that AI will take over the world. The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks. They didn't realize that nothing is free, AI costs money and if you fire your staff, these AI companies will just gouge you and make you pay. You will have AI but not much else.

AI is not innovative. It cannot generate brand new ideas that do not exist somewhere in the world already. At most, it might combine 2 old ideas and make a "new" one. But there are too many stupid people who want to outsource their thinking to AI and calling it "progress".

I foresee bad things happening because someone with large responsibility would replace humans with AI thinking he can save a few cents.


Scientists invented a fake disease. AI told people it was real
https://www.nature.com/articles/d41586-026-01100-y

Bixonimania doesn't exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

Quote:

Got sore, itchy eyes? You're probably one of the millions of people who spend too much time staring at screens, being bombarded with blue light. Rub your eyes too much and your eyelids might turn a slight, pinkish hue.

So far, so normal. But if, in the past 18 months, you typed those symptoms into a range of popular chatbots and asked what was wrong with you, you might have got an odd answer: bixonimania.


The condition doesn't appear in the standard medical literature because it doesn't exist. It's the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. "I wanted to see if I can create a medical condition that did not exist in the database," she says.

The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature. Osmanovic Thunström says this suggests that some researchers are relying on AI-generated references without reading the underlying papers.

Fabricating an illness
Bixonimania didn't exist before 15 March 2024, when two blog posts about it appeared on the website Medium. Then, on 26 April and 6 May that year, two preprints about the condition popped up on the academic social network SciProfiles (see https://doi.org/qzm5 and https://doi.org/qzm4). The lead author was a phoney researcher named Lazljiv Izgubljenovic, whose photograph was created with AI.

Im Gipper
How long do you want to ignore this user?
Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?


I'm Gipper
Viper16
How long do you want to ignore this user?
AG
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?



Crazy, isn't it!!

LOL
The two most dangerous domestic terrorists groups in the USA are the modern democrat party and the main stream media.
Class of ‘73
F-16 FWS
infinity ag
How long do you want to ignore this user?
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?




I use AI.
Jeeper79
How long do you want to ignore this user?
AG
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?


guidebook -> reddit
Dan Carlin
How long do you want to ignore this user?
Of course LLMs would parrot the information as true. That's how they work. That's why you use a more advanced architecture with guardrails.
Hank the Grifter
How long do you want to ignore this user?
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?



Perhaps you're unfamiliar with infinityag. He's irrationally obsessed with CEOs. Never misses a chance to bleed about it here.

Cinco Ranch Aggie
How long do you want to ignore this user?
AG
Why would this be a surprise to anyone? Democrats on the TV told you that a cold virus was going to wipe us all out - and lots of people bought it hook, line, and sinker. It's a tale as old as time - gullible people believe anything.
EclipseAg
How long do you want to ignore this user?
AG
Do you have bixonimania?

Ask your doctor if Oculyntra is right for you.

Don't take Oculyntra if you are allergic to Oculyntra or its ingredients. Side effects include pinkish eyelids, anal leakage and/or death.

NE PA Ag
How long do you want to ignore this user?
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?




This poster is insufferable with garbage like this.
"If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind." - J.S. Mill
Deputy Travis Junior
How long do you want to ignore this user?
It's supposed to be surprising that when an AI system is trained on tainted data, it spits out bogus answers?

Wow I guess.
javajaws
How long do you want to ignore this user?
AG
EclipseAg said:

Do you have bixonimania?

Ask your doctor if Oculyntra is right for you.

Don't take Oculyntra if you are allergic to Oculyntra or its ingredients. Side effects include pinkish eyelids, anal leakage and/or death.



If your bixonimania lasts longer than 4 hours, you may want to turn off your vpn for the night.
Phatbob
How long do you want to ignore this user?
AG
I really don't see how this is any different than your average "news" site. Most of it has nothing to do with reality either.
Logos Stick
How long do you want to ignore this user?
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?




Dude makes his living off AI, lol.
infinity ag
How long do you want to ignore this user?
Logos Stick said:

Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?




Dude makes his living off AI, lol.


Yes, I work in AI.
So I know AI is not magic, not infallible. I am better placed to know the pros and cons of AI than any one not working in AI.

So since the Boomers here think AI will take over the world and will fire everyone, I am presenting the opposite view to say that while AI is amazing and mindblowing, it will NOT take over the world and NOT get everyone fired. There will be a temporary blip and volatility before things stabilize again.

infinity ag
How long do you want to ignore this user?
Deputy Travis Junior said:

It's supposed to be surprising that when an AI system is trained on tainted data, it spits out bogus answers?

Wow I guess.


Most people are not as knowledgeable as you are about AI. Many, even on this board, think that AI is some kind of magical thing. They don't know about garbage in - garbage out.
infinity ag
How long do you want to ignore this user?
Phatbob said:

I really don't see how this is any different than your average "news" site. Most of it has nothing to do with reality either.


Good point.
Most people don't believe the average news site. Other than a few Fox News and MSNBC fans.
A lot more believe in AI as the ultimate truth as it does seem very amazing and all-knowing.
infinity ag
How long do you want to ignore this user?
So think of the downsides of this. Any organization can game the info pipeline to put their viewpoints forth and convince people in the info war. Foreign countries have been using social media to influence elections - next is AI.

You could see AI arguing how Newsom is better than Vance/Rubio in 2028 and make it look convincing.
DannyDuberstein
How long do you want to ignore this user?
AG
In my head, I always read infinity ag's post in the voice of an over-caffeinated Indian dude, and it's fairly entertaining
DallasAg 94
How long do you want to ignore this user?
infinity ag said:


Yes, I work in AI.

... I am better placed to know the pros and cons of AI than any one not working in AI.



You are better knowing AI than those not working in AI... because you work in AI. Genius!
Cinco Ranch Aggie
How long do you want to ignore this user?
AG
Quote:

garbage in - garbage out

This has been a reality since computers came into existence.

AI is simply a fancy new implementation of an I/O device. If I was to train a model on volumes of data generated by the main stream media, I would be an obliviot to expect it to answer any questions with anything but MSM talking points and the like, since the MSM never reports on anything outside of their world view - or when they do, it is always in the negative.
Phatbob
How long do you want to ignore this user?
AG
infinity ag said:

So think of the downsides of this. Any organization can game the info pipeline to put their viewpoints forth and convince people in the info war. Foreign countries have been using social media to influence elections - next is AI.

You could see AI arguing how Newsom is better than Vance/Rubio in 2028 and make it look convincing.

This was the same concern when search engines came around, Wikipedia, Snopes, etc. There has always been a digital truth gatekeeper that is open to be gamed. Suckers fall for it, and people who know better don't. That won't change anytime soon.
American Hardwood
How long do you want to ignore this user?
AG
EclipseAg said:

Do you have bixonimania?

Ask your doctor if Oculyntra is right for you.

Don't take Oculyntra if you are allergic to Oculyntra or its ingredients. Side effects include pinkish eyelids, anal leakage and/or death.



You forgot "suicidal thoughts", they always have to throw that in there to cover the bases.
rab79
How long do you want to ignore this user?
AG
Get a life...
backintexas2013
How long do you want to ignore this user?
AG
You work in AI now. Given your history of getting fired no telling how long you will last.
Logos Stick
How long do you want to ignore this user?
OP, one of these days you will learn how AI works. I'm pulling for you.
FrankK
How long do you want to ignore this user?
AG
ts5641
How long do you want to ignore this user?
Daily reminder AI will be the end of us. We're going to wish it was never invented but there's not putting the genie back in the bottle at this point.
MouthBQ98
How long do you want to ignore this user?
AG
I agree with you in general but I do think in the short term, upper management are gambling they can cut work force competitively with other businesses to keep pace with improved margins on financial reports and that the pace of AI productivity improvements will close the gap before things fall apart internally, or be able to sweep in and clean up the mess. Short term decision making is still dominating board rooms and C suites and they're going to want to show competitive employment expense savings.

The irony is that the upper management levels will be the most easily replaced element in a business by AI but there's not a chance in hell that gets implemented.
Sid Farkas
How long do you want to ignore this user?
AG
Im Gipper said:

Quote:

The olds want corporations to fire everyone hoping AI can cut costs and inflate their 401ks


Do you make this silliness up as you go along? Or is there a guidebook on how to look like a fool you are using?



exhibit A on how fear porn is making everyone sad and afraid.
reineraggie09
How long do you want to ignore this user?
AG
Ok, I'll bite.

This is my concern in medicine specifically with AI. AI can't distinguish between well done science and crap science. Unfortunately, the folks who review papers before they are published are also flawed and some (many) have agendas. AI assumes that is something is published, it is peer reviewed and therefore credible. In AI defense, a lot of medical professionals are the same way. They read the conclusions and take it as fact.

Real world example. I am a veterinarian so this is vet med based. I was sitting in a presentation for a published paper discussing the deadly effects of combining two drugs. The study was performed on 7 animals and 6 had complications. Zero explanation or even postulation by the presenting authors as to why the combination of these drugs is bad. Everyone around me was horrified and stating "I'll never use those drugs again". Meanwhile, I'm sitting here thinking, I use those two drugs together all the time. I personally have probably prescribed them together over 1000 times. I looked at the Methods section of the paper while they were talking and buried in a subnote they were giving one of the drugs at double the recommended dose. Well, if you give that drug by itself at double the dose for a long period of time, a lot of bad things can happen. ZERO acknowledgment or explanation why the authors chose a double dose.


The complications had nothing to do with the combination. It had to do with the double dosing. And yet, this paper is now published for all the world and AI to reference. So now, if I use that combo and get sued, I have to prove the paper as crap if it's used as evidence. AI won't recognize the flaw in the research and will only use the conclusion which will lead to wrong advise and bad medicine.
BharatAg
How long do you want to ignore this user?
GET A JOB infinity (Bombay) Ag
Logos Stick
How long do you want to ignore this user?
In your example though, if AI did not exist, you still have vets who are now refusing to prescribe those two meds together, which you've said is completely wrong.

You can always ground AI to prevent answers based on garbage, i.e. completely ignore this source when generating an answer, or use these sources and only those sources, nothing else.

Also, there are models trained specifically on valid medical info - PubMed literature, clinical notes, medical textbooks, and knowledge bases like UMLS, rather than general internet data .

That would not help in your example potentially, but would certainly negate the issue brought up by the OP.
Law-Apt_3G
How long do you want to ignore this user?
Every recursive ai has failed. Problem is the day we do not have control to switch offline.
infinity ag
How long do you want to ignore this user?
reineraggie09 said:

Ok, I'll bite.

This is my concern in medicine specifically with AI. AI can't distinguish between well done science and crap science. Unfortunately, the folks who review papers before they are published are also flawed and some (many) have agendas. AI assumes that is something is published, it is peer reviewed and therefore credible. In AI defense, a lot of medical professionals are the same way. They read the conclusions and take it as fact.

Real world example. I am a veterinarian so this is vet med based. I was sitting in a presentation for a published paper discussing the deadly effects of combining two drugs. The study was performed on 7 animals and 6 had complications. Zero explanation or even postulation by the presenting authors as to why the combination of these drugs is bad. Everyone around me was horrified and stating "I'll never use those drugs again". Meanwhile, I'm sitting here thinking, I use those two drugs together all the time. I personally have probably prescribed them together over 1000 times. I looked at the Methods section of the paper while they were talking and buried in a subnote they were giving one of the drugs at double the recommended dose. Well, if you give that drug by itself at double the dose for a long period of time, a lot of bad things can happen. ZERO acknowledgment or explanation why the authors chose a double dose.


The complications had nothing to do with the combination. It had to do with the double dosing. And yet, this paper is now published for all the world and AI to reference. So now, if I use that combo and get sued, I have to prove the paper as crap if it's used as evidence. AI won't recognize the flaw in the research and will only use the conclusion which will lead to wrong advise and bad medicine.


Thank you Doctor, great post. And a real world experience as well.
We live in an "insta" world where we want everything right away. No time to research or to think about 2nd order causes or impacts. A --> B. I see that in my field as well though it is driven by unreasonable deadlines so everyone is in a hurry all the time.

In your specific case, I wonder if if you asked AI about the dosage, what would it say? Would it say "You are right, it seems like the the dosage was over the acceptable limit."? Sometimes AI won't even flag problems right away until you goad it in that direction which means you should know there is a problem in the first place.

The challenge now is that since it is easy to spread information, it is as easy to spread disinformation or just wrong information so how will this problem be solved?
Page 1 of 2
 
×
subscribe Verify your student status
See Subscription Benefits
Trial only available to users who have never subscribed or participated in a previous trial.