By Shira Ovide
The Washington Post identified in recent days telltale signs of AI in a White House health report that cited nonexistent research. And attorneys keep getting scolded for using AI that generates bogus legal research in court filings.
You might believe that you would never be stupid enough to over rely on AI this way. You’re wrong.
There are decades of research chronicling our tendency to blindly trust powerful software even when we know better — like the proverbial person who follows Google Maps directions off a cliff. New forms of AI, which are simultaneously powerful, seemingly mysterious, error-prone and excessively hyped, turbocharge this problem.
A necessary companion to using modern AI, then, is the humility to anticipate that you will trust the technology even when you know you shouldn’t. Here’s how to fight that tendency:
Automation bias
Many AI technologies say you shouldn’t treat them as gospel.
ChatGPT’s text box says, “ChatGPT can make mistakes. Check important info.” Tesla says you must constantly be hands-on with its advanced car cruise control system that it calls “Autopilot.” Companies that sell facial recognition software used by police departments say the information is for investigative leads and not grounds for an arrest.
The problem, AI researchers say, is that those warnings conveniently ignore how we actually use technology — as machines that spit out the right “answer.”
Researchers have described for decades a series of human biases with computerized recommendations, including what’s sometimes called automation bias, or a tendency to defer to machines even when they contradict your common sense or professional training.
Generative AI systems including ChatGPT and image-generating Dall-E from OpenAI amplify this tendency, said Kate Crawford, a professor at the University of Southern California and author of “Atlas of AI.” (The Washington Post has a content partnership with OpenAI.)
“Generative AI systems have both an authoritative tone and the aura of infinite expertise, which can make users feel as though every response is correct without the need to verify it,” Crawford said.
My favorite example of automation bias: For years, many people trying to pinpoint the location of a computer or phone used by identity thieves, runaway children and online harassers found the location of a single Kansas farm. It was a computer glitch in common online mapping software.
Otherwise sane Americans drove to Kansas to confront the person they believed stole their phone or harassed them online — because a computer spit out the wrong answer.
If you think you’d never do something like that, you’ve learned the wrong lesson. We’re all prone to automation bias, especially when we’re stressed or worked up.
‘Distrust but verify’
Computer programmer Simon Willison has said it helps to assume that chatbots and other new forms of AI are wrong until you prove otherwise, an approach he described as “distrust but verify.”
That’s different from how we’ve come to treat many other technologies. Weather forecasts aren’t 100 percent accurate, but they tend to be reliable enough not to take an umbrella when the forecast is sunny. Mapping directions and Siri goof up constantly, but they’re often good enough, or at least relatively low stakes when they’re wrong.
Generative AI constantly makes up stuff, though, and it’s getting worse rather than better. This knowledge of the inherent flaws is a necessary starting point when you use AI to summarize information, edit your emails and brainstorm.
It’s also necessary to recognize that the wow factor of generating sentences or images doesn’t make the technology intelligent.
“Generative AI may seem like it’s smart because it has a conversational interface, but it is actually the opposite of smart,” said Meredith Broussard, a data journalism professor at New York University who wrote “Artificial Unintelligence: How Computers Misunderstand the World.”
Frank Pasquale, a Cornell Law School professor, has been researching the recent examples of lawyers caught submitting bogus legal citations from chatbots. Lawyers are required to make sure that anything submitted to a court is reviewed by a human beforehand, but like the rest of us, attorneys are tempted by AI shortcuts.
Pasquale said that “concrete repercussions,” such as steep fines or a temporary loss of lawyers’ law licenses, might help override the appeal of using AI and not checking its work.
The temptation that Pasquale described to cut corners shows that there’s a bigger-picture reason behind people pumping out flawed AI-generated material, said Evan Selinger, a Rochester Institute of Technology professor, who focuses on the philosophy of technology.
When your boss or child’s school “expect AI to boost productivity and jobs feel precarious, the pressure inevitably will lead to over reliance on AI tools,” Selinger said. “Many are doing what they believe it takes to survive."
