Well, none of these numbers add up.
Google's stupid actions matter because sooner or later, Google's AI will make its way into search. The company has already been testing this new Google, called the Search Generation Experience (SGE), with volunteers for nearly 11 months, and recently launched a new Google search for the main Google search, even for users who haven't opted in to the test. I started showing his AI answers in the results.
The new Google offers some useful features. But as you can see, sometimes they make up facts, misunderstand questions, give outdated information, or just chatter. Even worse, researchers have found that AI often elevates low-quality sites as reliable sources of information.
Normally I don't review products that aren't finished. But his test of Google's future has been going on for nearly a year, and the choices being made now will affect how billions of people get information. Also in question is the central idea behind the current AI craze. It's that technology can replace the need to look things up ourselves by simply giving us the answers. If a company with Google's money and computing power can't do it, who can?
SGE integrates the functionality of well-known search engines and chatbots. In addition to traditional results, SGE exports direct answers to queries interspersed with links to dig deeper.
SGE is a response to the reality that some people, myself included, are starting to rely on AI like ChatGPT to ask more complex questions or when they don't want to read different sites. Search optimization company Onely estimates that SGE reduces a user's overall research journey by 10 to 20 times by bringing together pros and cons, pricing, and other information in one place. doing.
Given our shortening attention spans, omniscient answer bots seem useful. But Google has a lot to work out. We expect our searches to be fast, but Google's AI takes a second or two to generate an answer. Google needs to balance an already fragile web economy. The company's AI answer could steal traffic from publishers who do the expensive and hard work of actually doing the research.
And above all, the new Google must deliver on its promise to answer our questions consistently and accurately. That's what I focused on in my testing, and I kept looking for examples where AI-enhanced Google performed worse than the old Google.
Try Google's AI answers
When you search on Google, often what you really need is a short piece of information or a link. The new Google's AI is so chatty that it's often annoying on a day-to-day basis.
A playful example: “What do Transformers eat?”
The AI's answer was that while the hypothetical robot would need some kind of fuel, it wouldn't actually need to eat or drink. Old Google, on the other hand, had the one-word answer I was looking for. It was “Energon”. (It's kind of magical fuel.) Just scroll down the page and the new Google will give you the answer.
This doesn't just happen to alien robots. He SE Rank, a company specializing in search engine optimization, tested his SGE on his queries for 100,000 keywords and found that the average answer generated was 3,485 characters. Ta. That's about a third of this column. One of Google's challenges is figuring out when it's better for AI to remain silent. In some cases, SGE will ask you to press the “Generate” button before writing out your answer.
Above all, we expect correct information when we search. Google claims that it has an advantage when it comes to ChatGPT because its SGE knowledge is up to date.
However, it turns out that the new Google is still struggling with the recent developments. 3 days rear We searched for “Oscars 2024” for the latest Academy Awards. It announced that the Oscars were still to come and listed several nominees.
And nothing undermined my trust in Google's AI answers more than seeing them confidently fabricate.
That includes facts about yourself. When I asked him about the award-winning series he wrote for the Washington Post, he said it was by someone he didn't know and gave a link to another website.
Then there was the time when SGE gleefully made up information that didn't even exist. When I asked about Danny's Dan Dan Noodles, a restaurant in San Francisco, he described the food as “very long wait times.”
The problem is, this is a fictional restaurant named after my favorite Chinese dish. Google's AI has no problem inventing information.
So-called hallucinations regarding real and fake topics are a known problem with current AI. The disclaimer above the SGE results says “Generative AI is experimental,” but that doesn't solve the problem. Google needs to find a way to say “I don't know” when you're unsure.
To provide answers to everything, Google's AI must determine which sources of information are trustworthy. I don't have much confidence in that judgment.
Remember the outrageous results regarding Zuckerberg's net worth? Professional researchers and long-time Googlers might suggest checking out the Forbes Billionaires list. Google's AI answer relied on his very strange ZipRecruiter “Mark Zuckerberg Jobs” page, which no longer exists.
In my testing, there was a pattern of questionable sources. At Onely's suggestion, I asked the new Google whether Apple iPhones or Samsung phones are more reliable. As a long-time reviewer, I can tell you many great sources of information on this, including professional journalists and repair organizations like iFixit.
Instead, the AI quotes random opinions of people from social media. Beyond the limited usefulness of one Reddit user's experience, how can Google know it's not a fake review posted by a phone manufacturer?
“Google SGE follows a different set of rules than the traditional search engines we know today,” said Tomek Rudzki, head of research and development at Onely.
SEO companies have attempted quantitative research on SGE values, but are limited by Google's requirements for test accounts. But we found a similar pattern of disconnections between old and new Google-linked sites. His Authoritas, an SEO software company, tested searches using his 1,000 shopping terms in late March and found that 77 percent of the time, his No. 1 domain in traditional search results was his AI I found that it doesn't appear anywhere in the answers I created.
SE Ranking also examined 100,000 keyword searches and found that question and answer service Quora is the most linked source by SGE. LinkedIn and Reddit ranked him 5th and him 6th. How often will these sources be employed in his 8th grade term paper?
SE Rankings found that simplelearn.com was the most linked domain for searches on technical topics, including many “how-to” questions. I had never heard of it before. The site calls itself an “online boot camp.”
“This trend not only reduces the quality of search results, but also reduces traffic and revenue for many small businesses, including affiliate websites,” says Anastasia Kociubinska, Head of SEO at SE Ranking .
Google says SGE is an opt-in experiment. But Google is already well past its scheduled end in December of last year, and has not provided any updates on when it will start searching for everyone. Google doesn't think SGE is accurate, fast, or profitable enough, and he may end up changing SGE significantly.
Even though Google appears to be falling behind in the AI race, they would be wise to move slowly. Microsoft's rival search engine Bing underwent a similar AI overhaul in February 2023, but its AI remains best known for being insane.
Google Vice President Elizabeth Reed, who heads SGE, characterized this as a work in progress in an interview.
“We're focused on making sure we get the experience right. There are a lot of different factors that go into this, including latency, accuracy, and usability,” Reid said. “As we've learned over and over again, we've found that it's quite nuanced.” In other words, sometimes AI is helpful, sometimes it's not, and Google is figuring out where to draw the line. It's still being explored.
When I shared the example in this column, Reed said that SGE's hallucination rate was “very low” and had decreased “significantly” since SGE's May launch, but that I avoided it.
“I don't want to downplay it. This is a technology challenge and we're taking it seriously,” Reid said. Placing a link right next to the AI answer is important so people can check the facts for themselves, she added.
Here's a suggestion. Google knows that correct facts matter, so it should disclose its own data on accuracy before introducing SGE to a broader audience. With billions of searches made every day, even 0.001% can add up to a lot of misinformation.
Another area of Google's focus is “trying to get to the heart of the question as quickly as possible and be able to explain it further,” Reid said.
Regarding citing low-quality sources, Google disputed the external research on SGE, saying it was based on a more limited search than what Google actually sees. However, the company declined to share its own data.
Reed said SGE doesn't have any different standards than old Google. “We're seeing more diverse sources come out. But the real goal is to keep quality content on top,” she said.
It is very difficult for humans to choose who to believe. Why does Google think its current AI technology, known as LLM (Large-Scale Language Models), has a role to play?
“They're not perfect,” Reid said. “We want to take this thoughtful approach because the brand trust that people have in Google is so important.”
The future of our information depends on it.
