Have you ever noticed that in the marketing industry, seemingly impressive statistics that indicate surprising new trends can sometimes have obscure origins?
They're usually big, eye-popping sounds that make you stop and say, “Wow!” Really? ' But if you look deeper, you might find that these punchy numbers are quoted in a way that doesn't faithfully reproduce the original statistics or their sources.
I've made something of a pastime out of debunking the statistics of marketing services that suffer this fate. You probably know the stats I'm talking about – the prediction that 50% of searches will be done via voice by 2020, or the 40% of Gen Z avoiding Google for TikTok. The claim that there is.
These statistics sound like big news, but they may be too big to be true. But these statistics are so popular and widely cited (often accompanied by reliable research organizations) that they must be describing something specific. The presentation will focus on them. People discuss the driving forces behind these changes. But the statistics that spark these discussions are often not what they seem.
While these statistics are by no means fabricated, they do not completely faithfully reflect the original findings of surveys, studies, and predictions, but rather are larger, seemingly more definitive versions.
Let me explain where this trend comes from, why I think it's a problem, and what we can do to counter its prevalence, especially in the world of marketing.
A love of industry statistics
Among these repeatedly cited troubling statistics is that there is a kind of generational loss, where the quality of images and data deteriorates as copies are copied over time (like in a game of telephone).
As a practical example, let's take a look at a statistic I recently wrote about. His claim is that 40% of Gen Zers use his TikTok instead of Google for searches. You may have seen this statistic brought up in discussions about the dominance of TikTok and how young people's online behavior is changing.
This comes from a quote from a presentation given by Google's senior vice president, Prabhakar Raghavan, during a talk. Fortune's Brainstorming technology conference. However, the original quote contained some additional details that significantly changed the emphasis of the statistics.
- Prabhakar was talking about using Tiktok and Instagram
- He said that “about 40%” of young people (an important caveat) use these social networks for certain types of information searches.
- He was talking specifically about looking for inspiration for a restaurant, rather than a broader type of search.
Follow-up by tech crunch It also clarified that this statistic only covers young people in the United States between the ages of 18 and 24. Additionally, this comes from an internal investigation that Google has not made public so far.
While many publications reported Raghavan's words with relevant context, some chose to focus on the punchier and more interesting aspects of the news in their headlines and articles. It didn't mention that this is not exclusive, for example, the fact that young people are “using TikTok to search,” it specialized in TikTok, and it specialized in discovering restaurants.
This made the simplified version of the statistic more popular than the original. Few people will question whether something is completely accurate, especially if the source (Google) is clearly indicated. A curious onlooker (or an avid B2B marketing journalist) can easily search for more information, but instead of quoting the original words verbatim, they should state a distorted version of the statistics. There are many articles he may come across in one.
Statistics, statistics everywhere
It’s not hard to find examples of this across the marketing services industry, including presentations, content marketing, and social media posts. In some cases, this is also true of broader business and technology journalism. We love good statistics. Incorporating good statistics is a concise and eye-catching way to explain or support an argument or point out evidence of broader industry trends.
But until we find out exactly where the claim “50% of searches will be done by voice in 2020” comes from, who has the time to sit down in front of a computer and go down the rabbit hole? There won't be anyone. In many cases, it seems sufficient to name the source in a citation (especially in presentations, where the user can go off and search for the original if desired). Blog posts and content marketing (and definitely sales pitches) may link to or reference publications that cite sources, rather than the sources themselves. (In rare cases, statistical information may be incorrectly attributed to the publication reporting it).
Without the ability to easily see or find the original source, date, region (many of the statistics are specific to the United States, but are cited as if they were worldwide) , context is lost, along with details such as demographics. These things can make statistics less applicable than expected, and in some cases completely obsolete.
Also, the research or technical institutes that provide the statistics may not be as rigorous as necessary. Another one of my “statistical concerns” is the numbers from Google Managing Director Sander Pichai at his 2016 Google I/O keynote. In it, he says that one in five of his mobile search queries is done by voice. Google app.
This statistic was often spread without any qualifications, but if you look closely at Pichai's accompanying presentation, you'll notice that Google groups the voices together. command Type things like “directions home” or “call grandma” into the voice “Search” bucket. Apparently not all.
Why is this important?
You may be wondering, “What's the harm in one or two slightly exaggerated presentation statistics?” Even if these numbers don't accurately represent the original findings, they still hold true for broader trends, such as the fact that TikTok is starting to play the role of “search” for young people.
The problem is that when you start paying attention to these distorted or reconstructed statistics, you realize that there are quite a lot of them. And, as I said earlier, the more they circulate, the more authoritative they appear.
And when you see a statistic that two-fifths of young people use TikTok instead of Google, companies will think they are using TikTok. Must Join us on TikTok, even if your reality is completely different. How many companies have jumped into voice optimization when they heard that in a few years half of all searches will be voice?
The marketing services industry has never had more research, whitepapers, and studies, but as Andrew Tenzer writes: marketing week Best papers in 2022, not all of this research is of high quality and some is conducted to support a specific agenda. Tenzer offers three tips on how to spot bad research, and also mentions techniques used to produce attractive and eye-catching statistics. These techniques also require caution.
Below are some of my own tips to help you spot and avoid skewed statistics.
Tips for supporting your argument with secondary materials and statistical information
1) If you quote statistics, be sure to find the original source
Yes, this requires some digging, and even more time, especially if the nice stat you've seen doesn't itself link to the original. However, if you google the exact number or percentage you're looking for along with the source, you'll usually get the right stats without much effort.
If this is not the case, or a clear source is not provided, the statistics may be skewed. If the cited source is a publication (as opposed to publishing original research or whitepapers), find the specific article the statistic was derived from. Did this publication actually cite anyone else? Can you find the statistics in question?
When tracking the original, be sure to note the publication date and whether the statistics are as recent as believed. Is this finding still representative of the current technology or marketing landscape?
Also, be aware of the context of the original statistics. Does it reflect what you thought the statistics were showing? What demographics are being talked about (if not obvious in the body of the study, they may be listed in the appendix) and regions (worldwide)? Statistics shown as size can often be specific to one region). such as the United States, or a few selected regions).
2) Ask yourself if the statistics reflect your own observations
I believe that the voice statistic “50% by 2020” would not have been so hyped if people had stopped to think about whether this was consistent with what we know about voice search in everyday life. I am.
Did they use voice search regularly? Did someone they know use it regularly? Did their experience with voice search make them feel like it was a good alternative to text-based web searches? If not, where did they think this sudden and massive change would come from??
This doesn't work for all statistics, especially those about groups to which the marketer in question may not belong. If you're not a young person, who knows exactly how they use TikTok? In some cases, people try to check statistics without questioning the underlying numbers. “I heard that 40% of young people use TikTok to search, so I ask them why they do that.”
The resulting response guarantees that it is true.
Nevertheless, this thought process could defeat some of the more ambitious statistics regarding emerging technologies such as VR and the Metaverse. If no one you know is using it and no one you know wants to use it, what are the chances? [insert-large-percentage-here] How many will do so in just a few years?
3) Citing your own statistics responsibly
When you find a statistic that perfectly fits your argument, like a missing puzzle piece, it's understandable to want to simply plug it in and not ask any more questions. Or create a think piece or presentation around it.
But is it better to base an argument entirely on slightly biased statistics, or is it better to be able to have an argument without relying on biased statistics?
Faithfully cite statistics, citing context, year of creation, and region, after doing the above due diligence of tracing original sources and subjecting statistics to common-sense “sniff tests.” (if applicable), demographics (if applicable), and other relevant details.
If the stats become less interesting in the process, perhaps it's time to find something more impactful.
Finally, I would like to reiterate a point made by Tenzer about why it is important to avoid shoddy research.
“Marketers cannot escape the role they play in normalizing poor research. We should all be very concerned about the lack of critical thinking. Take everything at face value. If we continue to take things as they are, companies will continue to flood us with meaningless information, and the quality of their decision-making will suffer in a vicious cycle.”
