Most foodies and wellness junkies have probably sampled kombucha, eaten jackfruit and tried CBD oil in the past few years as these once obscure products infiltrate the mainstream. But the truly hip will soon move on to sipping on pea milk, taking gaba supplements, and smearing their faces with bakuchiol.
Those are predictions from Black Swan, a London-based start-up that hoovers up data from social media, online forums, product review websites as well as other sources and then analyses it to divine what consumers want. Its artificial intelligence software purports to sift signal from noise to figure out which early trends are destined for mass adoption.
Black Swan has signed up 16 big consumer goods companies as clients in the past six months alone, including PepsiCo, Danone and McDonald’s, putting it on track to double revenue this year to reach £30m.
It is growing quickly because it offers something the leading consumer goods companies, whose revenue growth has slowed to a trickle in recent years, desperately need: an understanding of what fickle shoppers will buy in the future, as well as insights into how to best pitch products to them.
“It is like getting a crystal ball that we all have wanted for the past 20 years,” says Elaine Rodrigo, head of consumer insights and strategy for French yoghurt maker Danone.
Black Swan is one of a range of tech companies that are disrupting how market research is done inside the very industry that invented the discipline in the 1920s. From its early days, when Procter & Gamble sent staff out to quiz American housewives about how they cleaned their homes, cooked meals or did laundry, the field has evolved to help consumer goods companies decide what products to make, as well how to price and market them.
For all their record of product innovation, the industry is now trying to shake off the impression that it has become slow, bureaucratic and unimaginative. The makers of well-known foods, drinks and household brands have faced generally slower growth since 2012, which has made it clear that the big companies can no longer dictate consumer tastes as easily as they did in the past.
Worse, they are often caught off guard by trends such as veganism or high-protein “Paleo” diets and then have to scramble to respond when new brands start to take market share. Their complex supply chains, internal processes and adherence to quality and health standards also means that launching new products often takes a year or more, while a challenger brand might only take months to come to market.
Critics point to how the emergence of HaloTop, an American no-sugar, low-calorie ice-cream, surprised Unilever and Nestlé, or how meat jerky has taken off at the expense of traditional crisps.
Now some in the industry are betting that modernising market research can rekindle growth. Leading consumer goods companies want to upgrade decades-old techniques, such as consumer surveys, focus groups and retail sales data, which are seen as too slow, too expensive and often incomplete.
Enter start-ups such as Black Swan. They are using newer methods including online polls that can be done in a few days instead of taking the usual four-six weeks. One such provider, Zappi, has signed up dozens of consumer companies to its self-service platform. It allows marketers to set up their own studies and quickly test ideas with panels of people selected by age, income, or location. Other start-ups, such as London-based Streetbees, allow companies to conduct one-on-one video interviews via smartphone that can be arranged at a moment’s notice with consumers anywhere from Moscow to Oregon.
Then there are more data-rich, artificial intelligence tools that bring new quantitative rigour to a once fuzzy field. Examples include Signals Analytics, founded by two veterans of Israeli military intelligence, and Tastewise, a platform that analyses menus, blogs and online recipes to predict food trends.
The start-ups are challenging the leading market research agencies that have long worked for the big consumer goods groups, namely WPP’s Kantar, Ipsos and Nielsen. The old guard all saw their revenues and profits fall last year, in large part because of how technology is changing their businesses.
“In the insights industry, there is a real gap between what has been traditionally available and what we need today,” says Tim Warner, who heads PepsiCo’s consumer and market research in Europe and parts of Africa.
“There is a real burning platform feeling now,” he says, referring to a famous Nokia memo warning about disruption from the iPhone. “People want faster, more precise tools. The old methods that were invented before the digital era are not agile, precise, and predictive enough for our current needs.”
The stakes are high for consumer goods giants because online shopping and social media have lowered barriers to entry and eroded the advantages that used to come with scale. New upstart brands and local players have flooded into every category from beer and ice-cream to cosmetics and razors, positioning themselves as either more authentic, healthier or more environmentally conscious than traditional brands.
As a result, average revenue growth at the top 30 global consumer goods companies fell to just 0.4 per cent annually in 2013-18, from 4.5 per cent for 2007-2012, according to Bain & Company. Average growth in annual operating profits has halved over the same period.
In the US, large consumer goods companies have lost 2.4 percentage points of market share to smaller companies and private label products since 2013, according to Boston Consulting Group. Meanwhile, Unilever recently admitted it was losing share across about half of its business, with steeper losses seen in packaged foods compared with household goods and beauty products.
The changed reality prompted Jorge Paulo Lemann, one of the billionaires behind investment firm 3G Capital, which owns Kraft Heinz and Burger King, to liken himself to a “terrified dinosaur”. “We bought brands and we thought they would last for ever,” he said at an investment conference last year. “Now, we have to totally adjust to new demands from clients.”
After a decade when the industry has been dominated by the intense cost-cutting ethos pioneered by 3G, the search for quicker and better consumer insight is part of a new emphasis on investment. “In the past, data and insights were often seen as a cost centre and targeted for cuts. But now it has become a strategic asset,” says Mr Warner.
PepsiCo started working with start-ups and tech providers last year as it looked for better ways to understand its customers. The huge snack and soda group behind brands from Doritos to Gatorade took nearly nine months to figure out what it really needed, and realised that no one company offered a solution off the shelf.
So instead PepsiCo decided to work with five start-ups to build a set of tools knitted together in a proprietary platform that could be used by anyone at the company to spot trends earlier and quickly test out their ideas for products, packaging and advertising messages. It dubbed the system ADA, after Ada Lovelace, a Victorian-era scientist who is credited with writing the world’s first computer programme.
In addition to Black Swan for trend-spotting, Pepsi is also working with Zappi for quick polling and testing, and VoxPopMe to collect interviews with consumers. Two other London based start-ups, Been There Done That and Discover.ai focus more on product positioning and marketing messages.
One of the first projects to use Pepsi’s new market research tools was the healthier snack brand called Off the Eaten Path, which uses vegetables such as beans and peas to create crisps with no artificial colours or preservatives. When it wanted to launch a third product for the line last year, ADA identified seaweed as a trend ingredient that would attract health-conscious consumers. The new snack, rice and seaweed crispy curls flavoured with sweet chilli and lime, hit supermarket shelves just under 10 months later, which is relatively quick for a new product launch at a big food company.
This year, PepsiCo used the tools to create two flavours that aimed to modernise the Walkers crisp brand. Ingredient and flavour options were ranked according to the volume and quality of online conversations, as well as the potential for scale. They were then mapped against other sources of data to refine the best flavours for launch. The result: the BBQ Pulled Pork and Walkers Spicy Sriracha flavours.
“ADA allows us to get input from consumers really rapidly, in a more iterative way than in the past,” says PepsiCo’s Mr Warner. “When you’re developing a product, its packaging and advertising, you can test all of it, get really quick consumer input, or validate an idea, which helps guide decisions around what we do and don’t do.”
While such product testing may sound simple, it was not common practice among big consumer goods companies. Instead of being a central part of the innovation process, market research was often a separate department that did not interact regularly with sales people or brand managers. Instead of delving deep into consumers’ minds, people there spent much of their time writing briefs to commission external groups like Kantar and Ipsos to go out and research the ideas that came from the marketing side.
“Up until relatively recently, market research was all about mitigating risk of the decisions that the business had already made,” says Stan Sthanunathan, who has led consumer insights at Unilever since 2013. “Today our role has changed to anticipating consumers’ desires and creating their needs.”
For its part, Unilever is not solely relying on start-ups. About five years ago, Unilever built its own system to listen to consumers via social media, telephone help lines and rating sites, and today it has more than 30 teams of people doing such work in various markets. They monitor online chatter and feed back to brands on what they are picking up, but can also be commissioned by other parts of the business to monitor consumer perspectives on a specific issue.
Unilever has about 700 people working in market research. Although that number has shrunk by a third in the past four years, Mr Sthanunathan says his staff were producing more analysis than ever before because the new tools were more cost effective.
For example, instead of commissioning an “in-home use test” that follows 400 people using its Cif kitchen cleaner and a competitor product, Unilever can now just mine product reviews on Amazon to compare Cif’s product performance and that of its peers.
Another new tool at Unilever is called Idea Swipe, which was inspired by the dating app Tinder. It puts a new product idea in front of consumers on their smartphones — say, for an ice-cream flavour incorporating breakfast cereals — and asks them to swipe right if they like it and swipe left if they do not. The company says the app delivers better quality data more quickly than older survey techniques, and it has tested about 5,000 ideas in more than 25 countries since it was launched a year ago.
It is still too early to tell whether the adoption of digital tools to predict consumer trends and improve marketing will really be enough to help spur faster growth in the sector. Bain & Company says organic sales growth at the top 30 global companies hit 2.9 per cent last year — the best performance since 2013 — as the global economy expanded. François Faelli, who heads Bain’s consumer practice, says the big players “have upped their innovation game” and that it is now starting to pay off.
Others are more sceptical. Elio Leoni Sceti, a veteran industry executive who last year co-founded The Craftory investment firm to back challenger brands, thinks that big companies remain too disconnected from consumers. Their supply chains are slow and they remain too focused on physical retail channels instead of online, he says. Nor do their brands have the authenticity or sense of mission that power the newcomers, like the Craftory’s recent investment in TomboyX, which makes body positive underwear for people of all sizes and gender identities.
“Even if the big guys get good consumer insights, they often don’t listen to them,” says Mr Sceti.