- Reviews by AI tend to be more favourable.
- Human-created content more relatable.
Research from emlyon business school and Toulouse School of Management, both based in France, has revealed customers find it challenging to spot the difference between product reviews written by a human or AIs like ChatGPT. According to the research, AI-generated reviews are typically less-detailed, include false information, and tend to be substantially more positive compared to authentic feedback.
The study, conducted by Yingting Wen, Professor of Marketing at emlyon business school, and Sandra Laporte, Professor of Marketing at Toulouse School of Management, investigated whether AI systems could craft marketing content that connects with humans as effectively as content created by human marketers.
Professor Wen spoke on the increasing use of AI for marketing purposes, and an increasing number of written reviews by AI, saying, “As generative AI tools like ChatGPT are increasingly used in marketing, they help automate tasks like crafting social media posts and responding to customer comments, resulting in higher engagement and increased purchase intent.”
Wen and Laporte focused primarily on marketing materials like product reviews, social media posts, and ad copy, conducting three studies that compared human-created content with AI-created material. To understand if AI-generated content could achieve the same level of impact and engagement with audiences as human-created content, the researchers focused on the sensory experiences of chocolate and whiskey!
Keeping it human
The primary study used a text analysis tool to compare product reviews written by AI to those written by humans. The results showed that human-written content was more relatable and authentic. It was also less positive than AI-generated text.
In the second study, humans reviewed the content, as opposed to the text analysis tool, with the same conclusions made.
In the third and final study, the researchers studied social media posts, with half being branded and the other half being unbranded. And, a portion of the posts were written by human social media marketers, while the others were created by ChatGPT. The evaluation process saw human raters tasked with reviewing the content to judge its sentiment and assess overall engagement.
Compared to humans, AI struggled to produce content that had depth or meaning. The researchers discovered AI content was not as detailed and lacked a diversity in tone, style, or approach, compared to humans.
Reviews by AI more positive
Nevertheless, AI succeeded in composing content that had positive connotations and emotions with persuasive appeal. It was also more successful at convincing customers of being genuine on social media posts, particularly ChatGPT 4, which was found to be more effective than its predecessor ChatGPT 3.5.
Professor Wen emphasised the importance of human contribution alongside the use of AI. “Research shows that while AI-generated content can be effective, it still lacks the nuanced understanding and authentic voice that human creators bring to marketing, therefore human input is still needed in the process.”
The research highlights the important role a marketer still has, even with the advancements of AI. Rather than relying solely on AI, Wen and Laporte emphasise the need for human creativity, strategic thinking, and emotional insight to create content that resonates with audiences. AI still has a habit of handing out misinformation, as the study revealed, pinpointing some of the challenges that need to be considered.
