The setup: Two writers, two LLMs, one simple brief

In order to make our experiment as unbiased as possible in favor of humans or machines, we designed it to be intentionally narrow in scope. Two experienced marketing copywriters and two leading LLMs were provided the same information: a simplified version of a creative brief from a fictional activewear brand. The brief also served as the prompts for the LLMs.

The brief identified two target audiences: baby boomers who were established customers of the brand, and millennial customers who were new to the brand. The brief also provided a short paragraph of audience considerations (e.g., “they are budget-conscious and seek value-for-money products”), a short list of brand voice attributes (e.g., “wholesome and genuine”), and the desired actions the audience should take.

For each audience, the writers and LLMs were asked to produce two emails: one promoting a loyalty program and another promoting a 25%-off sale. Importantly, neither the human writers nor the LLMs were provided anything more about the brand or its products. We explained the work was part of an effort to win business for Deloitte Digital from the activewear brand. That meant the writers and LLMs were informed the work was part of an evaluation—but the copywriters and LLMs were unaware of exactly how or by whom their work would be evaluated.

The test: A focus group of a thousand

The unedited emails for both campaigns were then fed into a survey tool and a demographically distributed pool of 1,000 consumers rated each email on a scale of 1–5, based on the following criteria:         

• How personal and authentic it feels
• How relevant it is to personal interests and priorities
• How readable it is in terms of structure, grammar, etc.
• How clear it is in terms of messaging and purpose
• How actionable it is in terms of providing a clear next step (i.e., the quality of the call to action)
• Finally, we asked respondents to rate their willingness to act (i.e., make a purchase or sign up for the loyalty program) after reading each email.

Nowhere were survey participants told who (or what) wrote the emails—nor why we were asking for their assessments. So, like the writers and LLMs, survey respondents didn’t know what we were actually testing.

The results: GenAI proves it’s ready to run

For the first five criteria in our survey, consumers rated the GenAI-produced emails somewhat better than those produced by the copywriters, by an average margin of 4.5% across the two campaigns. GenAI’s outputs were most effective when it came to personalization, with consumers rating the emails nearly 7% better against that criteria. (Readability showed the smallest gap of 1.3%.)

Results against the first five criteria differed across demographic groups, with millennials more favorable toward the GenAI-produced emails than baby boomers. In comparison, older consumers preferred the readability of the human-written emails over those produced by GenAI, and rated the actionability of the emails equally.

Results also differed—though less so—between the two campaign types, with GenAI scoring better in the loyalty campaign than the buy-now campaign. The GenAI-produced emails were also more likely to receive high ratings (i.e., 4 or 5) across each of the first five criteria.

Of course, few consumers piece apart whether a marketing email is well written, clear or purposeful—and even fewer marketing organizations measure those dimensions when assessing the effectiveness of their content.

What ultimately matters is impact: Will consumers act? In this respect, our research revealed an important twist.

Overall, survey respondents were 3.1% more willing to buy or sign up for the loyalty program after reading the GenAI-produced emails, compared to the human-written emails. That preference held true for both campaigns and for both demographics.

But here’s the twist. Among respondents who expressed a high willingness to take action, the human-written emails significantly outperformed GenAI. Overall, the human-written emails were 6.4% more likely to earn ratings of 4 or 5. The gap approached 20%—the highest differences found in the entire survey—for millennials rating their likelihood to join the loyalty program and for baby boomers rating their likelihood to purchase. Only in two subsets of the data did GenAI outperform human copywriters when it came to high likelihood of taking action—and in those cases, the gap was relatively small.

Why would human-written copy be less actionable, but much better when it came to high likelihood of taking action? Because the human-written copy elicited the strongest responses—both positive and negative—while the GenAI-produced copy produced more middling responses.

The implications: The long game belongs to people working with AI

Ultimately, the results of our experiment were a mixed bag when it came to the quality and effectiveness of the content. When it came to efficiency, GenAI showed its muscle. It took a matter of minutes for an engineer to input prompts for each email, while the human copywriters spent an average of four hours writing each email. Total LLM token cost for crafting all eight emails was less than a dollar.

That might sound like a clear win. However, before declaring GenAI the new content marketing king, it’s vital to remember a central limitation of our experiment: the contextual information and details provided to the copywriters and LLMs.

In the real world, even a copywriter hired yesterday would know more about your brand, products, audience and voice than what our human and AI “contestants” were provided as part of this test. The seasoned creative director and product marketer reviewing the writer’s work would have even greater familiarity with the tactics and nuances that have driven marketing impact for your organization. You’ve probably already established processes for collaboration and iterative improvement between these members of your team. And your content is surely more effective as a result.

GenAI is ready to play a role in your content operations. It can aid product marketers by surfacing nuanced audience and market insights from campaign, media and customer data—provided you have the necessary analytics, data science and measurement frameworks in place. It can assist creatively by producing quality first drafts of marketing content, as this experiment supports. And, when trained on brand standards and regulatory restrictions, GenAI can also help by identifying risks across your marketing operations and activations.

In every GenAI use case, the key word is “assist.”

Just as you wouldn’t entrust a single copywriter to draft and publish marketing content without review by other team members, you shouldn’t expect GenAI alone to produce the best possible insights or content to drive marketing effectiveness. GenAI has a place on your team—but should not be the whole team.

After all, innovation and growth don’t sprout from data or technology alone, but rather from how those tools are used. The moves that define your brand’s market differentiation, customer trust and business growth still depend on the knowledge, contextual awareness and empathy that humans working together can provide.

By strategically leveraging GenAI to enhance collaboration, identify low-value tasks and open process bottlenecks across your marketing content supply chain, your people and GenAI can do what they do best, together.

Now it’s time to get in the game.