An online experiment exploring how consumers respond to AI-generated and human-written advertisements.
A research project completed as part of my MSc in Business Analytics & Artificial Intelligence at Warwick Business School, designed to compare consumer responses to AI-generated and human-written advertising copy.
The project involved designing an online experiment, collecting survey responses, and conducting the statistical analysis in Python. The aim was to understand whether the two advertisements produced different levels of willingness to click, and how perceived relevance and trust were associated with that intention.
The two FocusForm advertisements used in the experiment. Each participant viewed one version without being told who had written the copy.
1.
Business Problem — What Was the Challenge?
AI allows businesses to generate advertising copy quickly, but this does not necessarily mean that consumers will respond favourably to the resulting content. For marketing teams, the challenge is to understand whether AI-generated copy can support their advertising objectives without reducing the credibility or authenticity of the message.
This project explored that question using two advertisements for FocusForm, a fictional productivity app. One contained AI-generated copy, while the other contained human-written copy. The main objective was to compare how likely participants said they would be to select the advertisement’s “Learn More” button.
2.
Approach & Method — How It Was Solved
The experiment was conducted through Qualtrics, with participants randomly assigned to view one of the two advertisements. Both versions followed the same creative brief and used the same broad visual template. The human-written copy was supplied by an external designer with commercial advertising experience, while the AI-generated copy was produced using ChatGPT.
Participants were not told who had written the copy. This allowed the study to compare responses to the advertisements without introducing an explicit AI label.
After viewing their assigned advertisement, participants answered questions about their willingness to click, perceived authenticity, relevance, and trust. Each measure used three questions rated on a seven-point scale. After applying the eligibility, consent, completion, and attention-check requirements, the final dataset contained 60 participants, with 30 in each group.
Techniques:
- Data cleaning and preparation using pandas and NumPy
- Reliability assessment using Cronbach’s alpha
- Comparison of the two groups using Welch’s independent-samples t-tests
- Evaluation of mean differences using confidence intervals and effect sizes
- Multiple linear regression to examine the associations between relevance, trust, and click-through intention
- Regression diagnostics, HC3 robust standard errors, and sensitivity checks
3.
Results & Business Insights
The AI-generated advertisement received an average click-through intention score of 4.26 out of 7, compared with 4.07 for the human-written advertisement. Although the AI version scored slightly higher, the difference was not statistically significant (p = 0.686). Perceived authenticity also did not differ significantly between the two groups.
These results do not establish that AI-generated and human-written copy are equally effective. With only 30 participants in each group, smaller differences may not have been detected. The comparison also concerned two specific advertisements, so the findings cannot be generalised to all AI-generated and human-written advertising.
Perceived relevance and trust were both positively associated with click-through intention after accounting for the other predictor and advertisement condition. The regression model explained 41.6% of the observed variation in click-through intention.
In practice, participants who considered the advertisement more relevant and FocusForm more trustworthy also reported greater willingness to click. However, these relationships do not demonstrate that increasing relevance or trust would directly cause more clicks.
Average stated click-through intention for the human-written and AI-generated advertisements, with 30 participants in each group. Error bars show 95% confidence intervals. The difference was not statistically significant (p = 0.686). The study measured willingness to click, rather than actual clicks.
4.
Practical Implications
The findings did not provide sufficient evidence to select either advertisement as the stronger version based on click-through intention. For a business, the next step would be to test more copy variations with a larger audience and measure actual clicks and conversions.
Relevance and trust would also be useful areas to explore in further testing. This could involve changing how clearly the advertisement communicates the product’s benefits or how convincingly it presents its claims.
The study measured stated intention rather than actual behaviour, and the “Learn More” button was not clickable. A live advertising experiment would therefore be needed before using these findings to make campaign or budget decisions.
5.
Resources
The dissertation provides the full research background, experimental design, results, and limitations. The accompanying Python notebook contains the data preparation, statistical tests, and regression diagnostics used for the analysis.