Does AI-generated Personal Relevance Matter?

Testing relevant, personalized environmental stories

A marketing email from Fresh Clean Threads

Hi Zia,

Taylor here! I hope you had an amazing Christmas with your family and friends in Silver Spring.

I wanted to recommend our Black Performance Long Sleeve Crew - it's perfect for those morning walks through Sligo Creek Trail or catching up with friends at Downtown Silver Spring.

And for versatile layering, our Black Quarter Zip is a customer favorite that works great for the variable Maryland weather.

Does it create engaged learners?

Literature provides evidence it does

  • Foundational Framework (Priniski, Hecht, & Harackiewicz, 2017)
  • Personal relevance creates interest (Flunger, et al., 2021)
  • Personal relevance affects emotion processing (Bayer, Ruthann, & Schacht, 2017)

The Big Question

Can we change how participants think about environmental action by making stories personally relevant using generative AI??

What We Mean by "Personal"

🏠 Hometown-specific content

💧 Top environmental concern (their choice)

vs.

📰 Generic environmental story

The Experiment

N = 296 participants

Randomly assigned to:

✅ Treatment: Story tailored by Anthropic's API on the spot to their hometown + concern

⚪ Control: Less relevant story

More about the treatment (location)

More about the treatment (top concern)

Information flow 🌊

➡️ Participant enters survey data

➡️ State & Concern quietly sent OpenAI API with a prompt in the background

➡️ Participant sees 3-paragraph story on the next screen

➡️ Participant is asked follow up questions

Feature Treatment Group Prompt Control Group Prompt
Story Location Set in the ==specified city and state== ({{city}}, {{state}}) Set in a different U.S. location, not the given {{city}} or {{state}}
Environmental Focus Based on the top-ranked (#1) concern from the list Based on the lowest-ranked concern from the list
Protagonist Alex (no stated gender) Alex (no stated gender)
Tone & Style Neutral, pragmatic tone; practical action. Avoid emotion or transformation Neutral, pragmatic tone; practical action. Avoid emotion or transformation
Landmark Use Include a recognizable landmark from the same city, not central to resolution Include a recognizable landmark from the different city, not central to resolution
Embedded Climate Facts Weave in five climate-related facts naturally Weave in the same five climate-related facts as the treatment naturally

Control Group

City: Washington, DC,

Top Concern: Loss of Biodiversity

It was a balmy summer evening in Austin, and the sun was setting over the Texas State Capitol, casting long shadows across the city...

The conversation eventually veered towards food waste, the least of the worries at the meeting yet an ever-present issue Alex felt needed attention.

Alex decided to take a pragmatic approach. They began organizing 'harvest days' wherein volunteers would gather unpicked produce from the community plots before they spoiled... alleviating just a fraction of the larger crisis looming on the horizon.

TreatmentGroup

City, Chapel Hill.

Top Concern: Extreme Weather

Alex pushed open the large glass doors of the public library in Chapel Hill....

Alex's plan was to reduce emissions, in turn increasing the town's resilience to extreme weather. It was a small step...

First: Did the Manipulation Work?

(Did people actually perceive the difference?)

✓ Yes, It Worked

Relevance to their concern:

t = -7.63, p < .001

Relevance to their place:

t = -4.44, p < .001

Treatment group found stories significantly more relevant

We Measured...

📚 Learning: Quiz on story content

🌍 Awareness: Environmental concern frequency

😐 Affect: Mood and climate anxiety

✅ Behavior: Willingness to take action

The results?

No significant differences on any measure.

What We Have

✅ Manipulation worked perfectly (AI did its job)

⚠️ No change in learning, behavior, affect, or awareness

Help!

This is proof of concept...

But how do we make it stronger?

Methodological Questions

Was it...

  1. The wrong theory?
  2. The wrong execution?
  3. A mix of both?
  4. Something else entirely?

Your Expertise

What would you change about:

🔬 The design?

📏 The measures?

⚡ The intervention?

Let's Talk

We're here to learn from you!

Questions? Suggestions? Critiques?

These slides are available at ziahassan.com/naaee

Find me at linkedin.com/in/zia-s-hassan

zhassan4@jh.edu