Human Decision-Making with Persuasive and Narrative LLM Explanations
Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu
Key claim
Narrative explanations do not enhance decision accuracy significantly.
This study investigates how LLM-generated narrative explanations affect human decision-making. The key finding is that the persuasiveness of these narratives does not significantly improve decision accuracy compared to AI predictions alone, and may even slow down response times.
The study explores the impact of narrative explanations on decision-making, extending prior work on AI explanations.
The experiment is large-scale and employs solid methodology, though it lacks rigorous baselines.
Deep reliability assessment
The methodology supports the finding that LLM-generated narrative explanations do not significantly improve decision-making accuracy compared to AI predictions alone, but it may overclaim the benefits of persuasive narratives on reliance and response times. The results indicate potential trade-offs between reliance on AI and actual decision-making performance.
Reproducibility
Yes, the full stimulus set, participant data, and analysis code are available at https://osf.io/nfht2/overview?view_only=b7b7b3d6f84f471e89fabaa7bed4a94d.
Discussion questions
- What assumptions are made about the effectiveness of narrative explanations in influencing decision-making?
- How can builders apply these findings to improve AI systems without compromising user decision accuracy?
- What experimental conditions or variables would need to change to potentially invalidate the conclusion that persuasive narratives do not enhance decision-making?
Key figure
Figure 1 illustrates the average participant accuracy across different explanation conditions and datasets, highlighting the lack of significant differences in accuracy due to varying levels of narrative persuasiveness.