User:JScherer-WMF/drafts/sandbox/Simple Summary user test

Introduction

Many of our casual readers cannot comfortably read or understand Wikipedia articles. Most readers in the US can comfortably read at a grade 5 level,[CN] yet most Wikipedia articles are written in language that requires a grade 9 or higher reading level. Simple Summaries are meant to simplify and summarize a section of an article in order to make it more accessible to casual readers. We ran an unmoderated Userlytics study on a prototype of our Simple Summary feature with 8 participants. The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions.

Insights

We might have a hit on our hands with this feature. Participants found it easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than I expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.

Response was much more positive than I expected

All participants except one were extremely positive about the feature. I was expecting participants to express AI negativity, skepticism, and fear. The opposite was true. Even the participant who said they wouldn't use the feature explained that this was due to the fact that they don't need summaries, not because the feature was AI-driven.

• "I would love to see this on every article in Wikipedia."
• "An amazing idea."
• "I look forward to seeing Simple Summary."
• "I appreciate this [...] it shows that Wikipedia is innovative and future-oriented."
• "Very transparent and helpful"
• "It makes sense."

Ambiguity with the main article

Once they opened the simple summary, there were no major issues. All participants clearly demonstrated understanding of which sections were made by the AI and which weren't. However, most participants thought the article introduction was the simple summary before they opened the accordion. This probably had something to do with how the question was worded, so I don't want to overstate the insight. The signal is clear enough, though. We need to make it clear that the simple summary is __inside__ the closed accordion, not beneath it.

• Interventions:
  • Try different background colours, underlines, and other styling for Simple Summary to differentiate it from the main article. * Change the title to "View simple summary"

What is it summarizing?

Participants thought the Simple Summary summarized the whole article. Several said that the summary could have been a bit longer. One participant insisted that the summary should include "all necessary info." Another participant expected a different summary for every section in an article.

Trust was higher than expected

Three of the participants stated that they would actually trust a Simple Summary __more__ because an AI doesn't have biases like people do. "[Simple Summaries] are more factual than if a person wrote it [...] people like to lie." Another participant said "Assuming the article is accurate, the summary should be accurate, too." When asked what "unverified" means, one person said that the summaries should be verified by a more reliable AI. One participant said their trust level was a "7 out of 10", which seems appropriate. The distrust signals in the design (unverified badge and disclaimer) seem to have increased participants' trust because they increase the perceived transparency of the summaries and use of AI. One participant said the summary is "very clearly marked as AI."

AI already seems ubiquitous

Everyone knew what we meant when we said "machine generated". Participants immediately understood, at a high level, what the Simple Summary was and how it was made. The study has a very small sample, but this trend suggests a broad, if superficial, understanding of LLMs and AI. A younger participant, probably a university student, uses AI "for everything" and said "I trust AI 100% [...] but not for accurate mathematics." Another participant said that we're probably "a bit late" on introducing AI generated content into Wikipedia.

Simple Summaries seem useful

Simple Summaries are meant to increase the accessibility of Wikipedia content for readers whose literacy is lower than the reading levels in which articles are written. Judging from context clues, most of the participants were native speakers of languages that were not English. None of them knew a lot about dopamine. All of them except one said the feature was a good idea and they would use it themselves.

• "So much quicker to read and understand!"
• "The whole article isn't necessary."
• "Basic, surface knowledge"

Potential inverse proportion of literacy and feature acceptance

Judging from diction and other context clues, the less comfortable the participants were with the original article content, the more receptive they were to the feature overall. The only participant who said they would not use it was a seemingly native English speaker who used university-level diction in their responses. They expressed familiarity with deep reading for research. For them, Simple Summaries weren't useful because they would just read the article. This may suggest an inverse proportion of educational attainment and acceptance of the feature. This also suggests that people who are technically and linguistically hyper-literate like most of our editors, internet pundits, and WMF staff will like the feature the least. The feature isn't really "for" them. We need to consider this in our analysis of the experiment because we don't know the contexts of our experimental browser extension users. Assuming they are more literate (both technically and linguistically) than average, the feature may be less useful to them than it would be for typical, casual readers.