Wikipedia:Related Pages extension/RfC

RFC: Related Pages extension

Moved from WP:VPM.

This is an RFC relating to the Related Pages extension. It is currently available in beta. You may test it by enabling Related pages under Beta features in Special:Preferences. The WMF is not looking to deploy it at this time, and does not want to push it on Wikis that do not want it. We are willing to make reasonable changes if they will make the feature desirable or more valuable. This is an experimental use of the RFC process and has no outcome other than to generate feedback for the WMF. The Wikimedia Foundation Reading team wants to know:

• What needs to improve or change in the feature?
• Should any changes be made to our plans for testing the feature and measuring its performance?

Collaboratively drafted by Alsee, Melamrawy (WMF), and Jkatz (WMF). Alsee (talk) 08:07, 27 March 2016 (UTC)

Background

Related page example. From English Wikipedia's Protein article.

The Related Pages extension (also known as the Read more feature) is an experiment carried out by the Reading team in order to further engage readers and help them learn more. It is primarily geared towards users who are interested in learning, but do not have a specific goal in mind. A user reaching the bottom of an article is shown the title and lead image for three articles that relate to the article they just finished. The related articles are selected by software based on textual similarity to the current article, with a preference for articles with many inbound links, and a preference for articles carrying the templates for Featured article, Featured picture, Featured sound, Featured list, or Good article. (More details at Related_pages#FAQ.) Editors may override these software selections by adding up to three {{#related: page title}} to the page.

The premise of Related Pages is that users who get to the bottom of an article have passed over blue links and "See also", have finished the article, and are still interested in reading more. Surfacing articles that are similar might be exactly what they are looking for.

Related pages example on mobile web. From English Wikipedia's Protein article.

While Related pages is primarily intended to increase learning by offering suggestions to the reader, learning is a very hard value to measure. As a result, we have been looking at engagement with the feature. This has been released on apps and saw a 16% click-through (for users who saw it). On mobile web, there is a 19% click-through (for users who saw it) The desktop numbers are much lower at 3.4%. While this rate is far higher than the rate of any 3 links on the page (see Research:Improving link coverage) and certainly higher than those towards the bottom of the page (see graph) there is some concern that this is simply because there is an image. To counter that, there is data that suggests the new clicks are additive--meaning they lengthening reading sessions and are not taking clicks away from other links, but these results are not definitive.

Rationale: If readers are offered suggestions that are similar to the topic they are reading about, this will further engage them, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics.

We are using the RFC process as an experiment to see if can be a useful tool for determining product direction.

A brief summary of initial feedback at the project talk page

• The community was not consulted first and we would have helped you avoid major pitfalls.

Reading Team Response: There was no structured consultation in ideation phase, because we don't yet have clear mechanisms for doing so, however, there was an update around the team's thinking to start working on a "Read More" feature, similar to the one on the web, this was published in Reading team updates, published in early November 2015. It was announced on Wiktech-l. In addition, there was a post made on the English Wikipedia Village pump in mid November 2015, in order to update the community about the work happening on the extension. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature and others, we hope that you can help us move forward together.

• Some language Wikipedias, notably German and Russian, have policies prohibiting collections of related links unless it is an objective and complete set.

The Reading Team does not want to push the feature on wikis that don't want it. The reading team is not looking forward to break any community policies, however, it is essential to understand how exactly does Related Pages conflict with Themenring policy on German Wikipedia. It also is important to understand the different angles of the discussion on Russian Wikipedia, if there was a discussion.

• English Wikipedia policy on non-free images does not allow their reuse for decorative or navigational purposes.

Non-free images will be excluded.[1]

• This is the same as see-also and does not add additional value.

See also sections don't exist on all articles. The section has less presence on smaller Wikipedias. At this is a point of the discussion, and one aspects which this RFC can hep define, is how a feature that helps readers further learn about a specific topic by offering more articles related to the topic that they are reading about, can also help the editing community better shape our content in articles. The Reading Team believes the sustained high click through rate on mobile suggests that users are finding it valuable enough to justify the feature. For some articles software might provide objectively better at links, especially as the software improves.

• Wikipedia, unlike many commercial websites, is not about generating as many clicks as possible.
  • This is not about generating clicks. The click itself isn't of value. This is about availing more knowledge by offering more articles related to the topic that readers are learning about, and helping them learn more efficiently. A good website is composed of interlinked pages, that are easy to discover and find, which is one reason why Wikipedia identifies an orphan page as a problem that needs action.
• Many Read more suggestions are redundant to links in the article, and other suggestions can range from suboptimal to bizarre or even damaging. Generic products tend to be given links to arbitrary brands, for example Read more at the Hard disk drive article suggests Seagate Technology and at Car it suggests Mercedes-Benz. Potentially offensive articles and images can show up unexpectedly, such as images of penises and maggot infested wounds. At Korur language it suggested Anus, Anal and Poop (this specific example no longer works, since the automatic results were overridden). At List of serial killers before 1900 it suggests Peru national football team. At Murder of Kelly Anne Bates it suggests Batman.

Reading Team Response: For 'damaging' results, which are rare, editors can over-ride results. For 'suboptimal results', for now I think we should live with it because the readers seem to be deriving value.

• The software returns particularly poor results for Disambiguation pages, for example at Holocaust (disambiguation) it suggests Batman.

The Reading Team sees little utility for the feature on Disambiguation pages. The feature is being disabled on those pages.[2]

What needs to improve or change in the feature

If the feature is deployed here or on other Wikis in the future, how can we improve or change the feature? More specifically, is there anything that would need to change before you would want it on your wiki?

Should any changes be made to our plans for testing

Our first step in testing new products is to make them available under Beta features in Special:Preferences. Logged-in volunteers can try experimental new features. This is where we are now.

Next steps:

If Beta goes well then our next step in testing is normally to see how it performs for a limited number of randomly selected logged-out users (A/B testing). Due to privacy reasons, the best metric we have is going to be the % of people who see related pages who click on them--we are aware that this is not an ideal metric, but it is directionally helpful. For logged-in users, currently on mobile we are seeing a 19% Click-rate and a 4% click rate on desktop. Another alternative would be to run a survey to see if users who click on a related page entry have any difference in satisfaction over other users, but self-reported satisfaction results rarely change.

Or we can partially roll it out on several wikis at a time and see. The argument for the latter is that this has been on the apps for almost a year and seems to be doing well.

Or this can be just one of the extensions available for enabling on individual wikis which happened to request it per standard procedure Requesting wiki configuration changes.

Discussion

Please discuss on this page's talk page.