Readers/Reader Growth/Mobile page previews

As reading experiences continue to shift toward convenience and connection, we want to ensure that the mobile experience of reading Wikipedia allows for easy, intuitive, and clear article navigation.

As part of that goal, we tested Mobile Page Previews — a pop-up bottom sheet that appears when a reader taps a blue link, showing the thumbnail, lead paragraph, and an option to open the article — on mobile web article pages. Page Previews already exist on the Wikipedia desktop experience and both mobile apps. We wanted to test to see if they would be useful for the mobile web reading experience as well.

This was an experiment: we delivered it to a subset of readers via an A/B test, collected usage data for approximately one month, and then turned the experiment off. This work was tracked at Phabricator at T418812.

Results

Based on the results of this experiment, we do not plan to roll out Page Previews as a feature in mobile web.

We found that retention was the same across both groups, which means Page Previews did not meet our launch criterion of improving retention.

We also had statistically significant findings that short taps decreased and long taps increased, which were unexpected and disappointing. In addition, we saw that pageviews and session length both decreased.

We can infer from these findings that readers did not want to engage with Page Previews on mobile web. The relative decrease in short taps indicates that readers interacted with fewer articles when Page Previews were present, or, in other words, they preferred navigating directly into blue-linked articles. Similarly, the relative increase in long taps indicates that readers were working around Page Previews (e.g., to open the article directly, open it in a new tab, view a browser-native preview, etc.).

Background

As mentioned above, Page Previews (Popups) has been a successful desktop feature for years, showing a tooltip-style preview with a thumbnail and lead paragraph when the reader hovers over a link. They were intended to help readers understand a word or concept within the context of the subject they are reading without having to open multiple tabs or leave the original article. Mobile web currently has no equivalent — tapping a link navigates you away to that article immediately.

By reducing the cost of exploration of a link and allowing readers to gain context without navigating away from their original topic, Mobile Page Previews may make it easier for casual readers to get an overview of an article before deciding whether or not to browse to it. We expect this benefit to hold on mobile, where it’s suboptimal and clunky to read from multiple tabs in one session.

The mobile apps have their own page preview implementations and have seen positive feedback, suggesting this experience is useful for readers on small screens. For example, the version of this feature on the Android app led to a 20% increase in links clicked per page in September 2015. This experiment would test bringing a similar experience to mobile web, using a bottom sheet pattern appropriate for touch interaction rather than the desktop hover-triggered tooltip.

Hypothesis

If we test adding Page Previews on Minerva, we will see practically significant improvement to logged-out reader retention.

Experimentation

Type: A/B test via the Test Kitchen framework within the ReaderExperiments extension
Duration: ~1 month of data collection
Target wikis: ar, en, it, fr, pl, vi
Audience: Logged-out mobile readers
Control: Current experience (tapping a link navigates immediately)
Treatment: Bottom sheet preview on link tap

Design

The user flow would be as follows:

Reader is viewing an article on mobile web
Reader taps a blue link in the article body
Instead of navigating immediately, a bottom sheet slides up showing:
1. Page image (thumbnail) where available
2. Article title
3. Article lead, truncated if needed
4. "Open article" button to navigate to the full article
Reader can dismiss by tapping the X button or tapping outside the sheet