Wikipedia:Bots/Requests for approval/HostBot 4

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Approved.

HostBot 4

Operator: Jtmorgan (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 14:58, 7 November 2013 (UTC)

Automatic, Supervised, or Manual: Automatic

Programming language(s): Python, uses wikitools

Source code available: WIP, in this repo.

Function overview: Invites new good faith editors to play The Wikipedia Adventure. The Wikipedia Adventure is an interactive onboarding game designed to teach new contributors how to edit in a playful and educational fashion. It's funded by a Wikimedia Foundation Individual Engagement Grant and created by User:Ocaasi. The game takes place entirely in the editor's userspace and no outside Wikipedia pages are touched.

Links to relevant discussions (where appropriate): User:Ocaasi/TWAbag has the research and impact analysis plan and details

Edit period(s): Daily

Estimated number of pages affected: 100 per day during the course of the beta test (2-4 weeks)

Exclusion compliant (Yes/No): Yes

Already has a bot flag (Yes/No): Yes

Function details:

The sampling and data analysis plan for the TWA test will be similar to that used to evaluate the impact of participation in the Wikipedia Teahouse, which is described in the Teahouse metrics report and in this research paper.

We will invite a sample of 100 new editors to play TWA every day. The sample will be drawn from the set of users classified as “good faith” by the Snuggle tool developed by EpochFail. A sample of Snuggle data is available here. The criteria for invitation will be:

The user created their account within the past 24 hours
The user has made at least 1 main namespace edit
The user has a Snuggle desirability score of >.8. Blocked or banned accounts are excluded by this threshold, as are users who are likely to be editing in bad faith.
The user has not yet received a Teahouse invitation.

Invites to play the game will be sent via a talkpage invitation from HostBot. Users who receive an invitation and subsequently complete at least 1 level of the Wikipedia Adventure will serve as the Experimental group (Group A).

For every 100 editors invited to play TWA, another 100 new editors who meet the criteria for invitation will not receive one. Of these editors, those who subsequently make at least 1 edit to Wikipedia will serve as a basic experimental control group (Group B). We require at least 1 subsequent edit (after the hour when the user would have been invited to TWA, had they been included in Group A) in order to assure that the editors in this group would have had the opportunity to see the invitation--i.e. to make sure they had not already given up or lost interest in editing by the time of invitation.

A second control group (Group C) will consist of editors who received an invitation, did not play TWA at all, but who did make at least 1 edit to Wikipedia after receiving the invitation. This control group will be used to determine whether the invitation itself has any effect on subsequent editing activities, or long-term retention, separate from the potential impact of playing TWA.

The editing subsequent editing activities of the editors in Group A will be compared with those in Groups B and C. Metrics used to evaluate impact are likely to include number of edits, number of articles edited, change in Snuggle desirability score, and level of activity over time (retention), and may include other metrics.

Editors who start the game will be monitored for signs of increased vandalism to the encyclopedia, and cleanup actions will be taken by those monitoring as needed during the test.

Discussion

Just a quick note that development of wikitools is now going to be at GitHub. Mr.Z-man 15:30, 7 November 2013 (UTC)[reply]

Nice. Watching the repo now. Thanks! - J-Mo ^{Talk to Me} _{Email Me} 17:49, 7 November 2013 (UTC)[reply]

While you're working on it, you should read the PEP8. →Σ σ ς. (Sigma) 21:23, 9 November 2013 (UTC)[reply]

So, Wikipedia process and user experience improvement. Has this been discussed anywhere, or is it something off your own bat? Josh Parris 07:04, 8 November 2013 (UTC)[reply]

Ocaasi first proposed the Wikipedia Adventure over a year ago, and received support from the Wikimedia Foundation to move forward with the project earlier this year through an Individual Engagement Grant. The idea of a Wikipedia Adventure (and an initial prototype) was first conceived by Dcoetzee, I believe. There has been extensive discussion of the project over the past year, most prominently at The Wikipedia Adventure talk page. So, yes, it's very much Ocaasi's own bat :)

This bot request is for evaluation of the Wikipedia Adventure: which I am assisting with because I run a spambot, and I've done this kind of thing before.

Does that answer your question, Josh? - J-Mo ^{Talk to Me} _{Email Me} 19:14, 8 November 2013 (UTC)[reply]

Reading Wikipedia:TWA/Feedback shows me that the TWA is still in beta. Are you planning on messaging new users ASAP, or once the bugs are shaken out?

Wikipedia talk:The Wikipedia Adventure/Archive 1 and Wikipedia talk:The Wikipedia Adventure both show broad support for the game, but no mention is made of editing new users' talk pages. Is there somewhere else that the idea has been discussed, such as WP:VPR? If not, please take this to WP:VPR to form community support for this messaging task (presumably this research task of 100 samples is expected to be expanded out to an ongoing program if successful). Josh Parris 22:12, 8 November 2013 (UTC)[reply]

Hi Josh Parris! Hopefully I can address some of your concerns:

Extensive alpha: As you can see, the alpha bugfixing was very successful. We patched 200 bugs with over 40 editors participating. There are under ten bugs left, all non-critical. We do not intend to run the beta test until these are fixed, so not ASAP, but soon. Since these are minor bugs, they shouldn't take more than a week. Even with a beta, bug-free software is always a goal but not a current condition and we'll continue actively patching bugs whenever they come up. You note that TWA was well supported on the talk page and globally as a grant proposal.

Editing userpages: As for editing new users' talk pages, there was an extensive discussion of this on AN/I and the consensus appeared to be that it was not a problem provided the editors were notified and given an upfront disclosure of the game's behavior (Wikipedia:Administrators'_noticeboard/IncidentArchive806#Message_forgery). Posting to editor's talk page is indeed a feature not a bug, as it permits the game to have a degree of realism and interactivity while remaining entirely separate from the project's public-facing article pages. Several concrete steps were taken towards the end of disclosure: 1) All automatic posts have identifying edit summaries that they were "simulated automatically as part of The Wikipedia Adventure"; 2) There is an infobox on WP:TWA that states automatic messages are sent to userspace; 3) The second slide of the adventure reiterates that point that there are automatic messages sent.

Experimental focus: The purpose of this test is to gather data to evaluate the Adventure's effectiveness at increasing editor activity and retention and constructive behavior. We have no intent to expand beyond this initial test before we have data clearly demonstrating the test is unproblematic and then further that the game is effective. Indeed, after this moderately-sized beta test, we would take time to evaluate and analyze the data while also sharing it anew with the community to determine if further testing is needed and appropriate, or if a broader expansion would be useful and beneficial.

Do you have any other ideas that would help allay your concerns or further demonstrate consensus for a limited experimental beta-test? I'd be happy to take those steps if you have more ideas. Best, Ocaasi^{t | c} 09:07, 9 November 2013 (UTC)[reply]

Everything you say makes sense, and I support you trying this out. ANI seems to read as "You did a confusing thing! Oh, neat idea. Sounds good, I like it!" But again, there doesn't seem to have been any point where you've gone to the community and said "We're going to go and message 100 new editors, is that okay?". Messaging editors is something the community has historically been sensitive about. Go to WP:VPR, and if they don't say it's not okay, then you're in (I'm allowing for a tumbleweed response, which is not unprecedented). Josh Parris 11:03, 9 November 2013 (UTC)[reply]

Hi Josh Parris, I have posted a notification with details of the plan at Village Pump Proposals. I'll keep tabs on the discussion and we can revisit this after folks have had time to respond. Best, Jake Ocaasi^{t | c} 10:08, 10 November 2013 (UTC)[reply]

Hi Josh Parris, it's been a few days and there's zero activity. Any thoughts on how to proceed? Cheers, Ocaasi^{t | c} 10:57, 13 November 2013 (UTC)[reply]

Will your A-B testing be screwed up if you don't do the hundred in one hit? If not, then

Approved for trial (5 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete., otherwise speak up. Josh Parris 11:00, 13 November 2013 (UTC)[reply]

Hi Josh Parris. We can test out the bot function with 5 edits, however, in order to have any statistically valid data by the trial's end were going to need to send the invite to several hundreds of users. Most users who receive the invite will not play the game, so in order to have valid data from those who do play the game, we need a significantly larger sample. Our goal is to have that data during November and early December, for extensive analysis by January. I'm fully on board to have us test out the code in a small batch; however, we are looking for a broader implementation in order to gauge The Wikipedia Adventure's impact across a broad sample of new editors. Please let me know if I can explain our test plan any further. Thanks! Ocaasi^{t | c} 23:16, 14 November 2013 (UTC)[reply]

I fully understand your task. Let's just treat the five edits as a sanity test. Off you go! Josh Parris 04:16, 15 November 2013 (UTC)[reply]

Sweet! We're running the test today, Josh Parris. Thanks for stepping up and facilitating this process :) Ocaasi^{t | c} 11:09, 15 November 2013 (UTC)[reply]

Hey there Josh Parris. I've sent five test TWA invites. Check out the edits ( [1], [2], [3], [4], [5]) and let us know. Thanks, - J-Mo ^{Talk to Me} _{Email Me} 01:52, 16 November 2013 (UTC)[reply]

Unsurprisingly, that worked fine. Operator already has other tasks, is in good standing. Task has community approval. Approved.

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.