Wikipedia:Measuring conflict of interest editing on Wikipedia

This is an explanatory essay about the Wikipedia:Conflict of interest page.

This page provides additional information about concepts in the page(s) it supplements. This page is not one of Wikipedia's policies or guidelines as it has not been thoroughly vetted by the community.

Shortcuts

As of 2017 there have been no measurements of conflict of interest editing on Wikipedia. In Wikipedia, "Conflict of interest" (COI) editing refers to all the wiki editing which occurs in the context of a conflict of interest and which is said to have a likelihood of being a disruptive engagement, negatively impacting Wikipedia, and greatly stressing Wikipedia community volunteers who come into contact with this activity. COI is a popular topic of discussion on Wikipedia, and yet the community does not have access to basic data about the scope of the issue. Desirable data estimates would include how often conflict of interest editing happens, what percentage of the time it has positive/neutral/negative outcomes, and how much Wikipedia community labor the conflict of interest management procedures are consuming. Various participants in discussions form opinions in debates and take sides in policy discussions. Through all this, there is wide disagreement on the facts and premises from which any discussion should develop. As an example, some people might say that conflict of interest editing causes several serious problems daily which consume hours of labor to address, whereas others might say that conflict of interest editing rarely causes problems and is almost always a net positive.

This page discusses what data the Wikipedia community has to measure conflict of interest and what quality that data is.

Why Wikipedia needs measurements of conflict of interest editing

The Wikipedia community continually raises policy discussions, administers bureaucratic oversight, provides editor counseling, and finds itself in uncontrolled media attention as a consequence of addressing conflict of interest editing. Many Wikipedia community members who engage in these processes find them to be either negative experiences or an unfortunate use of their time.

When users propose policy in an attempt to improve Wikipedia's treatment of conflict of interest editors or the Wikipedia community volunteers who respond to them, recurring discussion points include that conflict of interest editing is a large problem, while others say it is not. Since there is no data personal perception varies wildly.

As of November 2017, these are the opposing sorts of perspectives which different reasonable, experienced Wikipedians might hold:

Conflict of interest editing is a major problem

Conflict of interest editing causes multiple major problems on English Wikipedia every day. It consumes about 50 hours of volunteer labor every day to clean up, with the end result being no net positive content contribution to Wikipedia, stress to the volunteers who addressed it, and dissatisfaction by the conflict of interest editor. Beyond all the problematic cases which the Wikipedia community identified, perhaps 3 times as many conflict of interest editors posted 3 times as much problematic content as was removed. The problem is terrible and Wikipedia is being greatly harmed.

Conflict of interest editing is a major benefit

Conflict of interest editing rarely causes problems on English Wikipedia. The conflict of interest editors are overwhelmingly a net positive in their contributions to Wikipedia. When the Wikipedia community of volunteers gives them an hour of support in reviewing their content or assisting them in the usual wiki peer-to-peer way, conflict of interest editors give 10 hours of labor back to Wikipedia in a positive way. There are problems from time to time, but those problems are not the norm. To make Wikipedia better, the community should encourage and seek to increase the amount of conflict of interest editing on Wikipedia. It would be useful to make tools and tutorials specifically to recruit more editors of this sort.

Determining the effect of conflict of interest editing should have some basis in some facts. There is room for difference in opinion, but there should be less doubt about how often a controversial process causes problems or is welcomed. Wikipedia needs some counts and estimates. Even anecdotes and educated guessing by Wikipedia users with experience and doing informal counts would be better than allowing conversations to proceed with imagined data guesses which might have a range of several orders of magnitude.

Unanswered basic questions

How frequently does conflict of interest editing happen on Wikipedia?
1. What are Wikipedia's tools for identifying and counting instances of conflict of interest editing?
2. What ledgers are there of identified editors with a conflict of interest?
How often is conflict of interest editing problematic?
1. Considering all identified conflict of interest editing, what percentage of time are those contributions good versus neutral versus bad?
2. How much ambiguity is there in differentiating good versus bad conflict of interest editing?
3. How often does conflict of interest editing result in community pushback, like reverting content, blocking a user, creating a dispute, or escalating the conflict to an admin?
When conflict of interest editing is problematic, how much Wikipedia community volunteer labor do volunteers provide to fix it?
1. How many Wikipedia volunteer editors address problematic conflict of interest editing?
2. What communities of Wikipedia editors exist which routinely address conflict of interest editing?
3. Among Wikipedia editors who address conflict of interest editing, what are their opinions on the value of this labor?
4. What skills does a Wikipedia volunteer need to interact with paid editors? To what extent do these interactions require training and expertise, or otherwise more valuable labor?
To what extent does conflict of interest editing affect Wikipedia's reputation?
1. How often does the media report stories about conflict of interest editing?
2. What reputation does Wikipedia have among conflict of interest editors?
3. How often do Wikipedia community volunteers interact with conflict of interest editors?
4. How often do Wikipedia readers view content provided by conflict of interest editors?

Ways to measure

Technological measurement

Wikipedia's software could automatically count and detect information about conflict of interest editing. As of November 2017 either no such systems exist or none are well known to the community.

Educated guesses

Ask Wikipedians who have experience in dealing with conflict of interest what their educated guesses are to the scope of the issue. These guesses will of course be limited to the narrow part of Wikipedia which they observe, but perhaps what they report could be insightful. Ask these questions:

Please volunteer #1 edit here

What is your best guess for how often the volunteer team in this project oversees conflict of interest editing events?
Imagine all the conflict of interest editing events you have seen in this project. Divide them into two groups, "acceptable or productive contributions" and "unacceptable or unproductive contributions". What is your guess for the percentage of conflict of interest editing events which are "acceptable or productive contributions"?
What is your best guess for how many hours Wikipedia community volunteers spend in this project addressing conflict of interest editing events?
Imagine that Wikipedians in this project with your level of experience provide 10 hours of support to other volunteer Wikipedians. What is your best guess of how many hours of "acceptable or productive contributions" Wikipedia is likely to get in return for that 10 hour investment?
Imagine that Wikipedians in this project with your level of experience provide 10 hours of support to conflict of interest editors. What is your best guess of how many hours of "acceptable or productive contributions" Wikipedia is likely to get in return for that 10 hour investment?
In 2017 English Wikipedia has about 40,000 active editors. In terms of Wikipedia experience, where do you think you rank among editors? Bottom 20%, second 20%, third 20%, fourth 20%, and or top 20%?
Please sign your username.

Survey communities

It would be helpful to get multiple guesses and a consensus from communities which routinely address conflict of interest editing. The following communities and forums are likely to have many participants with a lot of experience interacting with conflict of interest editors.

Human arithmetic

Anyone who has time and interest can go to discussion boards and do simple counts and addition of incidents. For example, anyone can find a place which discusses conflict of interest editing, count all relevant discussions in the past 3 months or year, and classify them in an insightful way. Doing this and sharing the result could provide starting data for determining the scope of the issue.

Seek organizational consensus

Certain WikiProjects, Wikimedia user groups, and teams might be organized enough to confer among themselves and issue a collective statement.

Statement from some members of the (project) team

"We talked among ourselves, counted our labor activities, and came to a consensus. We 5 people have each been editing Wikipedia for a year and have collectively responded to 200 (Wikipedia things). Based on everything we see, we are in agreement that we see at least (1-1000) instances of conflict of interest editing every 3 months. Of those W% meet Wikipedia standards and the remaining X% are problematic. Of all of those, Y% are exceptionally good and praiseworthy, and Z% are exceptionally problematic causing stress and a labor time sink."