Wikipedia talk:Identifying reliable sources (medicine)/Archive 17

Archive 10

←

Archive 15

Archive 16

Archive 17

Archive 18

Archive 19

Archive 20

Beall’s List of Predatory Publishers 2015

FYI: Beall’s List of Predatory Publishers is a list of scientific publishers who will publish any paper, regardless of quality, for a processing fee. In some cases the predatory publishers uses a name that is similar to or identical to the name of a legitimate journal. --Guy Macon (talk) 16:51, 15 August 2015 (UTC)

Problems with Systematic review usage

The current lede mentions Systematic review, the 1st article cite in the lede links to a page which appears to be based on publications from 1996 about Evidence based medicine (scope is patient care), and does not include most of the content given in the lede of SR. The second cite of the SR lede points to CRD’s guidance for undertaking reviews in health care, the 3rd cite goes to this publication Systematic Reviews in the Social Sciences A PRACTICAL GUIDE, which explains, "In the book we use the term ‘‘systematic review’’ to cover both those reviews that include a statistical summary of the included studies (a metaanalysis), and those that don’t. While we also use the phrase systematic ‘‘literature’’ reviews, not all evidence which may be useful for a review will of course appear in the published ‘‘literature.’’ Because of this, the term ‘‘research synthesis’’ (or ‘‘evidence synthesis’’) is becoming increasingly common. We have used the term ‘‘systematic review’’ in preference, as it is still so widely used, but we do not assume that only ‘‘published’’ literature can be reviewed.". Hence the current lede of SR is entirely at odds (see WP:OR) with Identifying reliable sources (medicine). We should rather mention something about meta analysis, and literature reviews from peer-reviewed journals unless the SR article can be fixed. prokaryotes (talk) 22:03, 16 August 2015 (UTC)

More on point http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/syst-review.pdf prokaryotes (talk) 22:24, 16 August 2015 (UTC)

Suggest you take it to that articles talk page. -- CFCF 🍌 (email) 22:30, 16 August 2015 (UTC)

I did, but since this policy here overlaps with SR, and SMcCandlish suggested above to change the term as well, i brought it here for discussion.prokaryotes (talk) 22:33, 16 August 2015 (UTC)d

I mean the problem isn't with this page, but at the article. The What is a systematic review? page you linked is a good introduction and explains why we are so intent on keeping this wording here. -- CFCF 🍌 (email) 22:46, 16 August 2015 (UTC)

Ok, i agree the paper is much less confusing. prokaryotes (talk) 23:23, 16 August 2015 (UTC)

The new wording "Primary sources should almost never be used for medical content."

With this edit, Alexbrn changed "Primary sources should generally not be used for medical content." to "Primary sources should almost never be used for medical content." Given the various discussions on this talk page about there being cases where primary sources are allowed, whether it's because of what WP:MEDDATE states about there being areas where "little progress is being made or few reviews are being published" or because an editor chooses to provide a primary source in conjunction with the secondary source, I felt that we should perhaps discuss if this wording is what we definitely want to go with in the lead of WP:MEDRS. For an example of a medical topic significantly lacking review articles, see clitoromegaly. Flyer22 (talk) 14:59, 8 July 2015 (UTC)

I'm as big a stickler for using reviews as the next person, but recognize that there are times when primary sources are not only appropriate, but the only sources to use. I'm not sure this change is an improvement. Yobol (talk) 13:05, 9 July 2015 (UTC)

Agree this new wording is not an improvement for the reasons previously given. Ward20 (talk) 17:56, 9 July 2015 (UTC)

• My position is "Primary sources should almost never be used for medical content before discussion and consensus on the talk page." Doc James (talk · contribs · email) 18:19, 9 July 2015 (UTC)

I agree with this wording too. If a topic does not have secondary sources, we generally don't consider it WP:NOTABLE to include an article on it in the first place. "Almost never" still does not exclude primary sources, but basically shows that someone is going to need to show a really unique case related to sourcing to use them. If someone can somehow demonstrate that a topic has weight for coverage, but we don't have great secondary sources and should instead carefully rely on some primary sources, I don't think it should be hard to demonstrate that. I don't see issues with the wording with that in mind. Kingofaces43 (talk) 00:47, 10 July 2015 (UTC)

• I am good with the new language or Doc James' version. I like the economy of the new language. Doc James' longer version could go in the body.... but I would change the new language to "content about health" instead of "medical content". Jytdog (talk) 00:52, 10 July 2015 (UTC)

Thinking about it more, I think looking at the body and lede this way may be the way to go. Alexbrn's original addition above (adding in content about health instead) makes for a good stern piece in the lede. Further along in the body, Doc James' addition could go in the Respect secondary sources section a bit later to show that a primary source is a huge exception to the rule and requires some work to use if that does happen. Kingofaces43 (talk) 00:59, 10 July 2015 (UTC)

Agree with the above suggestions. Doc James (talk · contribs · email) 03:09, 10 July 2015 (UTC)

Doc James's wording makes it seem like we should discuss using a primary source before adding it; as made clear above, first by me and then by Yobol, that should not always be the case. Then again, I think what Doc James means is that we should almost always discuss using a primary source before adding it; I agree with what he means on that. As for adding his wording lower on the page, the "Definitions" section is already clear that use of primary sources for health content should be a rare case; it uses the word rare twice. And the "Respect secondary sources" section is also very clear about not using primary sources, except occasionally. How much more do we need to stress this? Either way, because of past discussions at this talk page about the use of primary sources for health content, I still feel that Alexbrn's wording is too strong. For example, I've been clear that I sometimes like to provide the primary source in conjunction with the secondary source (which the "A primary source may be presented adjunctively to a secondary source." wording of the "Respect secondary sources" section currently allows); there is no problem with that since the secondary source is there for viewing. The bolded "generally" wording in the lead was strict enough with regard to discouraging primary sources. That was us enforcing the "be wary of primary sources" aspect of the guideline. We keep tightening the "primary source" aspect, and it happens each time a WP:Med editor finds misuse and/or excessive use of primary sources. Alexbrn's edit summary seems like recent misuse and/or excessive use of primary sources inspired him to add "almost never." But the misuse is not going to stop, no matter what wording we use. We have no proof that the bolded "generally" wording was working; furthermore, it had not been there for long, and already we've moved on to "almost never." The next step after "almost never" is "never"; and, well, "never" shouldn't be the case. All that stated, I agree with Jytdog that we should change "medical content" to "health content."

And, Kingofaces43, what do you mean by "adding in content about health instead"? Alexbrn simply changed a small piece of the text; "medical content" is in the previous and current wording. Flyer22 (talk) 04:19, 10 July 2015 (UTC)

My comment was about using "health content" instead of "medical content" as Jytdog pointed out. Kingofaces43 (talk) 04:26, 10 July 2015 (UTC)

I was confused by your wording because the "adding in content about health instead" part seemed liked you attributed it to Alexbrn. Flyer22 (talk) 04:33, 10 July 2015 (UTC)

Also, Kingofaces43, regarding your statement of "If a topic does not have secondary sources, we generally don't consider it WP:NOTABLE to include an article on it in the first place.", I don't think that the others were simply referring to no secondary sources. When I mentioned clitoromegaly above, I mentioned that it is significantly lacking review articles. It is reported in secondary sources, but review articles are obviously a different matter. So when citing studies for that topic, we will be citing primary sources...or secondary sources noting the primary studies. And some other medical topics on Wikipedia are similar. But you are correct that "If someone can somehow demonstrate that a topic has weight for coverage, but we don't have great secondary sources and should instead carefully rely on some primary sources, I don't think it should be hard to demonstrate that." Flyer22 (talk) 04:48, 11 July 2015 (UTC)

Often the first thing to come out is primary sources by those closely related to the topic at hand. They are thus often promotional and IMO we would do well to, much of the time, wait for decent independent secondary sources. Think of the topic CCSVI for example. Doc James (talk · contribs · email) 04:55, 11 July 2015 (UTC)

I don't dispute that. Flyer22 (talk) 05:09, 11 July 2015 (UTC)

The problem with this wording is that it creates confusion for editors who happen to be working on topics that are new and emerging fields. In these cases, primary sources might be the only sources available. In developed aspect of a field where there are plenty of secondary sources, yes, primary sources shouldn't ever be used. But not in other cases. I'm concerned some of our editors whom can't see in shades of grey might use wording like this to edit war primary sources that are necessary unless we temper it a bit. LesVegas (talk) 00:42, 13 July 2015 (UTC)

That is my concern, and the "areas where little progress is being made or few reviews are being published" and "new and emerging fields" aspects are why I started this discussion; we need editors to know that primary sources are not banned (WP:MEDRS is a guideline, not a policy anyway), and that there are legitimate exceptions to excluding them for biomedical content. Some editors who were and/or still are less well-versed in WP:MEDRS didn't even know that History sections, and Society and culture sections, are usually exempt from strict WP:MEDRS-compliant sourcing. WP:MEDDATE states as much of History sections. Flyer22 (talk) 01:01, 13 July 2015 (UTC)

I agree, that's quite a take-off from the rest of RS policy. "Should almost never" is not very helpful expression, but more confusing instead. If we want to add such definition, then we should also cover when the primary sources may be used. So far, "generally" leaves room for personal discretion, and we already have policies saying that "reliable secondary sources" should be preferred when available. For example, we might have a topic that has already established notability, but where some new findings emerge. As we know, it takes time before secondary sources are actually available, so primary sources may be considered, being careful not to break WP:UNDUE of course. Maybe a more common example would be that: we have a secondary source "A" making statement "B". The statement "B" is a bit ambiguous, so we may look up to the actual primary source that "A" is quoting to get a better picture, Cheers! Jayaguru-Shishya (talk) 19:25, 13 July 2015 (UTC)

That is not how you write policy/guidance in WP or anywhere. The idea is not micromanaging but setting broad and clear principles. With the language we are discussing, it is clear that someone would have to justify using a primary source; the justification would be discussed at the article talk page and DR from there. Jytdog (talk) 19:32, 13 July 2015 (UTC)

Reboot

It strikes me that much of the aversion to the "should almost never" wording is likely grounded in its prescriptive tone. A more descriptive approach could be simply "Primary sources are rarely useful for medical content." Comments? LeadSongDog come howl! 20:00, 20 July 2015 (UTC)

I see no issues there. Kingofaces43 (talk) 21:09, 20 July 2015 (UTC)

In the hands of an expert editor, primary sources can be extraordinarily useful. We discourage them because they result in WP:DUE violations as a result of cherry-picking by POV pushers, not because they are intrinsically non-useful for medical content.

I think that we need to write a WP:MEDDUE section or page, and to tie our preference for secondary sources to that. WhatamIdoing (talk) 08:19, 21 July 2015 (UTC)

I just noticed that you reverted back to "generally". As for LeadSongDog's wording, I wouldn't mind "rarely"; I mentioned above that "rare" is stressed lower (past the lead) in the guideline. As for WhatamIdoing's suggestion to write a WP:MEDDUE section, I trust that she could type up a decent or good section to address these issues. Flyer22 (talk) 03:04, 25 July 2015 (UTC)

As an addendum I've just said to an editor that per MEDRS, primary sources are "not reliable for health content". They then apparently consulted MEDRS and responded thusly:[1]

To claim that they "not reliable for health content per WP:MEDRS" is a misrepresentation at best or an outright lie at worst.

I see this again and again. Whatever we think we're saying, the repeated message that readers seem to take away from this guideline is still that primary sources for health content are okay. Alexbrn (talk) 06:13, 6 August 2015 (UTC)

Whether in a medical context or not, simply saying that primary sources are [not/rarely/generally not/...] useful isn't the point. The point is the purpose for which the primary sources are used. Thus primary medical sources should not be used to establish the efficacy of a treatment – no qualification needed. But once the efficacy has been established, if the article sets out details of the treatment, primary sources may be the most reliable for these details (e.g. the chemistry and pharmacology of any drugs involved).

This is a classic example of the danger of trying to reduce an issue which ultimately depends on human judgement (i.e. determining the purpose for which a source is used) to a rigid set of rules. It's bound to fail. Provide guidance, examples, explanations, but don't try to provide infallible rules. Peter coxhead (talk) 06:59, 6 August 2015 (UTC)

Broadly speaking, if new readers of MEDRS came away with the understanding you articulated, that would be great, particularly "primary medical sources should not be used to establish the efficacy of a treatment" (I would say "establish or imply" rather than just "establish"). But IME they often don't. Alexbrn (talk) 07:09, 6 August 2015 (UTC)

@Alexbrn: (Adding "imply" is clearly right.) IME too they often don't. But the solution isn't to keep writing more and more prescriptive statements, which just alienate experienced editors and confuse inexperienced ones. Peter coxhead (talk) 08:02, 6 August 2015 (UTC)

I don't believe that you (or anyone else) ought to say that primary sources are not reliable for health content. Some of them are: if you see a press release from a politician that says he's been diagnosed with a medical condition, then that is a self-published primary source about health content. It is also reliable for a statement that Politician Paul said that he had been diagnosed with ____. This guideline is about WP:Biomedical information, which is not the same thing as health content.

I think that Peter's example is a good one. I'll add, though, that "efficacy of a treatment" is maybe as much as 10% of the information that a good encyclopedic article should cover. We may need to re-write MEDRS to better explain what to do about the content.

Also: "ideal sources" (what MEDRS encourages) and "reliable sources" (the rock-bottom minimum that is required) are not synonyms. A primary source is reliable for quite a lot of material, even if it's ideal for relatively little. WhatamIdoing (talk) 03:08, 22 August 2015 (UTC)

Yes, 'health content' is too broad a term in general. I certainly didn't intend to outlaw the type of case you mention, the context in which I used the term was a dispute about whether Mate tea was a potentially effective treatment for certain conditions. Note that para 3 of MEDRS currently scopes it as follows: "This guideline supports the general sourcing policy at Wikipedia:Verifiability with specific attention given to sources appropriate for the medical and health-related content in any type of article, including alternative medicine" - which further demonstrates that MEDRS is a bit of a mess. Alexbrn (talk) 08:19, 22 August 2015 (UTC)

Thanks for the nudge. I've cleaned that up. WhatamIdoing (talk) 15:48, 22 August 2015 (UTC)

Do we allow sources that reference unpublished studies?

Do we allow reviews as sources for claims, in which some or most of the studies referenced in the review are unpublished and unavailable to the public? An example might be an FDA review which refers to many unpublished studies provided by the drug company, but not accessible to the public. SageRad (talk) 16:36, 30 August 2015 (UTC)

Conveniently, the sentence I reference above is applicable here, "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions." (emphasis mine). Yes, sources that reference unpublished data can be used. High quality sources such as the FDA should not be dismissed because editors themselves do not have access to the same data as the source. Yobol (talk) 16:42, 30 August 2015 (UTC)

Hmm. Thanks. I find that disturbing, personally, as it flies against all notions of transparency and peer-review which are supposed to be the basis of resiliency and error-correction in science. SageRad (talk) 18:29, 30 August 2015 (UTC)

{e/c} On the contrary, published data often have significant biases (see publication bias), so the addition of unpublished data to a secondary source such as a meta-analysis is often more neutral and accurate than only using published data. Either way, the use of unpublished data has no real relevance to the reliability of a source, per WP:MEDRS. Yobol (talk) 18:37, 30 August 2015 (UTC)

I question whether the FDA is a "medical" organization as per WP:MEDORG. SageRad (talk) 18:32, 30 August 2015 (UTC)

Of course it is, just as much as the CDC or any major governmental medical/scientific body. Those who dispute the conclusions of the FDA/WHO/CDC often dispute whether the FDA/WHO/CDC are reliable, but that says more about the biases of the individual editor and their dispute with the conclusion of the major medical body than it does about the reliability of the medical body themselves. Yobol (talk) 18:37, 30 August 2015 (UTC)

There was an authenticated letter sent by a group of FDA employees in 2010 that read in part:

... many other FDA managers who have failed to protect the American public, who have violated laws, rules, and regulations, who have suppressed or altered scientific or technological findings and conclusions, who have abused their power and authority, and who have engaged in illegal retaliation against those who speak out, have not been held accountable and remain in place. (CBS News)

I have a problem with trusting governmental agencies, especially when they have a history of corruption and are using information that is not available to the public to pass judgments. This article in JLME also speaks to the regulatory capture of the FDA. I am concerned about trusting a source that other reliable sources say is not reliable. I'd say it should not be viewed as flawless or without reproach, and we should not place blind trust in the FDA especially when their primary sources are unpublished and therefore not peer-reviewable. SageRad (talk) 20:07, 30 August 2015 (UTC)

Wikipedia is not a place to WP:RIGHTGREATWRONGS. The FDA is very much part of the scientific establishment that is the mainstream of scientific/medical thought in the world. Jytdog (talk) 20:10, 30 August 2015 (UTC)

(e/c) There are no publisher of sources that are "flawless" or "without reproach", and you can find examples of almost all high quality sources having some sort of black mark(s) in their record. In general, the FDA is a reliable source for medical information especially as related to their field of expertise (food and drug regulation). We are probably going past what the role of this talk page is for now, so unless there is a particular change you would like to suggest here, it's probably best to leave specific discussions of specific sources to the talk page of the article in question or at WP:RSN. Yobol (talk) 20:13, 30 August 2015 (UTC)

I'd like to be able to discuss general matters of this sort without being shut down. Please stop the shutting down of conversations. And i am not "righting great wrongs". Let us be able to speak about generalizations, with others, and let it stand for a while. Let's allow people to speak here. I'm completely serious in querying the reliability of the FDA and similar sources that use unpublished data to reach conclusions that are then represented as truth. I understand that no source is "perfect", but there is a built-in mechanism that enables peer review for most sources that we would accept as reliable. In other words, most reliable sources are not typically relying on inaccessible data for their reportage. SageRad (talk) 20:16, 30 August 2015 (UTC)

Countercultural conspiracy theories don't travel too far in discussions about policies guidelines. WP is mainstream and conservative when it comes to sourcing and institutions that produce them. By "conservative" i don't mean right wing, I mean trusting the institutions that produce and disseminate scientific knowledge. Jytdog (talk) 20:23, 30 August 2015 (UTC)

(e/c) I get that you believe you are right, and I'm just expressing my own view that attempts to get the FDA disallowed as a reliable source of Wikipedia would be an incredible waste of time, as the vast majority of medical sources and medical expertise recognizes the FDA as a highly reliable source. Best of luck trying to get a consensus around the dubious proposition that it can't be used as a source here. Yobol (talk) 20:29, 30 August 2015 (UTC)

I posed a question, a topic for discussion, on the topic that this page is about. If you have a problem with that, then don't take part in the conversation. Please don't use such a tone as "i get that you believe you are right".... what does that even mean? And i am not attempting to get the FDA disallowed as a source. I am questioning the mechanics of some sources that are accepted as secondary sources, when they are not peer-reviewable, rely on secret documents, and have a documented history of corruption. SageRad (talk) 20:42, 30 August 2015 (UTC)

Please show me any human institution that is free of corruption. Just one. Instances of corruption =/= rotton to the core, and you seem to be proposing the latter, which is just conspiracy theorizing. In general WP trusts the knowledge-producing institutions of our society. Jytdog (talk) 20:48, 30 August 2015 (UTC)

Of course, you're correct that no institution is inherently free of corruption, but that's a red herring because the point i was making was that there is a difference when a source uses inaccessible primary sources to reach a conclusion, and when that source has a documented history of corruption. It's not a good combination. We do need to think critically about credibility of sources, and i would hope there are no sacred cows. This is the place where we talk about general principles about how we decide what sources to trust. We cannot trust all "knowledge producing institutions" equally because they do not merit equal trust. That is why we have guidelines about what is a reliable source and what is not, and why. SageRad (talk) 20:53, 30 August 2015 (UTC)

This talk page is for the concrete discussion of improvements to this page, not for a general philosophical discussion. What is your concrete suggestion to improve this guideline? You've spent a lot of time trying to belittle the FDA but then say you don't want it disallowed as a source. What exactly is your concrete proposal to improve this guideline? Yobol (talk) 21:02, 30 August 2015 (UTC)

Where does it say that? Who made that rule that this talk page is only for concrete proposals and not discussions of principle? Seriously, i've seen several very interesting discussions get shut down in the last couple of weeks here, and i don't like that. Sometimes we need to have a general discussion to arrive at concrete discussions.

My concrete suggestion if i were to propose one, is to state that secondary sources that are peer-reviewable are preferable to sources that rely heavily on unpublished data. SageRad (talk) 21:07, 30 August 2015 (UTC)

You might try reading the very first sentence at the top of this page, which states, "This is the talk page for discussing improvements to the Identifying reliable sources (medicine) page." Discussion should be centered around improving the page, not just a general philosophical discussion about sources (that belongs on user talk page, talk pages of specific articles for discussion of specific sources in context, or on RSN). As to your proposal to deprecate sources that use unpublished data, I will oppose strongly based on my first comment above, which is that the published data is often biased, so better sources can be the one using unpublished data. I should also note that I find your determination that papers that use unpublished data are by definition not "peer-reviewable" as bizarre, but I think I've wasted enough of my time on this thread already, and will take my leave. Yobol (talk) 21:14, 30 August 2015 (UTC)

Nobody is requiring you to take part in this discussion. I'm interested to hear what others think as well. I do believe that this question is a discussion on possible improvements to the guideline. I don't see how a paper that cites unpublished data is reviewable by anyone who doesn't have access to the underlying data. I'm puzzled by that. SageRad (talk) 21:26, 30 August 2015 (UTC)

I would say that you have it exactly backwards. Reviews should absolutely take into account unpublished studies. If they do not then due to publishing bias they have a real chance of overestimating any effect. Also I see earlier you are referring to unpublished studies as "secret documents". There is a world of difference between difficult to access because no one wanted to publish it and a secret document. 2601:645:C201:3840:88CF:A3D4:4EFF:F687 (talk) 03:45, 31 August 2015 (UTC)

• Answer is yes Doc James (talk · contribs · email) 21:15, 30 August 2015 (UTC)
• Comment I think that is a very interesting question, and relates directly to WP:RS and transparency of RS: So far Wikipedia doesn't care. prokaryotes (talk) 23:48, 31 August 2015 (UTC)

Yes. We expect the authors of reliable sources to determine which sources they use, many of which would fail rs for Wikipedia articles. For example in writing about a figure in the news, they may rely on what that person, his or her friends, enemies and acquaintances have said about them, high school year book stories, etc. It is similar to what a court does. They rely on the evidence of people whose testimony does not meet rs and determine the facts. TFD (talk) 01:26, 4 September 2015 (UTC)

In fact, we go further than the question asks: we not only do not require our sources to cite published studies, we do not even require them to cite any sources at all. WhatamIdoing (talk) 06:01, 4 September 2015 (UTC)

Industry Funded Studies

Do we allow industry funded studies on Wikipedia articles? There has been a documented problem of funding bias with industry funded studies, so I was curious if MEDRS restricted such sources of information in some way. LesVegas (talk) 02:10, 30 August 2015 (UTC)

In general, we do not disallow sources based on funding: "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions." (emphasis mine). There are all sorts of biases that can affect a study, we should be careful how we assess the sources so as to not disallow otherwise good studies based on issues that may have no relevance. Yobol (talk) 16:31, 30 August 2015 (UTC)

We do also state Wikipedia:Identifying_reliable_sources_(medicine)#Use_independent_sources. But we should be using reviews independent of the primary source anyway. Doc James (talk · contribs · email) 21:20, 30 August 2015 (UTC)

Thanks for your reply, Yobol. If industry funded studies have shown to have bias (much more positive outcomes compared to neutrally conducted research), why should we allow it on our articles? Shouldn't we change what MEDRS says? I mean, why did we add that tidbit into MEDRS in the first place allowing any research despite possible shady funding sources which might skew its results, and why shouldn't we change MEDRS now to disallow such sources? Isn't it a real problem if a pharmaceutical company funds research showing excellent results for their product if neutral research shows mediocre ones? LesVegas (talk) 02:41, 31 August 2015 (UTC)

I think there are two issues here: 1) Doc James' point is well taken, most of the funded studies you are discussing are primary studies, which we should be avoiding anyways in favor of independent secondary sources such as reviews; avoiding primary sources in favor of secondary will help us avoid those funded sources already. 2) the fact that something is funded does not mean it is automatically "shady" or suspect. There are plenty of good quality industry funded studies as well. We don't want to throw out the baby with the bathwater. Yobol (talk) 02:56, 31 August 2015 (UTC)

We just want to use the best available sources. And yes often these are reviews of industry funded trials. Doc James (talk · contribs · email) 03:05, 31 August 2015 (UTC)

Yes, industry funded studies do eventually make up reviews and meta-analyses, so if there is a problem with industry funded sources, there's a problem with reviews that use them. But I guess it is being argued that even though there is a well-documented history with industry funded research being unreliable with far too many results toward the positive side, it's too problematic for us as editors to determine what's reliable and what's not based on funding sources. Some of it could be good and we'd throw out the baby with the bathwater as Yobol said, so we really shouldn't be questioning it, just ensuring it comes from a source that meets all of our other requirements. But is that really all we're supposed to do as editors? Really? LesVegas (talk) 22:20, 31 August 2015 (UTC)

Not necessarily. Often a good review will take that into account. Doc James (talk · contribs · email) 22:46, 31 August 2015 (UTC)

Should we toss out reviews if they review industry funded studies but don't seem to acknowledge the possibility of funding bias? I mean, if the reviewer just treats the industry funded studies like they're the highest quality in existence and seems oblivious to a well-documented problem in the field, there's no way that can be reliable, right? And the reviewers too would be suspect, right? LesVegas (talk) 23:31, 31 August 2015 (UTC)

No. Aside from that being WP:OR on our part, if a study has a problem (industry funded or not), that will be mentioned in the review of the methodology or specific claims made in the conclusions. If there is an issue with funding or lab-specific related, the reviewers will say, "Studies from lab Y, funding sources Z, etc. all had inadequate methodology for assessing X." This happens every now and then when labs are found doing shoddy work. There are lot of times industry funded trials as designed and analyzed entirely properly, and reviews in that case wouldn't mention any issues from the get go. The sometimes subtle difference that people miss is industry-designed studies. Those are the studies where there isn't independence in terms of the scientists doing the actual work and interpretation. Kingofaces43 (talk) 23:41, 31 August 2015 (UTC)

In a perfect world, reviewers pick up on problems with funding sources that skew results. But since this is an imperfect world, if they miss problems in these studies, why exactly should we not disallow these reviews as sources? LesVegas (talk) 23:53, 31 August 2015 (UTC)

this has been discussed to death in the archives. No. we do not perform peer review on papers. We do look for independent, high quality sources of the types described in MEDRS. Jytdog (talk) 23:56, 31 August 2015 (UTC)

Good quesion LesVegas, though even if funded a study can be useful, and it is not clear which studies are skewed or not. But we can report on skewed results with follow up studies, as we do frequently. However, there is a problem when studies are funded and then not published, but used for decision making.prokaryotes (talk) 00:05, 1 September 2015 (UTC)

Addition to MEDRS for clarity

There have been documented [issues with industry funded studies] and even though they have shown to be overly positive and could even be unreliable, it is clear that, while possibly unreliable, these are allowed as several editors here have shown MEDRS saying we should not be performing a peer review ourselves. And yet, this stance is difficult to reconcile with the stance some editors on Wikipedia have taken regarding studies that come from other countries, which have shown issues similar to possible bias seen in industry funded research. As a result, I propose changing MEDRS which currently reads:

• Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

to:

• Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, conclusions, or country of origin.

Since the country issue has been brought up repeatedly and editors have rejected research because of the country it was published in, I don't see any alternative but to amend MEDRS to what I did above. If there are no objections (and I don't see why there would be, based on everyone's arguments for industry funded studies), I will make the change. LesVegas (talk) 20:26, 1 September 2015 (UTC)

You misrepresent the discussions that have been held here about Chinese publications about acupuncture. Those are about a specific topic and specific country, and the generalization you want to make here is not helpful. Your persistent efforts to portray this as a bias fall flat when you look at the publications about the general poor quality of acupuncture research publications in China. Jytdog (talk) 12:37, 2 September 2015 (UTC)

Any "problems" they may have are the same as seen in industry funded studies. You might want to familiarize yourself with these in order to discuss this material at substantive depth. LesVegas (talk) 23:02, 2 September 2015 (UTC)

I think it may be helpful to remove "funding sources": there's certainly been some questioning of cigarette/e-cigarette material where the publications are industry-funded, but I do worry that this question - essentially of whether a source is a WP:IS or not - is too tricky to codify in a few words, especially if we're going to get into questions about Chinese/Russian research e.g. - so would leave the change at that. It's also often struck me that the sentiment in the current MEDRS text sits uneasily with WP:FRIND for fringe topics. Alexbrn (talk) 13:08, 2 September 2015 (UTC)

Arthur, I appreciate your intellectual honesty and your desire for consistency here. I can see both sides to allowing industry funded studies on the article. On the one hand, correlations have been shown with industry funding bias regarding psychiatric medications failure to warn about cancer risk, and have shown a huge statistical increase in "positive findings" for various pharmaceuticals over those funded by other sources such as non-profits or the government. On the other hand, there could be other factors at play, and not all industry funded research isn't tainted so we can't throw the baby out with the bathwater, and so on. The same can be said for Chinese or Russian research as well. It's never been shown to be as simple as "fraud" or "lies". It could be that acupuncture performed on Chinese people by Chinese doctors in China has a more powerful effect than it does on unbelieving Westerners in the US when it's performed by caucasians. It could be that they perform the acupuncture stimulation differently, or use differently sized needles. It could also be that they conduct studies differently for ethical reasons, choosing a control group that receives medication instead of sugar pill, and that's why we see different results. If things like that are the case, it's much harder to dismiss it as simply unreliable. Russian research is also performed quite differently than US/European research as well, confounding the issues. I'm sure all around the world these things we see different results for different reasons. Ultimately, things just need to be consistent in our guideline here so that editors don't get confused. LesVegas (talk) 23:02, 2 September 2015 (UTC)

I added "country of origin" because I doubt editors here would want to remove funding sources in order to remain consistent, and I am also in that camp now. I really think there are far too many variables at play with these things and as Yobol said we ultimately can't throw out the baby with the bathwater. LesVegas (talk) 00:40, 4 September 2015 (UTC)

Your whole argument here is incredibly WP:POINTY. Jytdog (talk) 01:15, 4 September 2015 (UTC)

Thanks for the substantive argument. See, I knew you had it in you. LesVegas (talk) 12:19, 4 September 2015 (UTC)

On the merits of the proposal: We shouldn't add this. One of the important differences between "acupuncture in China" and "industry-funded research" is that we have a wealth of other sources about acupuncture, but for some subjects, there are no sources that are entirely independent of "industry funding". If we banned all sources from China, we could still write a decent article about Acupuncture; however, if we banned all sources that either received "industry funding" (broadly defined) or were based on sources that received industry funding, we could not write any articles about experimental drugs or new drugs. For example, the FDA just approved rolapitant, and I have just looked up all of the registered clinical trials. Every single one of them was sponsored by a commercial drug manufacturer. You will not find a source that provides biomedical information about rolapitant, that meets MEDRS's quality standards, and that does not use data produced through "industry funding". This gap is why the two cases (one subject in one country vs entire industries) are not comparable. WhatamIdoing (talk) 06:14, 4 September 2015 (UTC)

WhatamIdoing, as usual you make an interesting point here. Two problems I see: one, that we have no checks or balances on industry funded sources the way MEDRS is currently written. We have many independent studies on soft drinks, but research making health claims about soft drinks that's funded by Coca-Cola is still as allowable as anything else. There's no cutoff whatsoever. Ultimately, we just need some consistency, somehow. Two, have Chinese sources ever been proven inferior, biased or less reliable to Western sources on any subject? The critical literature looks pretty much the same with Chinese sources for acupuncture as it does for industry funded sources, although with better explanations for discrepancies in findings, such as different ethical standards the Chinese have, (for instance placebo controls aren't allowed in China because half the study group wouldn't receive treatment). So for those two reasons, there is a huge logical issue preventing consistent application of MEDRS here. We just need to fix it in some way. I'm open to whatever, but MEDRS needs to be remedied for universal consistency, somehow. LesVegas (talk) 12:19, 4 September 2015 (UTC)

Never feed the trolls.

I don't really understand what you mean that placebo controls aren't allowed–and I would very much like that sourced. Often placebo consists of current treatment which is then compared with a group receiving a new treatment – seldom is the alternative no treatment.-- CFCF 🍌 (email) 14:43, 4 September 2015 (UTC)

This is becoming tendentious. the sources discussing the poor quality of Chinese research on acupuncture (including self-reflective sources by Chinese authors) were discussed to death on this board not long ago. I don't want to be mean, but it is time to WP:SHUN this WP:POINTY and incompetent (e.g discussion of placebo) effort. LesVegas has been claiming systemic bias since his first month here - before he understood any of the policies and guidelines, letter or spirit, and LesVegas still does not understand what we are up to here. Jytdog (talk) 15:18, 4 September 2015 (UTC)

Jytdog, you should know to focus on the content, not the contributor. You calling me tendentious is highly KETTLISH. I have yet to see you focus on anything relevant whatsoever. I will respond to your personal attacks, one by one, and after that I will cease to respond here because it's not the forum. If you still have a bone to pick, please do it on my talk page. First, this subject was discussed to death here not long ago, here, but not in the context of keeping MEDRS consistent. I am suggesting we either have to change MEDRS's stance on carte blanche acceptance of funding-source tainted research, or allow research from all countries equally. Frankly, to maintain consistency, we cannot have it both ways. I'm not sure you actually understand WP:POINTY since I'm talking about being consistent rather than rigidly applying policies I disagree with consistently. I happen to understand many policies here better than you do, including systemic bias, because when I started editing Wikipedia I sought out training in the adopt-a-user program, which is something you never did. Since you raised issue with my competence for stating the fact that placebo controls are ethically problematic in Chinese research, I could do the same with you while pointing out that you've been editing Chinese articles in a highly opinionated, yet ignorant, fashion all along, but I wouldn't do that because I don't expect my fellow editors to be all-knowing. Again, I won't be addressing or even responding to your personal attacks again. I won't acknowledge anything but true content on article talk pages with you again, and frankly wouldn't have even replied to this here if it weren't for the fact that you have persistently attacked me on this talk page. Going forward, if you have a valid point about my suggestions, I would love to hear them. LesVegas (talk) 01:38, 5 September 2015 (UTC)

Request for a list of high quality journals

There has been a request for use to create a list of high quality journals people can feel comfortable using. Wondering what peoples thoughts on this are? Have begun drafting something here User:Doc_James/RS#Journal_articles Doc James (talk · contribs · email) 19:56, 25 June 2015 (UTC)

Great project. Since "Good-vs-bad" is not exactly a black-and-white issue, I'd say the more annotation and explanation of how conclusions are arrived at, the more useful it will be. -Pete (talk) 20:17, 25 June 2015 (UTC)

I also see the need for this. But I think it's neither pragmatic nor is it possible to create such a list. Plus, I think some editors are likely to edit war anything that's not on the official list right off Wikipedia. What I think might be more workable is to tighten our guidelines for what journals are acceptable and what journals aren't. Can we come up with a better list of guidelines, based on principles of what an unreliable journal versus a reliable one are? The ones we have are a bit too vague, I think. LesVegas (talk) 16:45, 30 June 2015 (UTC)

As I suggested on Wikipedia talk:WikiProject Medicine, I agree with Las Vegas that a clearer guideline would be more useful than a list. A list of the highest quality journals could be created, but couldn't the contents be summed up nearly as well with a generalized 3 - 5 point criteria for inclusion?Dialectric (talk) 17:24, 30 June 2015 (UTC)

I am concerned that such a list will be misunderstood, as others note above, as a binary yes-no type of list. We'll get editors who insist that since they read claim A in a journal on the list, it must be added to our Wikipedia article; we'll get editors who insist that because claim B isn't from a journal on the list, it must be removed from Wikipedia. Both attitudes will be problematic. We already have a similar difficulty with editors who don't understand – or wish to abuse – the 'try to use sources from the last 5 years' rule of thumb. This will likely be even more messy.

Sometimes bad papers appear in 'good' journals. Reputable, sturdy, solid science involving unsexy topics or with unflashy results ends up in middle-of-the-road or lower-impact journals—regrettably, often side-by-side with dramatic, sexy results produced by crappy science. (With a high variance, the product of "study importance" and "study quality" is proportional to a journal's impact factor.)

Evaluating the quality of a journal article is difficult. Doing it with competence requires, literally, years of experience and training. While I appreciate the impulse to try to help editors understand the standards for sourcing on Wikipedia, there is a limit to what we can do with rules of thumb and examples and suggestions. There isn't a magic six-point checklist that will let someone with little or no experience reading and evaluating scientific literature to learn to do so in five minutes (or five hours, or even five months). An extraordinary amount in such an evaluation depends on context and contextual knowledge. TenOfAllTrades(talk) 19:47, 30 June 2015 (UTC)

Something along the lines of WP:ELPEREN might be doable - creating a list of source that are frequently discussed where there is strong consensus on how they should be used in MEDRS situations. Such a list could even include Quackwatch. --Ronz (talk) 01:19, 1 July 2015 (UTC)

A list of high quality journals does not equal these sources are useable. Sources are reliable per the content they support and not because they are deemed to be high quality. While it might be helpful to have a list of high quality journals that list in no way determines reliability per Wikipedia content. (Littleolive oil (talk) 01:33, 1 July 2015 (UTC))

But if a list of "high-quality journals" is created, even if it is festooned with disclaimers about not being an absolute complete list, then it absolutely will get abused to say that X must be included because it was published in a journal on The List, and Y must be excluded because it was not published in a journal on The List.

Also, does anyone else remember a discussion at WPMED a few years back, in which a bunch of mostly pre-med students tried to figure out what "common" or "typical" textbooks were being used for one specialty (gynecology, I think)? It basically failed. If we can't even get a list of a half-dozen textbooks created, by people who had to buy such textbooks, then I believe that an effort to pick "good" journals is doomed. WhatamIdoing (talk) 04:03, 8 July 2015 (UTC)

Agree with the skepticism above. I think in general, having a list of sources that are generally considered high quality (as an essay, perhaps) would be helpful. However, I agree that such a list will lead probably lead to more issues than less if incorporated as a part of this guideline. Yobol (talk) 13:07, 9 July 2015 (UTC)

• Comment Isn't this the list/place you were looking for https://wiki.riteme.site/wiki/Medical_literature#Instances Links from lede ? prokaryotes (talk) 21:30, 16 August 2015 (UTC)

This is exactly what you're looking for: Brandon–Hill list. Unfortunately Dorothy Hill retired in 2004 and it ceased publication. Some people call Doody's List a replacement, but Doody's List isn't quite the same (and is much more expensive). --Nbauman (talk) 09:46, 6 September 2015 (UTC)

Medical organization guidelines

There is some discussion taking place on Talk:Acupuncture over the content in this edit, which is based on guidelines of numerous medical organizations such as the American College of Physicians, the American College of Chest Physicians. I wanted to know if these are the kind of organizations whose recommendations are considered to pass MEDRS as per WP:MEDORG. Comments on the acupuncture talk page would be appreciated. Everymorning (talk) 19:39, 18 September 2015 (UTC)

Please read this comment. See Talk:Acupuncture#Medical organization guidelines. QuackGuru (talk) 20:08, 18 September 2015 (UTC)

Mid Quality sources

I have noticed a trend in the last few threads... debates that focus on defining the source in question as either "low-quality" or "high-quality"... and not considering whether it might be "mid-quality". I think we all agree that high-quality sources are preferred, and that low-quality sources should be avoided. But what about mid-quality sources? What often happens is that supporters of the mid-quality source try to elevate it to high-quality status... while opponents try to demote it to low-quality status. I think it important to assess sources realistically... so perhaps we need some discussion on how to present mid-quality sources. Blueboar (talk) 12:33, 6 September 2015 (UTC)

I think the reason we don't speak of mid-quality sources is because there of is such a divide between the low and high quality sources. It might seem confusing but what we are saying is basically that we accept high quality and top of the line Cochrane-level sources. Mid-quality sources would be primary sources such as RCTs and well-performed single studies–check out the pyramids I added above. We don't discuss these so much because Wikipedia doesn't allow them (or at least strongly discourages them). -- CFCF 🍌 (email) 21:51, 6 September 2015 (UTC)

You know, this is an interesting query well worth exploring. I think what we'd first have to do would be define what a mid-level source is. Is it peer-reviewed, but not in a highly well-known journal like JAMA? Does impact factor have any bearing on level? Or is it less about the publication and more about the type of study? So are meta-analyses superior to systematic reviews for identical claims? Do higher level sources use established reporting standards for their field? I would think that despite what mid-level sources are, we might still allow them but give them less weight obviously excellent sources. What that would probably mean for the article is that we should only use high-level sources in the lede while mid-level sources could be used elsewhere. High level sources might ought to be listed first in a paragraph, while mid-level sources would come later. All of that would, of course, be dependent on what we agree mid-level sources even are. LesVegas (talk) 22:39, 6 September 2015 (UTC)

This is the same idea that you have proposed already at the talk page of the acupuncture article and has been rejected. See Talk:Acupuncture/Archive_28#Chinese_Source_Idea and Talk:Acupuncture#The_Chinese_question Jytdog (talk) 00:11, 7 September 2015 (UTC)

Its appropriate for an editor to bring comments and questions to the larger community especially since in this case LesVegas is responding to Blueboar above (Littleolive oil (talk) 01:30, 7 September 2015 (UTC))

You also argued in those threads for inclusion of Chinese sources on acupuncture in some way like this. I do understand that both and LesVegas and likely Herbxue are pleased to find an opening in what Blueboar wrote above, for what you already wanted; likewise it is reasonable that Blueboar should be informed of the conversation into which he/she is walking Jytdog (talk) 14:54, 7 September 2015 (UTC)

Once again your response is a personal attack. No, I did not argue for inclusion of any sources carte blanche, Chinese or otherwise. Blueboar has been around along time and is policy expert in my opinion. I'm sure he can handle this discussion. Sheesh Jytdog give it a rest.(Littleolive oil (talk) 15:13, 7 September 2015 (UTC))

No it isn't an attack, it is a simple description. You regularly advocate for alt-med in WP. You also blatantly misrepresent me. I did not write anything like "any sources carte blanche," - I wrote "in some way like this." I don't know what you think that gains for you. Jytdog (talk) 15:48, 7 September 2015 (UTC)

• I've moved this back to its own section, here. This was posed as a separate issue and question from the RfC above, and should be dealt with as such. There may be a better way to do this if so please feel free.(Littleolive oil (talk) 14:29, 7 September 2015 (UTC))

We should not be judging the quality of something based on whether it is published in a "highly well-known" journal, nor should impact factor have any bearing. There are hundreds (thousands?) of reputable journals applying the same level of peer review and oversight. We should reject fringe journals etc., but as a scientist I give exactly the same weight to something whether it is published in something like JAMA, Heart or Gut, or if it is in a less well-known, lower impact factor journal like International Journal of Social Research Methodology, Knowledge-Based Systems or Informatics in Primary Care (all IF < 1). Bondegezou (talk) 17:04, 14 September 2015 (UTC)

Bondegezou, I believe that some of our editors unfortunately believe that IF < 1 defines "fringe journal". It can be difficult to identify fringe journals if you're not in the field, so some of them try to take shortcuts like that. User:DGG was meditating on a page at Wikipedia:Impact factors (since redirected by User:LeadSongDog to WP:Scholarly journal) to explain the limitations of that approach.

User:Blueboar, I'm not sure that I can agree with User:CFCF's statement that mid-quality sources are primary sources, or that we are really that close to banning them. One of MEDRS's structural problems is that it was written primarily to handle treatments for major diseases. It applies poorly to the other (90%) article contents for major diseases, and to the 90% of diseases that aren't so "popular".  ;-) Fixing that is going to require encouraging editors to use their best judgment, instead of imposing one-size-fits-some rules such as "you must only use secondary sources" or "six-year-old sources are bad". WhatamIdoing (talk) 04:41, 23 September 2015 (UTC)

Obviously, it depends on the topic. In an area where the are first rate journals, it is safe to assume that the second-rate journals publish the less important work. Within a popular, field, for the same type of journal, the IF can be a good guide to this. In narrow fields, where relative few people work, the IFs of all the journals will be low, and we simply have to use the best material. It can take a certain amount of knowledge of the literature in the area to figure this out. A mechanical approach to reference selection or article writing doe not work well. DGG ( talk ) 04:53, 23 September 2015 (UTC)

Request for Comment: Country of Origin

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Should we change MEDRS, which currently reads:

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

to

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, country of origin, or conclusions.

This proposal is to address only the addition of high-quality sources into the guideline. LesVegas (talk) 23:33, 5 September 2015 (UTC)

Comments

Support Wikipedia should never get into the business of discriminating against high quality sources because of where that research might have been conducted or published. However, this has happened before, where multiple Cochrane reviews were rejected while trying to replace an old source with them, partly on the basis that "all but one of its authors were Chinese," and therefore, they must be biased. This is not an isolated incident either, with some editors rejecting all sources published in China not on the basis of case by case analysis of the source's quality, but stating they're published in China and assume they must all be unreliable. Low quality sources that, for instance, are not peer-reviewed, should always be rejected. But MEDRS does not yet make it clear whether or not high-quality foreign sources should ever be rejected on the basis of country of origin, whether it be the authors of a publication or place where high quality research is published. And interestingly, MEDRS currently allows for industry funded research not to be rejected on the basis that it's industry-funded, while it says nothing about a particular country's research. And yet, despite whatever potential problems could exist, it is very clear that industry funded research can be very problematic to add into Wikipedia. Just recently, Coca Cola came under fire for funding scientific research showing that Coca Cola doesn’t cause obesity. And bad as that may be, MEDRS currently doesn’t allow us to prohibit such research. It states:

'Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.'

There are also many well-documented problems with industry-funded research in the way of psychiatric medications failure to warn about cancer risk with certain drugs, as well as showing a huge statistical increase in "positive findings" for various pharmaceuticals.

However, just because there is a documented problem with some industry funded research, that doesn’t mean there is a problem with ‘’’all’’’ industry funded research. And oftentimes, new drugs only have industry-funded research for sources. No other sources can be found. Reasons like that are why it's not permitted to reject high quality sources on the basis of them being industry-funded.

At the same time, some editors currently reject some sources on the basis of them being published in another country. One such example is with acupuncture where a 1998 research report showed that in Hong Kong, Taiwan, Russia/USSR, China and Japan higher reports of positive findings occurred when compared to England. In some ways findings like these could possibly mirror the problems in industry-funded research. And at the same time, as with industry-funded research, it’s difficult to discount all of it on the basis that there is a chance some of the more positive findings might possibly be due to publication bias. Further complicating the issue is that more positive findings might be due to cultural differences in scientific research between various countries, such as ethical dilemmas with using placebos where the Chinese have shown statistical deviations from non-Chinese trials, creating conditions where placebo alone is not ethically justified as a control. There are many other possible confounding factors similar to this which may explain differences for findings between various countries once found within this specific modality. Undoubtedly, this seems to cloud the issue. But making the issue more problematic, for us as editors, is that some of the research mentioned in the Vickers source might not even be of high quality, thus causing making any argument against high quality research moot.

Adding "country of origin" seems to put the wording more in line with Wikipedia’s project Countering Systemic Bias, as well as our WP:BIAS policy. By ensuring Wikipedia editor-level peer review doesn’t extend to country of origin, it also makes it consistent with our current stance on liberal allowance of sources no matter what types of funding may be received.

UPDATE For anyone who thinks this issue isn't happening at all or was just one isolated incident, let me show proof it is still happening. Within the past few days, an editor rejected research in a high quality journal, Medicine, because the "authors are Chinese." Like it or not, rejecting sources based on ethnic origin of authors is a real problem. We need to examine sources, rejecting or accepting them based on their own merits and nothing else. LesVegas (talk) 20:01, 8 September 2015 (UTC)

LesVegas (talk) 23:51, 5 September 2015 (UTC)

Oppose I see no reason why this proposal is needed. We could also add that we shouldn't oppose inclusion based on authors skin color–because racism is bad, but it isn't needed. Level and quality of evidence is far more important. -- CFCF 🍌 (email) 00:56, 6 September 2015 (UTC)

There have been incidents where high quality sources were rejected for reasons that could've been prevented with amended language, such as this. I modified my first comment, adding in one example of such a rejection, although there have been others. Therefore, it appears it is needed. Let me know if you still think this way after reading the link. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Question What does "country of origin" in the proposal refer to? The country where the study was conducted, nationality of researchers or subjects, or country where the publishers of the journal or book are located? The first two are fine but perhaps unnecessary as CFCF says above (can LesVegas point to instances where such criterion was cited and prevailed?) If the last, I'll be opposed to the change since assessing the quality of journal and its publishers is crucial for assessing medical and other sources, and neither high-quality nor sham publishers are distributed uniformly across the globe, and we on wikipedia cannot solve this real world problem by simply turning a blind eye to it. Abecedare (talk) 01:04, 6 September 2015 (UTC)

Good question. "Country of origin" is all-encompassing, meaning country of author, country where research was conducted and where it was published. Keep in mind, this RfC is asking about high-quality peer-reviewed publications that meet every other standard of quality in MEDRS. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Comment This appears to be an attempt to allow Chinese studies about acupuncture published in Chinese journals to be used to support acupuncture. We need to use high quality journals with a reputation for reliability. So oppose the underlying attempt. Doc James (talk · contribs · email) 01:27, 6 September 2015 (UTC)

Actually, there was an incident where an influential editor opposed the addition of a Cochrane Review because its authors were from China. DocJames, please limit your comments to high quality sources. This is what the discussion is pertaining. LesVegas (talk) 01:52, 6 September 2015 (UTC)

To which Cochrane review do you refer? Doc James (talk · contribs · email) 04:05, 6 September 2015 (UTC)

Actually, several. One of them was a review on fibromyalgia which had several authors that were Chinese. High quality sources should never be rejected on such a basis, don't you think? By the way, upon re reading your comment, are you suggesting Chinese studies or Chinese journals are all inherently low-quality? I certainly hope that's not what you were saying. Tell me I read that wrong. LesVegas (talk) 04:24, 6 September 2015 (UTC)

Yes that Cochrane review is a fine source to use. No I am not saying all Chinese journals are inherently low-quality. Many journals in many places however are low quality. Doc James (talk · contribs · email) 04:29, 6 September 2015 (UTC)

• Agree with Doc James. I have seen zero examples of any attempt to reject high-quality foreign sources on the basis of simply being from another country. The Chinese studies about acupuncture published in Chinese journals were rejected not because they were published in China, but rather that they were not high quality journals with a reputation for reliability. There are plenty of fringe journals pushing pseudoscience in the US and UK, and plenty of good, peer-reviewed science gets published in China. --Guy Macon (talk) 01:48, 6 September 2015 (UTC)

Guy Macon , you might check out the diff below where an editor rejected one such source on this very basis. You can't get much higher quality than Cochrane Reviews, and you can't get much lower editing than to reject one all because "all but one of its authors are Chinese." It's sad we have seatbelt laws and equally as sad we have to tell editors not to reject high quality sources because of things like where they're published, but I'm afraid that's the case. It's exhausting to waste time on such silly matters and doesn't hurt anything to add it in because we're talking about high quality, peer reviewed sources here anyway. LesVegas (talk) 02:34, 6 September 2015 (UTC)

I did check out the diff. As others have pointed out, it does not say what you claim it says. --Guy Macon (talk) 05:39, 6 September 2015 (UTC)

Guy Macon, I've seen you around and know you're a good editor here. I know it's shocking to think another editor would ever argue such a thing, but I was in the middle of that one and I can tell you that's what was meant regarding a set of Cochrane Reviews used to replace old research. Kww argued it shouldn't be used to replace an outdated review per WP:MEDDATE for several reasons, one being that the Cochrane Reviews used Chinese authors. If you would like more details on that, I'm happy to provide them. Again, I know it's hard to believe editors would ever say or argue such a thing, but sadly they do. The other Guy constantly rejects sources because they're Chinese, and you can see for yourself how he stereotypes them as universally unreliable. No regard for peer review in statements like that, just outright rejection of sources because they are Chinese. We shouldn't even have to discuss such matters as ethnicity of authors or place of publication on talk pages, it's distracting and disruptive; editors need to be focusing on quality of individual journals and that's it. LesVegas (talk) 06:09, 6 September 2015 (UTC)

• Comment Pending further info Is this change meant to address a non-hypothetical problem? Excluding a study based solely on country of origin is so strange that I'd be surprised to see that anyone had attempted it, let alone done it successfully, but has it happened? What were the circumstances? When I first saw this RfC, I thought "This looks like it's probably about something else" and DocJames' scenario seems consistent with this. Other comment: What counts as "personal"? Darkfrog24 (talk) 02:07, 6 September 2015 (UTC)

Darkfrog24, you might want to check a diff out where an editor opposed updating an old source with a series of Cochrane Reviews because all but one of its authors were Chinese. Yes, rejection of high quality sources based on country of publication or origin of its authors is a real problem. As long as sources are high quality, issues like country of origin or ethnicity shouldn't matter. LesVegas (talk) 02:28, 6 September 2015 (UTC)

Well this diff doesn't show someone saying, "Exclude these sources because they're Chinese." It's more like Kiwi saying, "I think this guy wants to exclude the sources because they're Chinese," but whatever. I'm changing my comment to mild support. The idea that we shouldn't exclude sources solely on country of origin is so obvious that we all should be doing it already and if a source really is unreliable, there should be other reasons to exclude it. Darkfrog24 (talk) 03:39, 6 September 2015 (UTC)

+support based on the diff linked above, where an editor simply assumed that a study was biased because other studies by other authors of the same nationality had been accused of bias. This is out of line. DES ^(talk) 03:17, 6 September 2015 (UTC)

Darkfrog24 and DES what you see operating in the diff is WP:REDFLAG. The key word in the diff is "suspect"; which is very different from "excluded". Jytdog (talk) 13:50, 6 September 2015 (UTC)

The key phrase in the diff is "I also note that of the supposed refutations, all but one include Chinese researchers on the papers: we've discussed that bias problem extensively, and there's no reason to believe it doesn't apply here" This is drawing a conclusion that the mere presence of Chinese researchers equates to bias. This is not acceptable. DES ^(talk) 13:57, 6 September 2015 (UTC)

DES, thanks for replying. LesVegas and others have been pushing and pushing and pushing to include Chinese sources on acupuncture in our article - the issue has been discussed a zillion times but keeps coming up, and it gets frustrating, and people write things quickly that are easy to misconstrue when diffs are cherry-picked, as LesVegas has done with KWW's statement. Jytdog (talk) 14:07, 6 September 2015 (UTC)

User:Jytdog, would a rule saying, "do not exclude based solely on country of origin" make it harder for you to exclude low-quality or otherwise problematic sources? I'm not familiar with this particular dispute, but it sounds like those sources are no good regardless, so this shouldn't make any difference. Would it help if another line were added elsewhere saying, "Sources that have not survived peer review may be excluded" or "sources from publications known to routine publish and fail to retract unreliable material may be excluded"? Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

The background facts are:

a1) our article on Acupuncture has been highly contested for a long time and is subject to DS per Arbcom - the core dispute is between science-based editors and believers in acupuncture (as with many alt-med topics);

a2) some acupuncture believers actually try at some level to deal with the relevant Wikipedia policies and guidelines (NPOV and particularly PSCI, FRINGE, MEDRS);

b1) Research (and reviews) produced in China about acupuncture (and other TCM) overwhelmingly come to positive conclusions about the efficacy of TCM for various diseases and conditions, and they come to positive conclusions way more often than research and reviews produced in the West.

b2) This discrepancy is explained by believers (in published sources) in various ways (e.g Westerners don't do TCM properly or there are special facts about TCM that make it impossible to conduct sound scientific research on it); scientists have explained this discrepancy in published sources by pointing to the immature level of development of Chinese scientific institutions.

c1) When you put a) and b) together, you get some acu-proponents pushing to use reviews originating from Chinese scientific institutions to support content in our article about the benefits of acupuncture (content that is not supportable with, or even contradicted by, other sources), and science-based editors tending to reject that content and its sourcing - especially when no other sources are brought (see the first bullet of WP:REDFLAG). This has been discussed extensively on this Talk page as well as on the article Talk page and the consensus has been to treat sources about TCM originating from China as "suspect" for the time being - not excluded, but not "green flagged" as high quality. This RfC was posted by an acu-proponent. You can see my response to it below. Jytdog (talk) 15:05, 6 September 2015 (UTC)

Jytdog, what is your opinion on rejecting Cochrane Reviews because its authors are Chinese? Are there no publications in China that are reliable? Do you have a source that all are unreliable and engage in publication bias? Do you think there could be other possibilities for differing results, such as ethical dilemmas with placebo controls in more instances? And since scientists have documented the same issues you complain about in Chinese research, as with industry-funded research, why do you hold a different standard for Chinese sources? LesVegas (talk) 15:22, 6 September 2015 (UTC)

• oppose The framing of serious issues with Chinese research publications on acupuncture as "discrimination" is the worst kind of red-herring, bad-faith, manipulative argument imaginable, especially when Chinese scientists themselves point out the problems. This RfC is the pits. There are solid sources describing the problems:
  • Li J, et al The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012. BMC Complement Altern Med. 2014 Sep 26;14:362. PMID 25256890
  • Vickers, Andrew (April 1, 1998), Do certain countries produce only positive results? A systematic review of controlled trials., Control Clin Trials {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Ferguson, Cat (November 26, 2014), Publishing: The peer-review scam, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Qiu, Jane (January 12, 2010), Publish or perish in China, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)

The quality will surely improve with time but LesVegas' effort to wave a magic wand and make the actual problems disappear under the banner of "discrimination" runs hard against everything we try to do at WP:MED with regard to using high quality sources. Jytdog (talk) 04:37, 6 September 2015 (UTC)

Jytdog, this RfC is about high-quality sources, not low quality ones which are clearly not allowed. But your inability to differentiate between these two categories and continue stereotyping all Chinese studies and Chinese authors the way you just did isn't surprising since you posted that scientific institutions in China aren't independent or mature, and therefore all of their journals are unreliable. Frankly, comments like that are why RfC's like this exist. But back to the discussion at hand. You never addressed high quality sources, which is what this RfC is pertaining to. You are opposing low quality ones, which we all should oppose on the basis of things like lack of peer-review and things of that nature. But opposing them simply because they are Chinese isn't acceptable, don't you think? LesVegas (talk) 04:57, 6 September 2015 (UTC)

You continue to mischaracterize the situation and to employ circular reasoning. The systemic problems with Chinese research on acupuncture are well documented and so the assumption going in, is that they are not high quality; per Guy's diff they are "suspect." The burden is on the person bringing any such source to show that the specific source is high quality. Per WP:REDFLAG one way that could be done would be to show that sources that are high quality say the same thing. This is not racism, it is dealing with immature level of development of Chinese scientific institutions, just like it is not racist to say that the institutions of Chinese law in the fields of environment and intellectual property are not mature. The air quality in Chinese cities is often poor; copyright infringement is rampant in China; Chinese publications on acupuncture almost always have favorable outcomes. Those are simple statements of fact, reflecting a society in development. They are not essentialist, racist statements - they are existential statements of fact and will change over time, as the situation in China changes. So WP:DROPTHESTICK. And I suggest you be much more careful in deploying this tactic of cherry-picking diffs and trying to discredit editors as racist. Jytdog (talk) 13:50, 6 September 2015 (UTC)

Jytdog, just the other day you said "we do not perform peer review on sources and should never reject industry-funded sources that almost always have positive findings. But Chinese research? Nope, Chinese scientists aren't independent or mature. I'm not trying to characterize you as racist, but frankly by defending one standard for industry-funded studies, and enforcing another for Chinese research while stereotyping all Chinese research as unreliable, never to be evaluated on a case-by-case basis, makes it difficult for me not to find a diff that characterizes you any other way. But let me allow you the opportunity to prove that you're not racist. Simple question: why should we allow one standard for industry funded research despite documented publication bias issues, and another standard for Chinese research, rejecting the notion that we should evaluate Chinese sources or Chinese authors on a case-by-case basis like we do everywhere else? LesVegas (talk) 15:01, 6 September 2015 (UTC)

Your mischaracterization of what I wrote and of MEDRS is malicious, incompetent, or both. This is not about conducting peer review, which involves critiquing the quality and extent of the experiments that were done, the presentation of the resulting data, and the conclusions drawn from the data. This has nothing to do with research funding. And I said nothing about "Chinese scientists" - I addressed institutions. Jytdog (talk) 15:14, 6 September 2015 (UTC) (strike; should not have written that. my apologies. Jytdog (talk) 15:48, 6 September 2015 (UTC))

Oh but it is about conducting a peer review based on industry funding. Those have the same documented issues you complain about in Chinese research, although I would argue, worse, since there are more confounding factors between Eastern and Western cultures (like different scientific ethics) and those can explain reasons for discrepancies in results between East vs. West, not simply publication bias. Simple question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? LesVegas (talk) 15:33, 6 September 2015 (UTC)

You do not seem to understand what happens during peer review of scientific papers; I explained that above. Jytdog (talk) 15:48, 6 September 2015 (UTC)

Jytdog, this is not about peer review in the publication process. I hope you aren't WP: IDHT'ing Yes, scientific papers are peer reviewed before being published in reputable journals. But the peer review we are talking about is that "Wikipedia editors should not perform peer review", specifically, in this instance, by rejecting high-quality sources on the basis of industry funding behind piece of published research. Again, please answer the question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? I really hope you can answer it. LesVegas (talk) 16:09, 6 September 2015 (UTC)

We already do evaluate on a case-by-case basis, and since this hasn't been a problem we don't need a clause against it in our guidelines. In order that they be followed–guidelines need to be succinct. We can't indiscriminately add clauses in order to address hypothetical problems or noone will read them–making it far harder to police.-- CFCF 🍌 (email) 05:44, 7 September 2015 (UTC)

On the face of it, the OP appears to want to get his low quality pro-acupuncture sources into our Acupuncture article by claiming they are high quality. I don't think the community is quite that silly. -Roxy the dog™ (Resonate) 10:28, 6 September 2015 (UTC)

Previously uninvolved editor here: 1) Even if the sources were rightly excluded, "they're Chinese" should not be numbered among the reasons for this; their own failings should be enough. Question: Why would "don't exclude based on country" make any difference? 2) Aside from this acupuncture issue, is the argument of exclusion-based-on-country a common enough problem to merit explicitly banning it? Do the words "don't exclude based on country" earn their keep for the space they take up (in Wikipedia's already Byzantine rule structure). Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

"They're Chinese" is a poor shorthand for a complex discussion and is not how the issue should be discussed nor should that phrase be used as a description of the stance of those opposed to this RfC or who treat sources about TCM produced by Chinese institutions as suspect. I am unaware of this issue arising outside the context of TCM. Jytdog (talk) 15:19, 6 September 2015 (UTC)

I can be more succinct than JD. 1)No. 2)No (to my knowledge). 3)No. -Roxy the dog™ (Resonate) 15:38, 6 September 2015 (UTC)

Oppose. While I agree that certain fringe journals are focused in particular countries, I don't think we should be determining reliability overall by country of origin. However, that can be a first red flag on certain topics for a deeper look. That's why I would oppose this language as I don't think it is needed. If a question on reliability comes up, there will be other qualities we look at for journals as well. There is also potential this language could be abused from a WP:BEANS approach (acupuncture does come to mind), so I think it's better that this is one thing left unsaid, but consensus in discussions on the idea that country or origin alone can't be used can always be linked if someone brings it up. Kingofaces43 (talk) 16:35, 6 September 2015 (UTC)

Strong oppose The focus should remain on the quality of the research and the reliability of the source. Either the research is good, or it is not. Either the source is reliable, or it is not. In what possible way does "country of origin" factor into anything, except to imply that some countries are doing poor quality research and publishing in unreliable sources, and must therefore be held to a lesser standard? TechBear | Talk | Contributions 20:39, 6 September 2015 (UTC)

I agree with you that the focus should always be on the quality of the sources themselves. In MEDRS it states, "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions". We aren't supposed to reject sources purely on those merits, just simply focus on the quality of the source itself, and yet MEDRS has to be explicit because I can only assume in the past editors have rejected high quality sources because of things like references, inclusion criteria, etc. Given that editors have rejected clearly high quality sources on the basis of nationality of author or country of publication (as seen from the diffs I provided) why haven't we reached a point where we need to explicitly tell editors they need to be focusing only on source quality and nothing else, like country? LesVegas (talk) 22:22, 6 September 2015 (UTC)

Support I have read most of this RFC and debated commenting. Most of the responses do not answer the question. The question is specifically about High quality sources. High quality sources should never be excluded based on the country of origin or nationality of the authors. Low quality sources will be rejected based on the fact they are low quality and this change will not allow them in regardless. I have looked at the diffs and there is a problem in some instances, though how wide spread is hard to say. AlbinoFerret 15:59, 7 September 2015 (UTC)

A blog post by Edzard Ernst, a leading academic in this area, reads in large part:

"In this case, you might perhaps believe Chinese researchers. In [this systematic review http://online.liebertpub.com/doi/pdfplus/10.1089/acm.2014.5346.abstract], all randomized controlled trials (RCTs) of acupuncture published in Chinese journals were identified by a team of Chinese scientists. A total of 840 RCTs were found, including 727 RCTs comparing acupuncture with conventional treatment, 51 RCTs with no treatment controls, and 62 RCTs with sham-acupuncture controls. Among theses 840 RCTs, 838 studies (99.8%) reported positive results from primary outcomes and two trials (0.2%) reported negative results. The percentages of RCTs concealment of the information on withdraws or sample size calculations were 43.7%, 5.9%, 4.9%, 9.9%, and 1.7% respectively.

The authors concluded that publication bias might be major issue in RCTs on acupuncture published in Chinese journals reported, which is related to high risk of bias. We suggest that all trials should be prospectively registered in international trial registry in future.

I applaud the authors’ courageous efforts to conduct this analysis, but I do not agree with their conclusion. The question why all Chinese acupuncture trials are positive has puzzled me since many years, and I have quizzed numerous Chinese colleagues why this might be so. The answer I received was uniformly that it would be very offensive for Chinese researchers to conceive a study that does not confirm the views held by their peers. In other words, acupuncture research in China is conducted to confirm the prior assumption that this treatment is effective. It seems obvious that this is an abuse of science which must cause confusion.

Whatever the reasons for the phenomenon, and we can only speculate about them, the fact has been independently confirmed several times and is now quite undeniable: acupuncture trials from China – and these constitute the majority of the evidence-base in this area – cannot be trusted."

This gives us a convenient way of identifying a large section of the literature as pseudoscience sensu stricto, and not to be regarded as RS. Richard Keatinge (talk) 16:17, 7 September 2015 (UTC)

Richard, I am glad you are giving this more analysis than others have. I have asked myself the same question- why have variations been noted? While publication bias is one possibility, there are others. For instance, placebo controls are rarely done and for ethical reasons control groups receive actual treatment of some sort. Another possibility is that in China they are performing acupuncture differently. The first time I ever had acupuncture was when I was in China, and after an accident, and I can tell you it was a very different treatment than any of the acupuncture I have had since returning to the US. It is also very different from Japanese acupuncture, or Korean, all of which I have experienced. The Chinese acupuncture I experienced used thicker needles and they adjusted them in such a way that elicited very strong stimulation. It was also done in a hospital, by medical doctors, so I knew in the back of my mind this was legitimate and this gave a different experience overall. The thing is, there are many variables. But worst case scenario, statistics have shown a 24 percent increase in positive findings of Chinese research versus Western research regarding acupuncture. That's still lower than the stats on industry-funded drug trials when compared to independently-funded sources, and yet we have explicit language in MEDRS that we shouldn't reject high quality sources because of funding. So why shouldn't we be consistent? LesVegas (talk) 18:26, 7 September 2015 (UTC)

• oppose it is an attempt to run around the essential purpose of MEDRES: that we only use the highest quality sources. where there is evidence that medical sources from a particular country do not measure up to the standards generally required we should NOT be using them. -- TRPoD aka The Red Pen of Doom 17:05, 7 September 2015 (UTC)
• Support: Adding country of origin to the policy/guideline is appropriate; nothing prevents anyone from applying a rigorous critique to the quality of a particular source. It is important not to mass-categorize entire groups of authors or publications simply by where they originate. It, for example, an academic journal from nation Foo publishes bad science, it can be assessed on its own merits or lack thereof, not the ethnicity of those producing it. Montanabw^(talk) 23:41, 12 September 2015 (UTC)
• Support. Inclusion should not be arbitrated on account of the author's, nor the content's cultural origin. The opposition makes claims that the guideline amendment is unnecessary, as such is already generally accepted, however I do believe in a differentiation between the two. We need a reference-able guideline to control cultural prejudice, which serves no place on Wikipedia. There should be no question of interpretation, the guidelines should be definitive. ExParte ^{talk | contribs} 06:18, 13 September 2015 (UTC)
• Oppose quote The framing of serious issues with Chinese research publications on acupuncture as "discrimination" is the worst kind of red-herring, bad-faith, manipulative argument imaginable, especially when Chinese scientists themselves point out the problems. This RfC is the pits. There is an awful lot of bad faith characterisation in this thread. Of course country, nationality etc. should not in itself be a criteria, but if the problem really did exist (which I'm not convinced of by the instances given), this wording would do nothing to solve it but simply distract from the central question of the quality of the source in a POINTY manner. Pincrete (talk) 19:04, 19 September 2015 (UTC)

There were several instances, one where Cochrane Reviews were being suppressed because of Chinese authorship as well as high quality journals such as Medicine where the same objections based on Chinese authorship were made. Since this RfC was on high quality sources only, can you please clarify: when we have obviously high quality sources, do you think country of origin should be an issue at all? LesVegas (talk) 22:50, 19 September 2015 (UTC)

Re country of origin question. Obviously not in itself. I don't ordinarily edit in Med. areas, (summoned by bot), however it is simply an established fact that some sources are less reliable than others in all areas (US Govt. 'collateral damage' figures?). I don't see why 'country' would be identified any more than any other factor. If there is real evidence of rejecting sources solely, or principally, on grounds of ethnicity or provenance, this is not the way to solve it. I was not persuaded that there was such evidence. You (and others) are repeatedly saying these are high quality sources, AS A FACT, but is that not what is disputed? Pincrete (talk) 14:20, 21 September 2015 (UTC)

I have never seen anyone dispute, for instance, that Cochrane Reviews or publications from Medicine, are not high quality publications unless they have Chinese authors in the studies they publish (which has been disputed). In fact, MEDRS says they unequivocally are high quality sources. One aspect that has been disputed is that if they have authors who are Chinese, then that makes them unreliable. My position is that country of origin shouldn't ever be used as an argument to exclude high quality sources. What we instead should be focusing on as editors is if the source passes MEDRS's established high-barometer for reliability and that's it, never rejecting sources because of things like country of origin. Do you agree? LesVegas (talk) 17:14, 21 September 2015 (UTC)

I was invited by a bot to make a comment, I am not either competent or willing to discuss the reliability of specific research. The comment I made is that the examples given do not persuade me that research is being rejected solely or primarily on the grounds of ethnicity or nationality, and that further, if it were, this would not be an effective means of remedying the problem. The provenance of any source, and its reputation for checking, is always a factor in assessing its reliability, whether we are discussing Russia Today, Fox News, Daily Mail, or a Govt. statement. I am in-expert on the protocols of medical trials, but even I know that allowing for 'placebo effect' is a cornerstone of such trials, yet you dismiss this factor above as 'Chinese doctors have an ethical objection'. Fine, then such trials have not been conducted according to long-established medical standards. IF the proof of 'racism' or irrational 'national prejudice' were as clear-cut as you appear to believe it to be, you should be taking this matter to a much higher 'court' in WP than this RfC. I am not persuaded that you are correct. Pincrete (talk) 07:47, 22 September 2015 (UTC)

• Oppose There's no noted problem with this occurring. The objections to Chinese sources for acupuncture and Indian sources for ayurveda is due to noted and repeated bias problems, not due to some imaginary racist motivation.—Kww(talk) 16:05, 23 September 2015 (UTC)

Actually, one of the noted problems was with you, when you deemed a series of Cochrane Reviews suspect because they had Chinese authors. I just wanted to correct the record here. I'll also note you continue to defend this behavior even today. Nobody is saying there's racist motivations on your or anyone's part for doing this, btw, just a wrongful assumption where you believe that because some studies have at one time shown possible (not proven) bias, all are therefore unreliable, even extending to what we consider the highest quality sources like Cochrane. LesVegas (talk) 20:17, 23 September 2015 (UTC)

You confuse "LesVegas is concerned about an issue" with "there actually is an issue".—Kww(talk) 20:57, 23 September 2015 (UTC)

• Oppose. Where there is a proven bias, as in the case of Chinese studies of acupuncture, we should just ignore them. (Off topic, but I also believe we should treat industry-funded reviews with a high degree of skepticism, and should ignore them when good, independently funded reviews are available.) --Anthonyhcole (talk · contribs · email) 02:05, 26 September 2015 (UTC)
• Oppose It's unnecessary to add that kind of language. We accept a source if it's published in reputable journals regardless of their origin. This proposal will only work for POV pushers as an avenue to weasel their not so reliable sources into articles, claiming regional bias. Darwinian Ape ^talk 08:50, 30 September 2015 (UTC)

Call for close

This thread should be closed as POV pushing and trolling. When Jytdog wrote "When the institutions that support science in China become independent and mature, their journals will become valuable sources. They are not there yet"[2], only to have LesVegas characterize it as " Chinese scientists aren't independent or mature"[3] -- something Jytdog never wrote or implied -- it became clearly evident that LesVegas is more interested in winning the argument than seeking the truth. Demonizing Jytdog as a racist is trolling, and we should close down this discussion rather than rewarding such behavior with further attention. --Guy Macon (talk) 20:00, 6 September 2015 (UTC)

Oppose closeI find it a little odd that nobody seems to want to answer why we have one standard for industry funded sources (written into MEDRS), and an entire other standard for Chinese sources where we can reject them on the basis of being Chinese and not on their own merits. Nobody has dared answer that question, and here, less than 24 hrs after the RfC was filed, an attempt has been made to close it before anyone answers this obvious and frankly embarrasing inconsistency. How paradoxical to then accuse me of not trying to seek the truth. And please don't accuse me of demonizing Jytdog as racist. He accused me of accusing him of racism, which is not the same thing at all. He said I was finding diffs that made him look racist and I told him essentially that was his problem, not mine. As for institutions, are they faceless, humanless entities or are they made up of human beings? So I suppose the institutions can be immature or not independent, but the scientists and publishers composing the institutions are mature and independent? Really? Especially when Jytdog and other editors are on record rejecting sources because they're Chinese, and not because of, say, lack of peer-review or lack of Medline indexing. If you read Jytdog's many diffs in full context he makes it clear it is impossible for anything Chinese to ever be considered reliable. I have not once seen him state, even here, otherwise. And at the same time he defends the practice of not rejecting sources because of industry-funding that has been documented to skew results, but opposes all Chinese research on acupuncture for those same reasons. I have asked multiple times for an answer, and he ignores it. Please, somebody, can somebody answer it? If we can come to a consensus on how to reconcile that glaring issue, and we can come to a consensus that high quality sources shouldn't be rejected on the basis of ethnicity of author, nor should they be rejected purely because of where they're published, I'm fine with that. When editors reject friggin Cochrane Reviews because authors are Chinese, we have a serious problem. LesVegas (talk) 22:11, 6 September 2015 (UTC)

I'm reminded of MastCell's WP:CGTW#8: Anyone who edits policy pages to favor their position in a specific dispute has no business editing policy pages. Corollary: these are the only people who edit policy pages. Yobol (talk) 22:16, 6 September 2015 (UTC)

• Support close This RfC is an obvious attempt to win a content dispute by declaring that editors objecting to certain sources are racist. Johnuniq (talk) 23:45, 6 September 2015 (UTC)

Oppose close - I do not think it is appropriate that everyone is focusing on the editor instead of answering the basic question this thread is asking - some of those opposed to the policy edit proposed asked some version of "why would country of origin matter?" Well, exactly, it shouldn't matter. And yet sources are rejected because "they're Chinese" and that is wrong. If there were reliable ways of gauging the quality of the research, each review can evaluated on its own merit, as it should be. The fact that the authors are Chinese shouldn't even come up, but it frequently does. So, we have a problem.Herbxue (talk) 14:12, 7 September 2015 (UTC)

Note to closer: Herbxue also has argued for using sources stemming from Chinese institutions in acupuncture articles and claimed discrimination with regard to applying REDFLAG to them Jytdog (talk) 14:50, 7 September 2015 (UTC)

Actually read that link again: I say "fine" to rejecting Chinese journals that do not have a reputation for quality, what I call "outrageous" is rejecting a review in a mainstream journal because the authors names are Chinese, and yes that actually occurred, and thus we have a true problem.Herbxue (talk) 15:25, 7 September 2015 (UTC)

You continue to conflate systematic problems with Chinese science on acupuncture with racism. This is not OK. Please provide me with any diff where you express an acknowledgement of the problem (there is not one in that diff btw - you acknowledge occasional problems, not institutional ones) I very much agree that the issue is not about individual Chinese scientists. Jytdog (talk) 15:59, 7 September 2015 (UTC)

Focusing on individual editors is taking the focus away from the debate. I have acknowledged problems I saw with TCM research firsthand in China, but I do not generalize them, and I do not believe there is adequate evidence to claim there is a "systematic" problem with "Chinese research". For me the issue here is whether Chinese researchers are inherently unreliable, and I don't believe WP editors can say that they are without evidence showing individual journals or educational institutions to be unreliable. If a particular review appears to be of low quality, then that in itself is reason not to use it. But if it appears to have solid methodology, I don't think it should be rejected on the basis of being "Chinese". Herbxue (talk) 21:20, 7 September 2015 (UTC)

• Support close Despite what our resident acupuncturist says, we should not use unreliable sources, (which is what they are) wherever they come from. -Roxy the dog™ (Resonate) 14:28, 7 September 2015 (UTC)

• This might have been an interesting RfC and it asks some very real questions in terms of sources. I would have been interested in pursuing the ideas presented here from all sides however, once again, discussion is shut down with name calling and personal accusations so that mature discussion is thrown out.(Littleolive oil (talk) 14:35, 7 September 2015 (UTC))

Note to closer: Littleolive oil also has argued for using sources stemming from Chinese institutions in acupuncture articles and per this agrees with Herbxue who commented above Jytdog (talk) 14:50, 7 September 2015 (UTC)

Thank you for including a link which belies what you mean to be an accusation and attack and links to my cmts which you paraphrase inaccurately.(Littleolive oil (talk) 15:17, 7 September 2015 (UTC))

• • I am becoming increasingly concerned with Jytdog's constant attacks and his inaccurate representations of editors and what they are saying and doing.(Littleolive oil (talk) 15:28, 7 September 2015 (UTC))

Naming you as an alt-med advocate is not a personal attack; it is a description that is easily supportable with diffs. And as I noted here, you are the one making blatant misreprentations here, as did LesVegas as noted in post opening this section. I don't know what you think this drama-mongering gains for you. Jytdog (talk) 16:36, 7 September 2015 (UTC)

Just so you know, being an alt-med advocate and being an editor who wants to make sure alt-med is treated neutrally and fairly on Wikipedia are two completely different things. LesVegas (talk) 18:08, 7 September 2015 (UTC)

Jytdog. No diif or diffs support overarching, sweeping generalizations regarding other people. Making false statements about people is a personal attack on those people and what they are and stand for. Wikipedia is not the real world, a place in which people are multifaceted and carry on their lives in complex ways. Do you realize that the use of exercise and the studies on its impact on health are relatively new. Is this alternative medicine and if so how many sensible people in the world today support this alternative medicine. If I use antibiotics but support exercise does this make me a supporter of alternative medicine. You are constantly making statements which attach motive and meaning to people and their actions. What do I get out of this. Nothing expect that I am tired of seeing false narratives created by editors which in the end are used and applied to sanction. It is possible to disagree with people and to even be aware that our own perspectives are based on our own point of view. Discussion which does not attack but attempts to understand and compromise can go along way towards creating good articles. You have once again made statements about me which are false with no proof whatsoever. You are creating a false narrative about another editor. You know nothing about what I support and do not support. This the third time. (Littleolive oil (talk) 17:25, 7 September 2015 (UTC))

Oppose close RFC's should run 30 days, this one is only 2 days old. Regardless if some are reading into the question something it doesnt say, it needs to run its course. AlbinoFerret 16:04, 7 September 2015 (UTC)

Support close, this RfC is going nowhere and is sufficiently ill-framed that it stands minimal chance of producing anything useful. Richard Keatinge (talk) 16:19, 7 September 2015 (UTC)

• Oppose close: RfC was simple, and straightforward concept; IDHT problem of editors reading more into it than exists notwithstanding. Montanabw^(talk) 23:41, 12 September 2015 (UTC)

• Oppose Close. This proposition brings up a valid point. Country of origin should not be, in general, a criteria for inclusion of a medicinal article. It doesn't matter what the purpose of presentation was, this should be a part of the guidelines. What's the issue with additional guidelines for inclusion arbitration? ExParte ^{talk | contribs} 05:34, 13 September 2015 (UTC)

• Country of origin should not be a blanket basis for condemnation across all possible subjects, but it might be for specific countries in specific subjects. It would certainly be the case for the subject and country being discussed be on the evidence a reason for at least the greatest skepticism. The prosposed modification is too great. The cases will need to be discussed individually. DGG ( talk ) 20:11, 13 September 2015 (UTC)
• Support close … This RfC is an obvious attempt to win a content dispute by declaring that editors objecting to certain sources are racist. and , this RfC is going nowhere and is sufficiently ill-framed that it stands minimal chance of producing anything useful. This seems like a lot of airing of bad feeling on the part of those who want different standards to apply to Alt Meds. Pincrete (talk) 19:12, 19 September 2015 (UTC)

This problem is still occurring

I have seen comments suggesting editors here never rejected sources because their authors are Chinese, despite diffs, so allow me to provide one more. Just yesterday, an editor did just that, rejecting a high-quality source because it had Chinese authors, so clearly this is a serious and ongoing problem that must be dealt with. This editor rejected research in a high quality journal, Medicine, because the "authors are Chinese," and therefore assumed to be incapable of not being biased Like it or not, rejecting sources based on ethnic origin of authors is a real problem. Sources in low quality journals should be rejected based on their own lack of merits. Sources in high quality journals should be accepted on their own merits. But sources should never be excluded for embarrasing reasons like this and it is a shame that in 2015 we have to have to write explicit language into our guidelines to keep behavior like this from happening. LesVegas (talk) 20:18, 8 September 2015 (UTC)

I think you're grossly oversimplifying a significant issue. It has been recognized for some time that research in certain countries tends to be uniformly positive about acupuncture, to the extent that it raises serious questions about publication bias. This line of thought is not—as you try to portray it—a form of racism among Wikipedia contributors. The relevance of national origin of research has been expounded both qualitatively and quantitatively in the reputable scholarly literature, going as far back as Vickers et al., 1998, who found that trials from China, Japan, Hong Kong, and Taiwan were uniformly positive about acupuncture and urged caution in integrating these almost-certainly-biased results into systematic reviews or meta-analyses. More recently, a systematic review conducted by Chinese authors in 2014 found striking evidence of publication bias in studies on acupuncture reported in Chinese journals. I think it's worth having a serious discussion about this, but your post is pretty much the opposite of serious discussion. MastCell ^Talk 17:56, 21 September 2015 (UTC)

MastCell, I'm all for having a serious discussion on this and hope you are too. For the record, I never claimed racism was a factor here, and my rather long-winded complex posts, if you read them, should show that I haven't oversimplified anything. In fact, we have written into MEDRS not to deny sources based on how they're funded, and yet we have extensive documentation of publication bias regarding industry funded research. No such bias has ever been proven with Chinese research, merely speculated as one of many possibilities for difference in findings. Why would we treat Chinese research different than industry funded research? A subject which, by the way, has much more extensive documentation of bias! That said, when editors reject Cochrane Reviews because they have Chinese authors, or reject obviously high quality Western-published journals because they have Chinese authors, maybe they're just belligerent POV pushers rather than racists. But they put their POV above all else and that's a problem. LesVegas (talk) 21:25, 21 September 2015 (UTC)

But this has never happened, your diffs don't support it. CFCF 💌 📧 00:09, 22 September 2015 (UTC)

Ummm, there is a diff at the top of this very thread where an editor objected to a journal article in Medicine because its authors were Chinese. Did you not see that? LesVegas (talk) 00:22, 22 September 2015 (UTC)

The diff doesn't say that, it speaks of 'a well documented bias' and expands later. Mis-quoting people doesn't strengthen your case. In other areas of WP should we not be allowed to say that govt X, TV station Y or news outlet Z, has 'a well documented bias' and should therefore be treated with extreme caution ? Pincrete (talk) 07:56, 22 September 2015 (UTC)

Pincrete, I am sorry I forgot to show the actual source, but, anyway, here is the source that was removed. As you can see, it is a high quality journal that just so happens to have Chinese authors who did the meta-analysis. The diff in question was trying to apply the "well-documented bias" to this meta-analysis and systematic review. There is another one where the same thing happened regarding several Cochrane Reviews used to update an old claim. They just so happened to have Chinese authors and that's a problem. LesVegas (talk) 22:43, 22 September 2015 (UTC)

When you actually read the conclusions in that paper they say very little, and above all they comment on the lack of qualitative studies in the field. It states "Finally, the included RCTs were all conducted in China, so more studies are needed" and "Reporting biases could not be detected by funnelplot due to lack of adequate RCTs."
From the entire paper a single positive sentence was taken and copied verbatim into Wikipedia (amounting to copyright violation). Maybe the article has a role in the sources, but it did not properly support the statement it was used to support and the comment you linked reflects that. CFCF 💌 📧 23:12, 22 September 2015 (UTC)

CFCF, I agree with most of what you said above. Yes, it was a copyright violation. And yes, the quote wasn't fully representative of the source. And by the way, I didn't add the quote or source, that was another editor. I had nothing to do with it beyond watch it get removed. But the reasons it was removed were wrong. Removing it or amending it for reasons you stated are perfectly fine. I have no problem with that. Editors shouldn't be supporting removal of sources because the authors are Chinese. I hate to belabor the point, but you said above that it never happened and my diffs don't support it, so I just have to clear my name here. LesVegas (talk) 02:27, 23 September 2015 (UTC)

The article states it is a problem that their study has included so many chinese studies, so I find nothing wrong with the comment you've linked. CFCF 💌 📧 07:23, 23 September 2015 (UTC)

CFCF Do you have access to the full article? Would you mind posting what they say re Chinese studies? It was not in the free text they provided. LesVegas (talk) 20:08, 23 September 2015 (UTC)

LesVegas, I am not competent to assess the virtues or weaknesses of specific medical research, my opinion on that would be valueless. However, specific pieces of research are not the subject of this RfC, rather a general principle. Pincrete (talk) 08:25, 23 September 2015 (UTC)

Question for Kww

User:Kww is mentioned repeatedly above, and it doesn't look to me like anyone has pinged him about this or given him a fair chance to explain his off-the-cuff comment that most of the "authors are Chinese" on a particular source. (Does that refer to their race? Their citizenship? Where they were trained? Where they're currently working? A quick guess based upon last names? It could mean almost anything.) IMO it would be fair to let him have his say if he's interested. WhatamIdoing (talk) 04:32, 23 September 2015 (UTC)

The diff shows an edit by Guy (JzG). The history of Talk:Acupuncture says Kww hasn't posted there since July. Johnuniq (talk) 04:53, 23 September 2015 (UTC)

Yes I'd love for him to explain himself because I and other editors could never get an explanation back when I confronted him about it. But as I understand it, he doesn't edit much anymore since his desysopping, correct? LesVegas (talk) 15:00, 23 September 2015 (UTC)

Actually I just looked at his edit history and I see he occasionally and casually edits here and there still, so maybe we can finally get an explaination on the specifics. LesVegas (talk) 15:03, 23 September 2015 (UTC)

I don't respond much unless pinged. The diff you are providing is JzG's, but I'll defend it. The bias of Chinese medical journals and Chinese studies submitted to Western medical journals is well-documented. For such sources to discover beneficial effects to acupuncture isn't particularly surprising, nor is it likely to indicate that acupuncture has any particular effect beyond placebo. There are similar problems with sources related to ayurveda. While I understand the slippery slope of racism, it's also important for us to note that associated with some of the rituals that are misrepresented as medicine there are one or two countries that have a vested interest in portraying that ritual as if it were effective. As for Les Vegas's proposal above, I don't think anyone is attempting to remove sources due to a personal objection to the country of origin: it's due to well-documented and objectively provable objections to the country of origin.—Kww(talk) 16:03, 23 September 2015 (UTC)

Thanks for the reply, Kevin; I appreciate it. Thanks also to John for pointing out that I had the wrong diff. I apologize for the sloppy post; I meant this diff. WhatamIdoing (talk) 16:12, 23 September 2015 (UTC)

As for the corrected quote, note that my primary objection was that even taken at face value, the sources didn't support the proposition Les Vegas was making: "not effective for a wide range of conditions" isn't contradicted (or even particularly weakened) by evidence of positive effect for a narrow range of conditions. I didn't get into a detailed analysis of the sources themselves. I will say, however, that it's a fair bet that Z Zheng, CCL Xue, J Shang, X Shen, J Xia, X Zhu, L He, and J Song being among the reported researchers in a small handful of reviews represents a reasonable foundation to suspect bias. If there was actually a meaningful claim being made, I'd scrutinize the sources more carefully.—Kww(talk) 16:29, 23 September 2015 (UTC)

Thank you for the explanation. Kww's answer goes right to the heart of the matter. If authors have Chinese names, we should suspect bias even if it's in a Cochrane Review, as if Cochrane is incapable of vetting the material themselves, but we, the lowly Wikipedia editor are. Frankly, I'm really not surprised. LesVegas (talk) 20:04, 23 September 2015 (UTC)

Nor should you be. What's surprising is that you would think that we should close our eyes to an indication of bias noted in reliable sources. Note that I did not reject the sources outright, but was arguing primarily on the basis that your conclusions didn't follow from their statements even if we presumed they were completely accurate.—Kww(talk) 20:54, 23 September 2015 (UTC)

Yes, I recall very well. Your primary argument was that the Cochrane Reviews didn't support replacing a 6 year old claim with an update. Your secondary argument was that they might be unreliable anyway because they have Chinese authors. That's what is being disputed here. Please note that the wording of the MEDRS amendment in this RfC says "do not reject high quality sources because of x,y,z...or country of origin." It does not say do not reject high quality sources if they are being used to support a claim that isn't supported by those sources, or is more accurately supported by another source. Policies elsewhere already cover those objections. LesVegas (talk) 21:41, 23 September 2015 (UTC)

If you had found material that actually supported an update, I would listen. Once again, "not effective for a wide range of conditions" and "is effective for condition x, y, and z" are not contradictory. They are barely related. The original source surveyed the impact on over thirty conditions and you found a source that weakened the position on, at most, two of those conditions. As for your propensity for misinterpreting sources and using them to support claims they don't support, I hope this debate makes that problem more obvious to a wider variety of editors. Your motivation for requesting this change has been noted by numerous editors: you are upset that people want to see corroboration in other sources when you can only offer Chinese sources supporting a claim about acupuncture. Given the noted bias problems, that's a quite reasonable demand. If something is actually true, Western sources will eventually catch up.—Kww(talk) 22:08, 23 September 2015 (UTC)

You're back to omnisciently assuming my motivations again. Just for the record, my motives are simply to encourage editors to examine source quality and not reject them for paltry reasons like the last name of their authors. LesVegas (talk) 22:21, 23 September 2015 (UTC)

Examination of source quality does require taking bias considerations into account. Your requested modification is basically a demand to ignore one potential source of bias. As for omniscience? No, pretty much any observer of your edits and discussions will come to a similar conclusion.—Kww(talk) 22:38, 23 September 2015 (UTC)

"Examination of source quality does require taking bias considerations into account" is just another way of saying high quality sources that we never question anywhere else for any other claims might not be high quality at all if we see it has Chinese authors, whom we ought to be suspicious of because they might be biased. Thank you for making your position clear, over and over again. LesVegas (talk) 23:03, 23 September 2015 (UTC)

I support being suspicious of any biased source (including Coke-paid researchers on the effects of soda pop on obesity [on which I didn't comment, apparently], ayurvedists on the benefits of heavy metal poisoning, Christians on the historical existence of Jesus of Nazareth, among many other examples a quick search of my edit history will uncover).—Kww(talk) 23:58, 23 September 2015 (UTC)

Ok, I'm glad to see you are consistent. I'm glad you brought up the Coke funding of obesity studies which is an example I have brought up here as well. MEDRS currently states that editors shouldn't reject high quality sources based on their source of funding. So that would mean Coke studies are perfectly ok to use, as are countless numbers of other industry funded research which has been shown to be far worse than Chinese studies on acupuncture. Why should we have one guideline where we don't reject industry funded studies, but we do reject sources because they originate in China or have Chinese authors? To me, I wouldn't even have raised this issue if I didn't see an inconsistency in our guideline where we allow carte blanche acceptance of industry funded, obviously tainted and conflicted research, but we reject Chinese sources because they might be tainted. I have never accused anyone here of being racist. I honestly believe editors here are good people, some just have blindspots and I know I do too. But frankly, if we continue to greenlight biased industry funded research but disallow Chinese sources, that is racist. If editors here want to consider Chinese sources suspect, while I deeply disagree with that, I can understand the point. But we had damn well amend MEDRS to treat industry funded sources the same way. We're not going to have two standards here. LesVegas (talk) 04:00, 24 September 2015 (UTC)

You shouldn't attempt to fix the inconsistency in the wrong direction: two wrongs don't make a right and all that.—Kww(talk) 04:15, 24 September 2015 (UTC)

The question of industry funding was discussed at length very recently on this very page. There, WP:DNFT was invoked and I think it now applies here too. Time to close. Alexbrn (talk) 04:24, 24 September 2015 (UTC)

You can ignore a problem or inconsistency, you can IDHT, but editors shouldn't be insulted for pointing out problems. And previously, that was invoked by an editor who much of the Wikipedia community has complained about for GMO advocacy and personal attacks and I have no reason to doubt that his invoking DNFT wasn't a great illustration of both his advocacy and insults. I can certainly see why a GMO advocate doesn't want to limit industry funded sources in any way. This same editor also believed Chinese sources should be limited rather inconsistently. But I expected much better from you, Alexbrn, than to ignore and insult and hope this embarrassing inconsistency just goes away. And as much as I deeply disagree with Kww on this issue, I at least have to respect and admire his consistency here. And we should be consistent, one way or another. While I think the best way with our policies is to allow high quality sources regardless of industry funding or country of origin (because it's too hard to say that because some have been bad, all are bad), we cannot have one set of guidelines for some sources and another set of guidelines for others. I know it's uncomfortable for some editors to admit this partisan application, but If we bury our heads in the sand and pretend it's not inconsistent, that won't magically make it consistent. LesVegas (talk)

You've illustrated the problem quite nicely: there's no particular correlation between the conflict of interest inherent in the Coca-Cola funded obesity research and the general agreement of all responsible researchers that GMO as currently practiced presents no health or public safety issues. While I abhor the inconsistency, I can understand why people would see a slippery-slope problem on the funding issues. After all, most good research is funded by people connnected with science, and science pretty much rules out the anti-vax, anti-GMO, pro-ayurveda, and pro-acupuncture camps, giving those people an incentive to object to nearly all research as being biased in one way or the other.—Kww(talk) 00:05, 25 September 2015 (UTC)

Well, I don't claim to be an expert on the latest in all things biotech, but I know Monsanto funds much biotech safety research and even when" purely independent research" is touted in the press it's often funded by The American Society for Nutrition whose own website says is funded by Monsanto. They fund the science all over the place. And this idea that industry-funded research was only a problem with Coke today and big tobacco in yesteryears is not accurate at all. What I do know is there is well documented issues with pharmaceutical industry funded research in the way of psychiatric medications and in failing to warn about cancer risk with certain drugs, as well as in showing a huge statistical increase in "positive findings" for various pharmaceuticals just to name a few. They were all connected with "science" yet clearly have shown worse issues than the Chinese because they're not explainable by a slew of confounding factors from an entirely different culture and a medicinal art many researchers have openly admitted is very difficult to study anyway. I see no argument for why we should treat Chinese studies differently than those paid for by Monsanto, Merck or the Marlboro Man. LesVegas (talk) 19:29, 25 September 2015 (UTC)

It's a balancing act: whether an individual drug does or does not create a cancer risk is something with gradations of bias that can range from damning (a study by the manufacturer that contradicts all other studies would need to be treated with great suspicion, for example) to minor (a study by a research group funded by multiple sources, including the manufacturer, which falls in line with similar studies by groups not involving the manufacturer). In the case of Chinese studies of acupuncture, we're dealing with a group that tends not to publish negative results, refuses to use standard placebo-based methodologies, and comes to conclusions that researchers using more rigorous research techniques cannot reproduce. There's no reason to suspect that the results are accurate, and every reason to suspect that the results are due to flawed research and reporting. There's nobody arguing to exclude all Chinese research, just to avoid relying on a known trouble spot: Chinese studies of treatments based on Chinese folk remedies and superstitions. Is there anyone objecting to Chinese studies of pharmaceutical treatment of tuberculosis or anything like that? I think not.—Kww(talk) 23:01, 25 September 2015 (UTC)

Actually, negative results in China weren't seen in 1998, they are now. But sure, in more recent times it's noted with different results seen in US studies. Some of that could be publication bias, some of that could be due to other factors. Fact is, we don't know. But industry funded pharmaceutical results are also seen to have much better results than independent research shows and those are comparing results within the same country. We have one variable to deal with there, not countless, so it makes publication bias much more likely a factor than is the case with the Chinese. I'm talking hard facts and statistics here, not your opinions of Chinese culture or your opinions of alternative medicine, but documented facts alone. And by the documented facts alone, even in the worst case-scenario with the lowest quality published research in China, Chinese studies on acupuncture have actually less possibility of publication bias than industry funded pharmaceutical studies do from the standpoint of variables. Any reasonable person examining facts alone and not swayed by opinions or POV must agree with this. And yet we specifically allow industry funded pharmaceutical studies on Wikipedia, and some of these same editors want to reject Chinese studies on acupuncture. LesVegas (talk) 16:59, 26 September 2015 (UTC)

Do we all understand the outcome here?

So we have a bit of a procedural issue with RFCs, which is that one editor lists almost all the expired RFCs at WP:ANRFC, so that some busy WP:NAC can carefully add a colored box and a closing statement to each of them. I think this is an ongoing WP:NOTBURO problem, a waste of time for most RFCs, and sometimes even an insult to the participants' good sense and ability to understand their own conversations, but let me ask just to be sure:

Does anybody here need any outside help in figuring out whether or not this discussion shows support for the proposed addition of the words "country of origin" to that sentence in this guideline at this time?

If nobody actually needs any help with understanding the outcome, then perhaps we can prevent him from wasting someone's time with telling us what we all already know. WhatamIdoing (talk) 17:49, 6 October 2015 (UTC)

The discussion above is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

Questions about RFC closure - Country of origin

We do reject sources because they are from a specific country because there are sources that are poor sources and there are bias sources. According to the close the country of origin is a legit consideration where RS have identified it as an issue. There have been hard data (as contrasted to stereotypes) that have identified a systematic problem that is normally identified with an affiliated country of origin. Therefore, this edit seems to contradict the close. QuackGuru (talk) 17:00, 20 October 2015 (UTC)

The close that I wrote said no such thing. Besides, the suggestion that the edit that the close clearly and directly indicates is appropriate nonetheless contradicts the close by some extreme, odd interpretation is utterly nonsensical. Get the point. Discussion closed. --Elvey^(t•c) 03:34, 24 October 2015 (UTC)

User:Elvey, perhaps you would come back and clarify a few things for us. For example, you wrote "We cannot override WP:V or WP:RS." In what way would rejecting some sources, on some subjects, from countries with a strong reputation for the low quality their sources, constitute "overriding" WP:V or WP:RS? WhatamIdoing (talk) 21:35, 20 October 2015 (UTC)

Reread Albino's comment. To what end do you seek clarification? --Elvey^(t•c) 03:34, 24 October 2015 (UTC)

I am flummoxed as to why that single comment is singled out as of more weight than those of multiple other editors. This seems to me a very questionable closure. CFCF 💌 📧 12:02, 24 October 2015 (UTC)

I think that would likely be because Albino's comment stated that multiple editors were ignoring the fact that we are talking about high quality sources and some editors opposed low quality sources, which wasn't the purpose of the RfC. LesVegas (talk) 12:53, 24 October 2015 (UTC)

This is not true. The statement in the guideline is about high-quality types (emphasis in the original) of sources, not about high-quality sources. A meta-analysis is a high-quality type of source, but it can be a low-quality source (e.g., if it's outdated, poorly done, or irrelevant). WhatamIdoing (talk) 21:32, 28 October 2015 (UTC)

Yes, there are meta-analyses that are poor quality because of age, journal integrity, etc, but they are not poor quality simply on the basis of where they are published or the country of origin of its authors. That was the purpose of the RfC, and no, just as we cannot exclude sources because they receive industry funding, we also cannot exclude them because of country of publication. Not one single editor addressed why we don't reject industry funded sources on the basis of known bias, but we should ban sources based on speculative and unconfirmed publication bias due to country of origin. Not one editor. Note that Elvey mentioned as much in his close as well. There were multiple factors here really. LesVegas (talk) 19:23, 29 October 2015 (UTC)

It still completely fails to address the actual need for the addition. No discussion has been shown about not including high quality sources based on country of origin, this is a red herring and a useless bloating addition. CFCF 💌 📧 10:02, 26 October 2015 (UTC)

This discussion centres around attempts by SPA editors to crowbar low quality sources supporting various degrees of efficacy not shown in high quality sources of ALT-Med articles, particularly Acupuncture. Other attempts have been made, to WP:MEDRS for example. Characterising mainstream editors as racially prejudiced by SPA's has been happening for a while now, and is a particularly nasty tactic. -Roxy the dog™ (Resonate) 10:20, 26 October 2015 (UTC)

While I am surprised my comment was mentioned. I am not surprised with the close. My comment did point the problem with some of the responses. They were off topic. A problem that looks like its repeating down here. The RFC question specifically was about High quality sources. Low quality sources will be rejected regardless of what country they are from because they will not even pass WP:RS. AlbinoFerret 13:54, 26 October 2015 (UTC)

Are there examples of editors rejecting high quality sources because of racial prejudice? -Roxy the dog™ (Resonate) 14:30, 26 October 2015 (UTC)

A couple were mentioned during the RFC. I did point out I was unsure if it was widespread, but even if it isnt very widespread its a bad thing that should be stopped. It hurts the project in rejecting even a few high quality sources and makes the project look bad focusing in the ethnic angle. I think those editors who are concerned this will allow low quality sources in shouldt worry, low quality will always be excluded. Just point out the problems with the source that make it low quality and dont point out where they came from. Doing that give those pushing them a reason to argue. AlbinoFerret 14:46, 26 October 2015 (UTC)

I think this might work: "While country of origin per se is not a suitable reason to reject a source, it is appropriate to consider in cases where reliable sources have identified systematic problems in the medical literature associated with specific regions or countries."

QuackGuru (talk) 20:31, 26 October 2015 (UTC)

While that may be what you want in the section, it does not say what the closer said. Here is the section:

"This addition should NOT be read as a PC ban on any mention of country of origin (or founding source, etc.) when necessary to refer to studies with hard data (as contrasted to stereotypes) that have identified a systematic problem that is normally identified with an affiliated country of origin, as mentioned by Richard Keatinge. Likewise, this addition should NOT be read as a changing the longstanding policy that sources from publications known to routinely publish and fail to retract material proven unreliable may be excluded."

What it appears Elvey is saying , and Elvey can correct me if I am wrong, is this close is not a ban on discussing problems with sources in a discussion of a source that talks about them. It doesnt appear to be a loophole to insert arguments about a source, based on a country, just because other crappy sources have come from there. What might be better to say is "It is better to look at the quality of a source, if the source is of low quality it should be excluded." AlbinoFerret 20:58, 26 October 2015 (UTC)

CORRECT on both counts. (The comma in the last sentence should be a semicolon or period.) --Elvey^(t•c) 02:27, 29 October 2015 (UTC)

Your proposal "It is better to look at the quality of a source, if the source is of low quality it should be excluded." is not about country of origin. QuackGuru (talk) 21:01, 26 October 2015 (UTC)

See the specific part of the close. See "when necessary to refer to studies with hard data (as contrasted to stereotypes) that have identified a systematic problem that is normally identified with an affiliated country of origin, as mentioned by Richard Keatinge." QuackGuru (talk) 21:02, 26 October 2015 (UTC)

Thats because the RFC has already said that country of origin is not a valid exclusion. I assumed that country of origin is added to the list of other things that should not be considered. The whole purpose of that section appears to be to tell people to look for high quality sources, then some things that should not be considered. I propose adding a sentence at the bottom to direct editors to, instead of looking at the country or funding, to look at the quality. AlbinoFerret 21:07, 26 October 2015 (UTC)

Yes, it says not to stop discussions about a topic that discusses low quality sources, not a loophole to allow discussions that we have already said should not take place like excluding a source based on ethnic origin. AlbinoFerret 21:09, 26 October 2015 (UTC)

You said "Thats because the RFC has already said that country of origin is not a valid exclusion." That what was written and that was what was added.

"While country of origin per se is not a suitable reason to reject a source, it is appropriate to consider in cases where reliable sources have identified systematic problems in the medical literature associated with specific regions or countries."[4] QuackGuru (talk) 21:11, 26 October 2015 (UTC)

My close is clear. It's not appropriate to --Elvey^(t•c) 02:27, 29 October 2015 (UTC)

You are adding a loophole that I dont believe the closer added. Lets wait for them to chime in as I have pinged the closer in a few posts ago. AlbinoFerret 21:14, 26 October 2015 (UTC)

Albino Ferret is right, in fact, it would violate the spirit of every single RfC done across wikipedia if we always added a reference tag and then just put whatever summary those opposed wanted. Adding in a summary of whatever you want to be read as a caveat is gaming an RfC outcome. Besides, the RfC wasn't about "should we say 'country of origin' and then have these caveats?" it was about the wording, "country of origin" specifically. And, to that, the answer was "yes," it needs to be added in. But since the consensus reading did mention other specifics and since some editors are persistent in wanting something else, I figured a link where readers could see the full consensus read/closing comments would be the best compromise. LesVegas (talk) 21:25, 26 October 2015 (UTC)

The outcome of the RfC is not constrained to be binary. It's intended to gauge consensus, and the closer found consensus to include 'country of origin', but with the caveat that it might be a legitimate consideration where "hard data" demonstrate a concern about biased literature. MastCell ^Talk 22:52, 26 October 2015 (UTC)

The point is that the addition is completely useless, and not supported by the RfC. QuackGuru – while I agree with the intent of your clarifications the fact is we are just introducing bloat. If any policy is to be taken seriously it needs to be succinct, and can't include hypothetical clauses that have never been proven to be needed. CFCF 💌 📧 22:48, 26 October 2015 (UTC) 

When the RfC is a question asking "should we add 'country of origin' to this list?", and the consensus reading says "Yes" removing it entirely is going against consensus. LesVegas (talk) 23:09, 26 October 2015 (UTC)

We're trying to get some clarification of some of the major issues with the close and a possible new clause, it isn't something we will do on a whim–we need to have a agree on what (if anything) to include. We should wait until clarification of what Elvey meant in the close summary. Currently the meaning of the text is very murky, and as such it may be challenged for not adequately summarizing the consensus. CFCF 💌 📧 23:15, 26 October 2015 (UTC)

(Late reply) Elvey, I'm trying to figure out what the relevance of your statement about "overriding WP:V and WP:RS" is. Imagine that you are trying to decide whether a source is reliable. Do you believe that considering the country of origin, e.g., to avoid citing the notoriously bad Soviet science, would somehow a constitute "overriding" WP:V? Could you point to any sentence in WP:V that would be violated or "overridden" by doing that?

Also, I have read Albino's comment, and you seem to have overlooked a critical difference between what the guideline says and what Albino wrote. The sentence in the guideline talks about high-quality types of sources (e.g., a meta-analysis is a high-quality type of source; a case study is a low-quality type of source). Albino talks about high-quality sources—a quality that takes far more into consideration than the type of the source. "High-quality types" and "high-quality sources" are not the same thing. It's possible to have a low-quality meta-analysis, and just like it's possible to have a top-quality case study. WhatamIdoing (talk) 21:32, 28 October 2015 (UTC)

I don't dispute that it's possible to have a low-quality meta-analysis. What's clear is there was consensus that "country of origin", per se, is not a valid method to identify a low-quality meta-analysis. --Elvey^(t•c) 02:27, 29 October 2015 (UTC)

What does "overriding WP:V and WP:RS" is? It seems to mean that MEDRS is wrong and that MEDRS overrides WP:V and WP:RS to exclude sources that meet WP:V and WP:RS. Therefore, country of origin such as from China are good even if they are poor quality and bias. QuackGuru (talk) 21:51, 28 October 2015 (UTC)

You didn't respond to my comment about your extreme, odd interpretation of what I said. I feel grossly misrepresented, and an apology would certainly have been welcome. "We cannot override WP:V or WP:RS" means just that. Surely none of you dispute that "We cannot override WP:V or WP:RS." There was consensus that "country of origin", per se, is not a valid reason to reject a source. I claim neither WP:V nor WP:RS contain any sentence consistent with allowing "country of origin", per se, as a valid reason to reject a source.

Again: You need to drop the stick. Insisting my close not clear by misrepresenting what I said with an extreme, odd interpretation is not going to fly, and you've just done that for the second time in this section, this time by claiming I've said MEDRS is wrong. STOP. It's disruptive and uncivil. I said no such thing. Not liking the close is not a valid reason to reopen it. I don't see any clarification request that hasn't been adequately addressed by AlbinoFerret or myself. Re-closing. --Elvey^(t•c) 02:27, 29 October 2015 (UTC)

No, you haven't answered my question. There are two basic ways to approach this. Either your comment about overriding WP:V and WP:RS is:

• pointless blather, with just as much relevance as you saying "Don't kick puppies" or "Be nice to your neighbors" in the middle of this (in which case, you should just remove it), or
• you actually meant to communicate something relevant, that editors need to know and understand (in which case, you need to explain what you meant, because everybody's confused).

The obvious assumption is that you meant what you said. What you said seems to be that (when relevant/appropriate/etc.) editors should not take notice of published academic research that says (for example) Soviet psychiatric research is a bunch of garbage (and garbage because it came out of a country that had difficulties with the concept of apolitical science), because you believe that doing so would be based on "country of origin" and that discarding sources on the grounds of country of origin somehow override WP:V and WP:RS.

Is that what you meant? WhatamIdoing (talk) 03:24, 29 October 2015 (UTC)

I'm not sure what Elvey meant, but I know some editors here have already exhausted this editor (a volunteer, as we all are) with complaints, so I'm willing to take a stab at it from my reading of Elvey's close. Documented issues with poor science conducted by a particular piece of research are a valid reason to exclude that individual piece of research because it doesn't meet the barometer of a high-quality source, per WP:V and WP:RS anyway. Excluding such research, however, because it is Russian or Soviet and they had scientific issues and therefore it all must be garbage science, is not an appropriate reason for rejecting it. In other words, what are the reasons that individual piece of research is invalid? And if it is documented invalid, it isn't high quality anyway. Funding sources have a well documented history of much worse; Elvey mentioned in their close that country of origin wasn't worse than funding, per consensus (since nobody seemed to tackle that question I raised.) LesVegas (talk) 19:38, 29 October 2015 (UTC)

This was the previous proposal: "While country of origin per se is not a suitable reason to reject a source, it is appropriate to consider in cases where reliable sources have identified systematic problems in the medical literature associated with specific regions or countries."

It is known that there is publication bias from Chinese journals. We cannot claim "country of origin" is not a problem given the evidence. QuackGuru (talk) 20:24, 29 October 2015 (UTC)

That's a 2005 source, and it does illustrate publication bias in some fields. For those fields it notes, it may apply to Chinese sources published prior to 2005. LesVegas (talk) 23:00, 29 October 2015 (UTC)

LesVegas, your reply does not address my question at all. So far as I can tell, there is nothing in WP:V or WP:RS that requires us to accept sources that have characteristics which editors deem suspicious. However, it leads me to a question for you: Imagine that (it's 1985 and...) you are looking at a source produced by the well-documented mess that was Soviet psychiatry on the question of people with schizophrenia who claim to be political prisoners. Do you think that specific source could be excluded:

(a) because it was produced by the Soviet psychiatric system, which is an extraordinarily well-documented disaster (and therefore the odds are very high that it, too, is bad) or

(b) only if that individual source were called out, by name, in another reliable source, as an example of bad research?

Another way to put this: If a source was produced by a Soviet psychiatric institution, is it "tarred with the same brush" as the rest of the field, or "innocent until proven individually guilty" of politically manipulated science? WhatamIdoing (talk) 00:39, 30 October 2015 (UTC)

User:WhatamIdoing, you're right in that WP:V and WP:RS don't require us to use sources from countries with issues, but they also don't require us to exclude them on that basis. These policies simply tell us what a high quality source is and what it is not. The way I read the close is based on the content of the RfC expressed over and over again, which is we have our rubrics for determining what a reliable source is. That's the WP:V and RS part. So we use WP:V and WP:RS to determine if it is a high quality source. These rubrics do not allow us to exclude sources that meet WP:RS standards because of their country of origin, much as they don't allow us to reject sources because of industry funding issues. And the way I see it, the issues with the Soviet disaster likely produced journals that wouldn't meet MEDRS standards anyway, certainly being called out by reliable sources. If this occurred today, you would undoubtedly find a slew of journals being slammed by academia for problems. Having occurred so long ago, I'm sure there's some, but the point is moot because those journals are so old they wouldn't be reliable sources anyway per WP:MEDDATE. LesVegas (talk) 19:06, 30 October 2015 (UTC)

I think your going to have to clarify your thought process here - I can't tell how your different statements are related. And neither RS nor MEDRS are about finding high quality sources, they're about finding acceptable quality sources for Wikipedia, with MEDRS focusing on medicine. CFCF 💌 📧 19:23, 30 October 2015 (UTC)

MEDRS is WP:RS but specific to medical claims. We don't use low quality sources. What other statements do you not understand? LesVegas (talk) 19:38, 30 October 2015 (UTC)

Elvey the clarification that I asked you about is from an earlier discussion, that some want to add a loophole to the "country of origin" by stating it still can be considered with language like this as a note "it is appropriate to consider in cases where reliable sources have identified systematic problems in the medical literature associated with specific regions or countries" saying that the last paragraph of your close says that. Is this what you wrote in that last paragraph of the close? AlbinoFerret 04:46, 29 October 2015 (UTC)

Editors are waiting for clarification. Editors do not agree with this change. See "The outcome of the RfC is not constrained to be binary. It's intended to gauge consensus, and the closer found consensus to include 'country of origin', but with the caveat that it might be a legitimate consideration where "hard data" demonstrate a concern about biased literature."[5] QuackGuru (talk) 19:32, 30 October 2015 (UTC)

Reread Albino's comment. To what end do you seek clarification, and of what? Clarification answers, AGAIN: "We cannot override WP:V or WP:RS" means just that. Surely none of you dispute that "We cannot override WP:V or WP:RS." There was consensus that "country of origin", per se, is not a valid reason to reject a source. I claim neither WP:V nor WP:RS contain any sentence consistent with allowing "country of origin", per se, as a valid reason to reject a source. No, I did not say that. There is no caveat in the close; just a clarification. --Elvey^(t•c) 02:37, 31 October 2015 (UTC)

Phrasing

We're having significant issues implementing the result of the RfC, in part because it was ill-formed, and in part because it had a very unclear close. I think the situation demands more discussion in order to sort out the proper phrasing. Before staring a new RfC or anything similar I think we should tally our suggestions:

This is the last suggestion:

Editors should not perform detailed academic peer review. Do not reject a high-quality study-type because of personal objections to: inclusion criteria, references, funding sources, country of origin or conclusions.

References

I think the major issue with this is bloat – we're basically wasting time explaining the obvious here. Is it needed at all, and why not forgo it entirely? Since country of origin pew say has never been proposed to be a reason to exclude sources, why should we waste valuable space when it's going to need such a lengthy explanation that the clause really it doesn't mean anything at all? CFCF 💌 📧 01:38, 31 October 2015 (UTC)

The wording is ambiguous. It does not improve or clarify anything. QuackGuru (talk) 01:42, 31 October 2015 (UTC)

All this talk about country of origin is misguided because it does not matter what this guideline says—if there is a reason to consider country of origin when assessing the reliabity of a particular source for verification of particular text, the country will be considered. A guideline cannot prohibit common sense. Naturally anyone saying "that source has to be rejected because it came from China" can be ignored, but there may well be reason to be cautious about some types of research which a guideline cannot rule out. Johnuniq (talk) 02:36, 31 October 2015 (UTC)

This is insane. I asked what clarification question was given no answer, but got no reply. Archiving. --Elvey^(t•c) 02:37, 31 October 2015 (UTC)