Wikipedia:Bots/Requests for approval/Dapperbot

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at Wikipedia:Bots/Noticeboard. The result of the discussion was

Approved.

Dapperbot

Operator: WikiMacaroons (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 08:27, Thursday, August 13, 2020 (UTC)

Automatic, Supervised, or Manual: automatic

Programming language(s): Python

Source code available: Still in development (Will change this once I've done it)

Function overview: Automatically uses the Covid-19 API to update Coronavirus disease rates.

Links to relevant discussions (where appropriate):

Edit period(s): Perhaps twice daily?

Estimated number of pages affected: 2,760

Exclusion compliant (Yes/No): Yes

Already has a bot flag (Yes/No): No

Function details: Every day, perhaps twice a day, the bot sweeps all Covid-19 articles, updating the rates. It would perhaps use a dedicated Covid 19 REST API to scrape the data and automatically edit each article. There would perhaps be some templates that would be laid out on each page so the bot could edit them.

Discussion

Would you make a few manual edits with your own account that illustrate the edit you would make with your bot account and link those diffs here? — xaosflux ^Talk 11:26, 13 August 2020 (UTC) [reply]

Extended content

@Xaosflux: For example, this diff shows the updating of the cases count in the UK. On any pages related to Covid-19 in the UK, that template will be there and show recent data. Wiki Macaroons^Cinnamon? 11:59, 13 August 2020 (UTC)[reply]

@WikiMacaroons: I'm looking for an actual live update to something in an article - not an isolated sandbox - can you do one of those? — xaosflux ^Talk 14:45, 13 August 2020 (UTC)[reply]

@Xaosflux: Ah I see, sorry. So the rates of Covid-19 on each page listing them will instead be a template. For example, under Russia, instead of saying "15,000" etc, it will have "{{Russia Covid-19 Cases}}". The bot will do a sweep every few hours of the Covid cases and update each template, thus updating them on the page. Unlike in this diff, where I give an example of the process, it will not be a userspace template, I just used that for demonstration purposes. Wiki Macaroons^Cinnamon? 19:04, 13 August 2020 (UTC)[reply]

@WikiMacaroons: so every time you update a fact, I'm assuming you would update the reliable reference as well? Having a template to just store one number, then using that in an article doesn't sound like a good idea - as it makes the article harder for others to edit. If this was maybe only being used in an info box, using wikidata and having the bot update wikidata might be better. — xaosflux ^Talk 19:44, 13 August 2020 (UTC)[reply]

@Xaosflux: So the bot could potentially edit the structured data on Wikidata, or simply update the tables without the templates, as in this diff. Which would be your preference? Wiki Macaroons^Cinnamon? 20:08, 13 August 2020 (UTC)[reply]

@WikiMacaroons: so in that edit you made a change to a value, but that value was supported by a reference, {{cite web |title=Оперативные данные. По состоянию на 23 апреля 10:35 |url=https://xn--80aesfpebagmfblc0a.xn--p1ai/ |website=Стопкоронавирус.рф |accessdate=11 August 2020 |language=ru}}. Are you now supporting your edit with a new reliable source, did that source change, etc? Think of this the same as if you just made that edit - where is your source, how are you showing readers the current source? — xaosflux ^Talk 01:09, 14 August 2020 (UTC)[reply]

@Xaosflux: That's a good point. I notice that particular source is actually a live dynamic one, meaning that the bot could constantly webscrape from that. However, I'm not sure if this applies to all of the references in the table. Wiki Macaroons^Cinnamon? 07:49, 14 August 2020 (UTC)[reply]

@WikiMacaroons: so basically if you are going to make statistics edits, it is expected that you will also include current reference information -- your plan to just use a template with a bare number in it doesn't work for this - not to mention that using such templates in articles is generally a bad idea. Outstanding a larger discussion where you have gain a large consensus to make these edits in this manner (please provide a link if this happened already) I'm going to decline your bot request. If things change, you can always open a new request. Your idea may be useful for wikidata, but we don't control bots on wikidata here on Wikipedia (see wikidata:Wikidata:Requests for permissions/Bot for their process). Using a bot to maintain a statistical value there could be useful, though I expect they would also be looking to have you update the reference link, then it comes down to our editors if we want to load wikidata information to encyclopedia articles or not. Will hold this open for at least a day in case there is something I'm overlooking, but like I said if things change you can always just open a new BRFA. — xaosflux ^Talk 15:06, 14 August 2020 (UTC)[reply]

@Xaosflux: You didn't comment on my previous message: I had a thought that if all the references were dynamic, like the Russian one, then it could webscrape data from those and the refs would consistently be up-to-date. Wiki Macaroons^Cinnamon? 15:15, 14 August 2020 (UTC)[reply]

I'm trying to think of a clearer way to say what I mean... Wiki Macaroons^Cinnamon? 15:19, 14 August 2020 (UTC)[reply]

Ok, here's a better explanation. When the bot enters the page, it will go down the list of countries. Assume here that, like the Russian source, each ref is a dynamic site, that makes live upadtes on its Covid values. The bot will webscrape the reference, and put the new value on the table. Is that a little more clear? Wiki Macaroons^Cinnamon? 15:41, 14 August 2020 (UTC)[reply]

@WikiMacaroons: so in your example edit here - is the source of your data still the same ref that is shown (Стопкоронавирус.рф) ? Or are you using a new source? And certainly the "accessdate=11 August 2020" value is no longer valid, since you have a new access date - but you didn't update the reference. Anytime you change a statistic that is supported by a reference, the reference listed should be what is currently supporting the statistic. Also in that article, the entire table is labeled as "Summary table of confirmed cases in Europe (as of 11 August 2020)" - but you didn't advance the date there either. — xaosflux ^Talk 16:16, 14 August 2020 (UTC)[reply]

@Xaosflux: So, in theory, the bot would update all of that. It would update the access date of the reference, and the "Summary table of confirmed cases in Europe (as of DATE)". The data that I put in that example edit was just an arbitrary placeholder, the bot will webscrape from the actual reference, meaning that if the reference is dynamic, it doesn't need to be changed. Wiki Macaroons^Cinnamon? 16:22, 14 August 2020 (UTC)[reply]

Also, in answer to "is the source of your data still the same ref that is shown", yes, as the bot will gather the dynamic data from that site. Is that unclear? I want to make sure I'm explaining myself correctly. Wiki Macaroons^Cinnamon? 16:24, 14 August 2020 (UTC)[reply]

@WikiMacaroons: OK, lets try this again: manually, under your own account - make one edit that you think is good, that would be the same as what you are proposing your bot would do for you. Use an edit summary that is appropriate to the edit (don't mention the bot in your own edit's edit summary at this time - make the edit becuase it is something you think actually needs to be edited), use the minor flag if appropriate. Place the single diff here. — xaosflux ^Talk 16:28, 14 August 2020 (UTC)[reply]

@Xaosflux: this diff shows the general kind of edit the bot would be making. Wiki Macaroons^Cinnamon? 16:41, 14 August 2020 (UTC)[reply]

Keep in mind that the references don't need to have their urls changed, as they show live updates. Wiki Macaroons^Cinnamon? 17:08, 14 August 2020 (UTC)[reply]

OK, @WikiMacaroons: that looks better, you listed that you want to run this on 2000+ pages, do you have the list of the pages somewhere (is it dynamic like category based?). Is the format of the data you will be changing on all of these pages the same today? What will your bot do if the format changes on a page? — xaosflux ^Talk 17:53, 14 August 2020 (UTC)[reply]

@Xaosflux: Ah, well I got that rough digit from getting the amount of articles in Category:COVID-19. I expect that not all of them contain rate counts, is there a category for this? Also, if the format changes I'll try to get the bot to find that, and notify me so I can rework the code to fit the new format... there may be a better way. Wiki Macaroons^Cinnamon? 19:11, 14 August 2020 (UTC)[reply]

@Xaosflux: For a trial, do you reckon I should just focus it on the main continent Covid-related pages? Wiki Macaroons^Cinnamon? 12:45, 17 August 2020 (UTC)[reply]

@WikiMacaroons: the trial will need to cover at least all of the different kinds of things that would get approved - it doesn't look like these are using a standard template - just wikitables - have you reviewed how many different things you will need to code for? — xaosflux ^Talk 13:32, 17 August 2020 (UTC)[reply]

@Xaosflux: I could potentially make the wikitables their own templates to edit, and leave a comment saying where to go to edit them Wiki Macaroons^Cinnamon? 15:44, 17 August 2020 (UTC)[reply]

@WikiMacaroons: those sort of design and content issues are something that you really need to work out before programming, this bot approval request is premature right now, you need to have a solid repeatable plan to have the bot component of it evaluated. The plan needs to be something that has existing support for the types of edits that will be made, or you will need to establish the consensus for the edits you will be making just as if you will be making them yourself (for example if you plan on introducing a new template to replace tables to thousands of articles). — xaosflux ^Talk 16:18, 17 August 2020 (UTC)[reply]

@Xaosflux: I'm so sorry for being vague about the plan, I've been a bit busy but I'm free now. I've partly programmed it, here's the plan:

There will be templates of wikitables for each Covid page about a continent, minus Antarctica:

The bot will sweep templates of those continents, so the pages themselves will be dynamically updated. There will be a  that tells you where to find the template to edit if you're a newcomer. It will also go into the countries within those and have templates for each graph of rates. Again, thank you, you're a saint for putting up with me. Wiki Macaroons^Cinnamon? 18:07, 17 August 2020 (UTC)[reply]

@WikiMacaroons: our goal in BRFA is not just technical, but also to make sure there won't be editor interaction problems. For your idea above, this is getting much clearer - and the scope of the bot now sounds like it will be "update a small list of templates" which sounds very manageable. I'd like to see the templates created and in use first - so either you or others working on these pages updates should do that (and make sure you don't get push back on something like "we shouldn't use a template here"). You may want to discuss this with Wikipedia:WikiProject COVID-19 first. There is no time limit on these BRFA's as long as there is momentum. — xaosflux ^Talk 18:41, 17 August 2020 (UTC)[reply]

@Xaosflux: "update a small list of templates" - I like the sound of that! I've left a message at WP COVID. Wiki Macaroons^Cinnamon? 09:25, 18 August 2020 (UTC)[reply]

Right, @Xaosflux: Here's where we're at. I've spent a while programming the webscrapers for the bot, but have found a fatal flaw that means that I'll have to put aside next few days reworking the code. Hope to have the result with you soon. Wiki Macaroons^Cinnamon? 19:59, 9 September 2020 (UTC)[reply]

Phew, @Xaosflux: That took longer than expected. The two templates that are currently available for listing covid rates are Europe and Asia.

I hooked up the value-changing script that the bot would run on to my account and it created these diffs:

Europe

Asia

Would you rather we set Europe and Asia running now while I figure out the other 4 continents, (not doing Antarctica, obviously) or program all the continents before approving the bot? Thanks, Wiki Macaroons^Cinnamon? 16:31, 2 October 2020 (UTC)[reply]

P.S: I realise that the numbers don't have 1000 separation commas. (e.g 1000000, not 1,000,000) I've just put that in the code, so should be ok next time Wiki Macaroons^Cinnamon? 19:18, 2 October 2020 (UTC)[reply]

@Xaosflux: Sorry to ping again, I don't know if the previous ping went through. Wiki Macaroons^Cinnamon? 17:03, 12 October 2020 (UTC)[reply]

{{BAGAssistanceNeeded}} can't look at this right now - but queuing it for attention. — xaosflux ^Talk 17:40, 12 October 2020 (UTC)[reply]

@WikiMacaroons: How far along are you in figuring out the other continents? I think the others could be easily added as an amendment once ironed out...there are benefits to both ways. Would you object, Xaosflux? --TheSandDoctor ^Talk 18:45, 27 October 2020 (UTC)[reply]

TheSandDoctor, I'm about halfway through North America, but Europe and Asia are fully finished now. Thanks for replying. Wiki Macaroons^Cinnamon? 19:21, 27 October 2020 (UTC)[reply]

@TheSandDoctor: Would you rather that I finish North America before we move forward? Wiki Macaroons^Cinnamon? 18:25, 3 November 2020 (UTC)[reply]

@WikiMacaroons: How long do you figure the rest of NA will take? --TheSandDoctor ^Talk 06:29, 4 November 2020 (UTC)[reply]

TheSandDoctor, I've been a little busy and haven't been working on it, but now that I'm back on top of it, I reckon it can be done in the next week. Wiki Macaroons^Cinnamon? 19:12, 4 November 2020 (UTC)[reply]

@TheSandDoctor: Phew, sorry that took longer than expected. Here are the diffs for the latest runs: 1 2 3. Thanks for your patience. Wiki Macaroons^Cinnamon? 17:00, 22 November 2020 (UTC)[reply]

@TheSandDoctor: Sorry to be harassing you, I’m just checking that the previous ping went through. Wiki Macaroons^Cinnamon? 15:31, 25 November 2020 (UTC)[reply]

Approved. Looks good to me. As per usual, if amendments to - or clarifications regarding - this approval are needed, please start a discussion on the talk page and ping. --TheSandDoctor ^Talk 18:54, 25 November 2020 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at Wikipedia:Bots/Noticeboard.