Wikipedia:Bots/Requests for approval/RonBot 4

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Approved.

RonBot 4

Operator: Ronhjones (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 23:32, Friday, March 9, 2018 (UTC)

Automatic, Supervised, or Manual: automatic

Programming language(s): Python

Source code available: User:RonBot/4/Source1

Function overview: Currently DatBot6 reduces all tagged JPG and PNG files in Category:Wikipedia non-free file size reduction requests. RonBot2 does the same for GIF files. SVG files are, by default, sorted into Category:Wikipedia non-free file size reduction requests for manual processing (along with TIFs and PDFs). This bot will reduce just the SVG files in that category.

Links to relevant discussions (where appropriate):

Edit period(s): Daily Run, expect 5 hours max

Estimated number of pages affected: About 2500 now, then a few files each week

Exclusion compliant (Yes/No): Yes

Already has a bot flag (Yes/No): Yes

Function details: Uses a modified version of the code that DatBot6 and RonBot2 is using. Treating the downloaded file as text and applying the methodology of User:Ronhjones/SVGreduce. There are a few files which do not easily work (alternative name for width (wide), and when there is "width=XX%"), these are skipped. Would like to do same as RonBot2 - modify {{non-free reduce}} to move SVG back to parent category, and set the skipped files to the "manual reduce" catagory

Discussion

@Ronhjones: This is uncontroversial (well, no more controversial than image reductions required by policy always are), but I was under the impression automating this was somehow technically difficult. Have you used this methodology before successfully? ~ Rob₁₃^Talk 04:40, 22 March 2018 (UTC)[reply]

I have to thank User:JoKalliauer for enlightening me on how the initial svg tag is formatted and how to easily change the nominal size. That's all summed up in User:Ronhjones/SVGreduce. I have done about 400 odd files manually using this methodology, editing on-wiki with User:Rillke commons script for editing SVG files (See my contributions 26th Feb and earlier). I have also run the bot (with saves only to my hard drive) and only found a few files that caused a problem and they are now excluded from processing...

I found someone had used "wide=" instead of "width=".
There were a few with width="100%" height="100%"

The code is set to skip those. I think the the best plan would be to put the SVG files back into Category:Wikipedia non-free file size reduction requests amd any file the bot does not like gets moved to Category:Wikipedia non-free file size reduction requests for manual processing in the same way RonBot2 does for GIF files - that's not in the current source code yet, but can be easily added. Ronhjones ^(Talk) 19:18, 22 March 2018 (UTC)[reply]

@Ronhjones: If you have problematic files you don't know how to change the previewsize without changing the content of the svg, message me with a link to an example and I am happy to try to take care about it. Now most pics in the manualcategory are in the form: width="[[digit].]+(px|mm|pt|pc|cm|in)*" height="[[digit].]+(px|mm|pt|pc|cm|in)*" which should be changed to width="www" or height="hhh" if viewBox= is in the orginal SVG JoKalliauer (talk) 11:05, 24 March 2018 (UTC)[reply]

Thanks for that. As it stands, if we can get permission to run the bot, from the dummy runs done - I estimate that <0.5% of the files will be skipped for a proper manual evaluation. Ronhjones ^(Talk) 18:00, 24 March 2018 (UTC)[reply]

Added "pc" and "cm" dimensions to the source code. Ronhjones ^(Talk) 14:45, 25 March 2018 (UTC)[reply]

{{BAGAssistanceNeeded}} If this is uncontroversial, then how about a trial? Ronhjones ^(Talk) 22:36, 4 April 2018 (UTC)[reply]

Approved for trial (50 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete. ~ Rob₁₃^Talk 01:16, 5 April 2018 (UTC)[reply]

Trial complete. - Results documented at User:RonBot/4/Trial Ronhjones ^(Talk) 22:06, 5 April 2018 (UTC)[reply]

A)Because files have to be reuploaded and saved completely there should be basic repairing of damaged files at the same time like: (I just looked first 10 reported files)

replace xlink:href="data:image/jpg;base64, with xlink:href="data:image/jpeg;base64, as in File:CRC_Logo.svg
fake svg (SVGs that only contain one rasterimage) should be exported as jpeg or png as in File:Bristol_Pitbulls_2010_Logo.svg (to File:Bristol_Pitbulls_2010_Logo.png)

B)About the file which didn't work if there is a viewBox="[-[digit]. ,]+" remove all height="[[digit].]+(px|mm|pt|pc|cm|in)*" and remove aswell all width="[[digit].]+(px|mm|pt|pc|cm|in)*" and insert width="www" height="hhh"

In this case it should be changed from

<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" wide="500" height="442.4"
	 viewBox="0 0 500 442.4" enable-background="new 0 0 500 442.4" xml:space="preserve">

to

<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" wide="500"
	 viewBox="0 0 500 442.4" width="50" height="44.24" enable-background="new 0 0 500 442.4" xml:space="preserve">

C)Generally to keep the aspect-ratio (also if scale the file). If you set a desired width, I would not round the height to an integer.

D)In my opinion the "target pixel count" is picturedependend: f.e.: File:Cycling_(track)_2018_Commonwealth_Games.svg should have a lower target pixel count than File:The_Guardian_al-Qaeda_recruitment.jpg

PS: Im quite negative about it because, in my opinion to meet fair-use the precision of SVGs hast to be reduced (or converted to png), not the size of the preview. (This is independent if you do it manually or with a bot.)

JoKalliauer (talk) 18:19, 6 April 2018 (UTC)[reply]

A) Repairing. Nice idea, I think better done with a different bot, this will only work on currently tagged oversized SVGs and new uploads. There are thousands of existing SVGs which won't be touched. Need a bot to run through ALL the files (maybe the next one for me). Total 20,000 SVGs on en-wiki, I'm only going to look at 2500.

B) Agree, but my estimate is that there will be so few files, it's an easy manual task.

C) No problem, I'll check the code so there is no rounding - see User:RonBot/4/DummyRun.

D) Agree, but not something the bot can easily determine.

E) I know that, and I know other editors have the same opinion on how reduction is done, this is just one facet. My suggestion would be to allow editors to apply a (new) tag where they think that the precision of SVGs has to be reduced. However, I know there are SVGs (not oversize) that are currently tagged with {{non-free reduce}} - in the hope that someone would do such a process - I've not seen any movement in that direction, but there are so many big files tagged, that they are rather lost in the big list. Ronhjones ^(Talk) 23:59, 6 April 2018 (UTC)[reply]

@Ronhjones: Trial looks fine. Before this is approved, could you edit the text of {{Non-free reduce}} to give instructions to editors who want their file reduced by a human rather than a bot? In some cases, as noted above, this may be desirable for an SVG (and other images as well). ~ Rob₁₃^Talk 13:37, 13 April 2018 (UTC)[reply]
@BU Rob13: I have...

Adjusted the {{non-free reduce}} to mention vector editing programs, state manual reduction is an allowed option, and explained that svg will only be reduced by changing the nominal size, and not reducing the number of vectors.
Created {{Non-free manual svg reduce}} - as a usable template where editors require a reduction of vectors to conform to "low res" can apply to any image.
Fixed the "switch" to move SVGs back to the parent template (current reducing bots are set to skip SVGs anyway), then this bot can pull the SVGs from Category:Wikipedia non-free file size reduction requests and put the few it has to skip into Category:Wikipedia non-free file size reduction requests for manual processing just as RonBot2 does with GIF files. I just need to insert that bit of code from RonBot2 into this bot.

Hope that sounds OK. Ronhjones ^(Talk) 19:40, 13 April 2018 (UTC)[reply]

Code changed and saved in the source page Ronhjones ^(Talk) 19:53, 13 April 2018 (UTC)[reply]

All of the original SVG files have been deleted so I can't see what the bot has changed, but the discussion suggests that the bot operator has misunderstood what reducing a file means and that none of the files in fact were reduced. Reducing means removing non-free content, but it looks as if the operator simply added the <svg> tag. That tag does not contain any non-free content so changing the tag does not mean reducing or increasing the file, only changing how the non-free content is presented. If the files were too big before the bot edited them, then they are still too big. In other words, {{non-free reduce}} should not have been removed from the files as it seems that the bot didn't reduce anything at all. --Stefan2 (talk) 19:57, 13 April 2018 (UTC)[reply]

I personally tagged the vast majority of these files, many, many months ago. They were tagged because they have a nominal size way in excess of the guideline size, thus making the wiki software generate a png image that appears very large on the image page, which is unacceptable. As I explained in point (2) above, I have made another template ({{Non-free manual svg reduce}}), which can be used to indicate that you (or other editors) think that more editing work is necessary to achieve proper non-free status. However that is not something that can be done by a bot, and needs a personal touch. If you don't like my wording of ({{Non-free manual svg reduce}}), then you are free to alter it. This bot was always about reducing the nominal size. Ronhjones ^(Talk) 17:27, 14 April 2018 (UTC)[reply]

@Stefan2: I note you tagged one file, after I did the nominal size reduction manually (there were a couple of others, but deleted), so I have changed the template to {{non-free manual svg reduce}} for you (File:Arunachal Pradesh Seal.svg). Ronhjones ^(Talk) 22:37, 14 April 2018 (UTC)[reply]

Stefan2 might also want to note that the documentation in {{non-free reduce}} says (and I did not write it - it was added 08:16, 17 December 2010‎) For non-free SVG file, add {{Non-free reduce}} if the vector image displayed at excessive nominal size. Ronhjones ^(Talk) 23:42, 15 April 2018 (UTC)[reply]

The template looks wrong. The text seems to have been added in Special:Diff/343420609 but there's no edit summary so I don't know if there was a discussion about the wording at that point. Maybe someone never discovered it as the type parameter rarely is used. I'm not sure why the template even has a type parameter in the first place since most files are reduced by a bot within a day after the file was tagged, and the bot doesn't pay any attention the parameter.

WP:NFCC#3b is about removing non-free content, not presenting the same non-free content in a different way. The nominal size in the <svg> tag is just an arbitrary number - it can be set to anything and only affects how the file is presented on the file information page. In my opinion, changing this arbitrary number does not reduce or increase the file's "size" but just looks like a redundant edit which produces unnecessary clutter on people's watchlists. For pixel graphics, reducing a file means merging two or more pixels into one single pixel, subject to the condition that you hardly should notice a difference if the image is used as a small illustration for an article. For vector graphics, the logical reduction is then that two or more vectors are merged into a single vector, subject to the same condition that it is hard to notice a difference when the file is used in a typical Wikipedia article. --Stefan2 (talk) 23:08, 20 April 2018 (UTC)[reply]

All the best made plans...! moving svg to parent, seems to upsetting DatBot 6, so the switch in {{non-free reduce}} modified to put svg in Category:Wikipedia non-free svg file size reduction requests, and adjusted code to reflect that.Ronhjones ^(Talk) 23:36, 15 April 2018 (UTC)[reply]

I agree with Stefan2, therefore processed the first files in this category using a script (cleanupSVG):

Of course, you could decrease the file size a little bit more (but generally reducing further, would only lead to a negligible file reduce, compared to the previous file-reduction), but the values used in my script is a good trade-of between precision and file size. Which is comparable to a resolution of 0.1Megapixel, so generally there are minor visible changes if you zoom the picture to full screen. JoKalliauer (talk) 14:33, 21 April 2018 (UTC)[reply]

@JoKalliauer: OK, therefore can I therefore suggest that Stefan2 or yourself take over this BRFA? For several reasons...

I think the bot owner should be able to maintain the code, as issues arise, it's clear that that will be impossible for me, it's not written in a code I know, nor do I exactly understand what it actually does.
I'm retiring in six weeks, I intend to move my current bots over to the Wikimedia Toolforge, so that they are not relying on my PC, as I may not be at home to fix any issues. I can then use my 12 inch tablet.

Can I also ask what will the code do with an embed raster graphic? I was considering another bot to highlight those - see Wikipedia:Village_pump_(proposals)#Proposed_Bot_for_tagging_BadSVG_files.. Do also remember that there are over 12000 non-free svg files below the nominal size of 100,000 pixels - I know some editors have made their complex logo a mere 20x20 on the nominal size, knowing it makes zero difference on enlarging Ronhjones ^(Talk) 15:27, 21 April 2018 (UTC)[reply]

@Ronhjones: My script, basically calls 3 different svg-Optimizer (scour, svgo, svgcleaner). If you want to see how to optimize SVGs: SVGOMG is an easy, optimiser with a simple graphical interface (works in every common browser). JoKalliauer (talk) 16:03, 21 April 2018 (UTC)[reply]

@JoKalliauer: I'm not a computer programmer, just a mad organic safety and development chemist. I cannot implement and maintain your script in my code. If you want this, then someone else will have to run the bot. Ronhjones ^(Talk) 17:03, 21 April 2018 (UTC)[reply]

I'm a civil Engineering PhD-Student, so I'm not a computer programmer either. I just call three Optimises with options, which I think fit for Wikipedia. Since they are quite complicated, I think a file-size-reduction is to "complicated"/"idividual" for a Bot, and some optimisations need much computer-resources. In my opinion: Reducing the precision should not be done with a bot, and reducing the preview-size (can be done with your bot), but is in my opinion useless. JoKalliauer (talk) 05:37, 22 April 2018 (UTC)[reply]

It's not useless as it does what {{non-free reduce}} currently specifies (even if Stefan2 does not agree, but it's been like that for 8 years). It does stop the wiki software making an unacceptably large png (wiki software assumes that not all browsers will show an svg, and makes png images where they will be shown) - having non-free pngs of 1000x1000 can never be acceptable. Also I tagged 90% of the current "oversized" svgs solely on the basis of the nominal size. If RonBot3 gets approval it will do the same. I agree that a "superbot" to do a full optimisation is very unlikely, that does not stop this one moving on, until someone writes it. Such a "superbot" would have to run through the entire set of SVGs, as there are plenty below the 100,000 nominal size that would need treatment (currently 14,615 SVG files in en-wiki}}.Ronhjones ^(Talk) 15:27, 22 April 2018 (UTC)[reply]

Summing up - Stefan2 and JoKalliauer would obviously like the bot to do more, but as they say, it's unlikely to be doable with a bot. So going back to the original idea of doing what {{non-free reduce}} has stated for the last 8 years - For non-free SVG file, add {{Non-free reduce}} if the vector image displayed at excessive nominal size. The trial is OK, we just need some final approval. Thanks. Ronhjones ^(Talk) 15:32, 22 April 2018 (UTC)[reply]

images should be rescaled as small as possible to still be useful as identified by their rationale, and no larger
— Wikipedia:Non-free_content#Image_resolution

It is as unlikely that a Bot can derive case-dependent the maximum allowed resolution.

Write a guideline for reducing the preview-size for SVG, if it would get an Category:Wikipedia_content_guidelines, then your bot can start imediatly. But since SVGs don't have any resolution, Wikipedia:Non-free_content#Image_resolution is not valid for SVGs, as long as the preview-sizes of Vectorgraphis is not explicitly included in this guideline. Discuss it with the community, if they agree with it; write it explicitly into the guidelines; and then the approval would be great. But to apply for approval for a bot, which is not supported by the community does not make sense. Here is place for discussing the approval, not how to handle NFC.

conclusio: The community first has to support the edits, and not run a bot against the community. (So first fix the guidelines.) JoKalliauer (talk) 16:07, 22 April 2018 (UTC)[reply]

It is valid (as I said earlier, as we are generating unacceptable oversized pngs - or do you consider that acceptable? If you wish to change what was set up 8 years ago, then you are free to start a new discussion at Wikipedia talk:Non-free content. I am just following the system that is in place. Ronhjones ^(Talk) 22:24, 22 April 2018 (UTC)[reply]

Pinged on this on my talk page, I think this is a reasonable task, though it's not as critical compared with the other bots. This is basically cutting down the size of the pre-cached PNG previews of an SVG on the appropriate File: page, as well as optimizing the SVGs (cropping away empty space, etc.) We can't stop people from dropping a 10,000 px wide SVG image on a page due to the nature of SVG, but we can minimize the cached image sizes to reasonable small sizes. Or another way to put it - the MediaWiki software appears to require us to generate these PNGs for an SVG, so by minimizing the displayed resolution, we better meet NFC. --Masem (t) 22:33, 23 April 2018 (UTC)[reply]

A user has requested the attention of a member of the Bot Approvals Group. Once assistance has been rendered, please deactivate this tag by replacing it with {{t|BAG assistance needed}}. Can we move forward with this one? We have had a look to see the original idea for the use of {{Non-free reduce}} on svg files. The template was changed back in 2010, and we find that user:gringer first added "nominal size", this was after there were some discussions in the Graphics Lab (one here) - that were telling editors to make sure the nominal size was reduced to not infringe NFC. This was the basis for me tagging all the "oversized nominal sized" svg files earlier in the year (I estimate >90% of the files in Category:Wikipedia non-free svg file size reduction requests were tagged by myself for being oversized. Thus there are, in effect, two facets for svg reduction (1) is reduction of "nominal size", which has been stated on the template page, unchanged since the 2010 edit, and also (2) reduction of vectors (favoured by Stefan and JoKalliauer). It's clear from the above that only option (1) can be automated and option (2) has to be manual. In an ideal world, we would get the developers to fix the MediaWiki software so that the png on the image page is a sensible size - however we know that that would take a very considerable time, even if we could get them to agree to do that. Therefore the only way to stop MediaWiki making these oversize NFC png images (as agreed by Masem above) is to reduce the nominal size. Once the nominal size has been sorted, then editors could (if they wish) add {{Non-free manual svg reduce}} to indicate that there is an excessive amount of vectors within the image. Note that reducing the nominal size, not only makes MediaWiki display a suitable size png, but also reduces (somewhat) the sizes of the pre-formed "other resolutions" - e.g. File:Kalilangan Bukidnon Municipal Flag.svg is 5,400 × 2,700, and has pre-formed pngs available of "320 × 160 pixels ; 640 × 320 pixels ; 1,024 × 512 pixels ; 1,280 × 640 pixels ; 5,400 × 2,700 pixels." - reducing the nominal size to 100,000 pixels would make the biggest pre-formed png about 1/16th the size.Ronhjones ^(Talk) 18:02, 1 May 2018 (UTC)[reply]

Approved. Sorry this took me so long to get to; I've had a lot going on off-wiki, and when I finally did have time to get some of this paperwork done, my computer went and died on me. It's clear that the "non-free SVG" debate isn't going to be solved here, but the benchmark enwiki has used for eight years has been nominal size. This bot addresses at least that issue, so I'm approving it. ~ Rob₁₃^Talk 17:11, 26 May 2018 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.