Talk:Standard RAID levels/Archive 2

Archive 1

Archive 2

Performance

Its not clear to me that the various benchmarks cited have anything to do with the performance of Standard RAID configurations, either relative to one another or on an absolute basis. This is because the benchmarks cited are on small machines (PCs, personal Linux) using software RAID and small block sizes. Such configurations have a number of bottlenecks and it is simply not clear what is the bottleneck. While it is clear that RAID doesn't seem to to much in some of these bendhmarks that may simply mean that the things RAID does well, mainly data rate, isn't a bottleneck in such systems. I'd suggest server scale benchmarks by server or storage system vendors such as HP, Sun, IBM, EMC, NetAps, Hitachi be cited rather than the ones currently used. I further suggest most of the current sections on performance should be stricken as undue. About the only think that can be accurately stated is RAID doesn't seem to offer much performance value on PCs and other small systems. Tom94022 (talk) 01:04, 20 September 2015 (UTC)

Hello! Regarding my recent additions, I do agree there's a substantial amount of dubiousness, just as there always is when performance levels are evaluated. I also agree that all performance-related sections could be made far better. However, I disagree that benchmarks made using "small" PCs and software RAID are irrelevant because the days of "ironclad" hardware RAID solutions are long gone, let's just have a look at ZFS that, as a purely software implementation, is much more useful and reliable (as in resilient to silent data corruption) than "good old" hardware RAID solutions. Even the "hardware" RAID implementations are nothing more than software running on resource-constrained embedded systems acting as RAID controllers, which many times have different bottlenecks. Furthermore, doing a benchmark on a "small" PC with more than a few CPU cores and a few fast SSDs pretty much equals in performance to benchmarking an older large dedicated storage system.

Please don't read this as any kind of disrespect, it has nothing to do with something like that. Of course, I'm more than willing to discuss this further, and it would be great to have a few sources that provide large-system benchmarks. — Dsimic (talk | contribs) 07:02, 21 September 2015 (UTC)

Did u miss my small block size problem with most of the cited benchmarks - 16 KiB is way too small to measure RAID performance.

Most RAID is sold by the big iron folks with the RAID implemented in the subsystem in a mixture of hardware and firmware, benchmarks run on "small" machines thru the OS and its file system, ZFS or others, are just not relevant to any real world and therfore mislead any reader. It's comparing apples and kumquats. It is fair to say that benchmarks on small machines show little benefit in performance from RAID but to generalize beyond that is just wrong. Tom94022 (talk) 07:47, 21 September 2015 (UTC)

Well, you can configure and use pretty much whichever block size. How many people buy "big iron" storage systems, and how many buy "small" servers with many drives? Let's remember that "small" servers form a large chunk of the real world. — Dsimic (talk | contribs) 08:08, 21 September 2015 (UTC)

Furthermore, please see some of the EMC VNX documentation, which describes EMC's products "designed for midtier-to-enterprise storage environments":

• https://www.emc.com/collateral/white-papers/h12145-intro-new-vnx-series-wp.pdf
• https://www.emc.com/collateral/white-papers/h12090-emc-vnx-mcx.pdf

Everything points into the VNX platform being nothing more than a rather ordinary "small" PC running EMC's specialized software. No "hardware" magic seems to be present. — Dsimic (talk | contribs) 09:41, 21 September 2015 (UTC)

The last time I looked at the storage subsystem market, the revenue and drive units were dominated by big iron subsystems; the EMC VNX subsystem being on the low end of the market. Yes there were a lot of small users but they didn't use a lot of drives nor generate a lot of revenue. Again, for those small users the system bottlenecks are such that they can't gain much, if any, performance benefits from RAID.

Perhaps I should have said results from runing "third party software benchmarks under OS software not necessarily optimized for RAID" is inappropriate for this article. The software and hardware running in the EMC VNX should be optimized for RAID, maybe it isn't, but that's what benchmarking is for. Having worked on storage subsystems I am pretty sure the "sofware" that moves the data in the big iron storage subsystem is highly optimized for that task; one might argue it has no OS and very efficient data movers. It is definitely different than the environment used in the benchmarks cited.

Since the primary berformance benefit of RAID is data rate (all but RAID 1) and perhaps access time in RAID 1 the benchmarks attempting to measure the absolute or relative performance of various RAID storage subsystem should try to minimize other performance losses, e.g., if u think the test is compute bound, run with faster computers until the results don't substantively change with faster computers. The 16 kiB block used in the AnandTech HDD benchmarks is about a 0.1 msec data transfer native and 1/2 that on RAID 0; this is nothing compared to seek time, latency and perhaps some of the OS and file system computation times. The only valid conclusion is the benchmarks AnandTech used are not sensitive to data rate. 01:13, 22 September 2015 (UTC)

Oh, I totally agree that the software running on the EMC VNX boxes is surely highly optimized for the task, compared with the Btrfs' RAID implementation that runs inside Linux, which is a general-purpose operating system. One could argue that the Linux kernel is also highly optimized in general, but it can hardly reach exactly the same level of optimization and resulting hardware utilization for every single application that's achievable in single-purpose software. Though, Oracle sells some quite large storage boxes that use ZFS (and I'd bet they run Solaris), see this or this, for example.

However, I strongly disagree that small systems can't see any performance benefits from RAID. I've had more than a few "small" PC servers with two to 15+ hard disk drives (PATA, SATA, SCSI, NL-SAS, SAS) in RAID 0, 1, 10 and 5 configurations, and almost every single of them saw huge performance benefits from RAID. Some of the services and applications running on those servers wouldn't even be able to work without the I/O performance levels achieved using different RAID configurations. With PCI Express becoming mainstream, PCs pretty much got rid of virtually all bottlenecks; it's PCI and PCI-X what looked almost like a joke in any larger configuration, which is why I wrote "almost every single of them". :)

Speaking about my additions to the article, I tried to provide further benchmark-based comparisons between single-drive and RAID 0/1 configurations, which is the main theme of our "Performance" sections. By the way, IMHO it makes sense to have such comparisons in the first place, simply because we should know how exactly RAID compares to single drives. I agree, it would be great to have the results of such benchmarks run on "big iron", but I really don't see anyone running such single-drive vs. RAID benchmarks on "big iron" – tried searching for them, no bueno, people buying "big iron" don't seem to care about that. — Dsimic (talk | contribs) 06:06, 22 September 2015 (UTC)

I agree that small systems can see benefits from RAID, the fact that the benchmarks cited did not sugggests to me that they are not reliable for this article.

I rewrote the RAID 0 section to give it some balance, leaving in the material in about desktop performance even though IMHO it is not relevant.

Cost & Performance Issues has some interesting comparisons but not to one drive which again I am not sure is particularly relevant to this article. Do you think any signinificant number of end users make such a tradeoff other than perhaps a few who implement RAID 1 (FWIW I will probably go RAID 1 SSD on my next desktop) ? Tom94022 (talk) 20:24, 22 September 2015 (UTC)

Please let me try to find additional small-system RAID benchmarks and add them as references. The definition of a modern "desktop" is rather stretchy – even on a modest modern dektop PC you can have more than a few virtual machines running at once, for example, and that surely can use the additional performance RAID can provide even on a small number of drives. Moreover, software developers run multiple virtual machines very often as part of their standard environments that run on usual desktop PCs. Let's also take into account servers, some of which are very similar to better desktops – virtually none of them come with a single drive, and only a very ignorant system admin would run a server without the data redundancy provided by RAID 1 or 10, for example.

Speaking about the need for having the single-drive vs. RAID performance comparisons, I'd say those are rather relevant. I've seen more than once desktop PCs that had two or more hard disk drives, but people didn't use them in any RAID configuration, explaining that by potential slowdowns and complexity of software configuration. Thus, IMHO it's at least that readers may benefit from such performance comparisons.

The comparison you've provided is a really good one, we should probably see how to integrate it into the article as a reference. Your edit looks fine to me, I've just tweaked the wording a bit and mentioned computer gaming as a rather important user of the technology. No matter how much gaming "rigs" may look ridiculous, they are a very important consumer of advanced PC technology and they actually drive many of the advancements. — Dsimic (talk | contribs) 03:36, 23 September 2015 (UTC)

The reason I don't think single drive comparison is particularly relevant to this article is because very few admins (including single users as their own admins) really make that trade off; let's face it, the reliability and capacity of one HDD is so good that most people don't need or have more than one per system, e.g., a recent survey of corporate laptops found 97% using less than 200 GB. Perhaps a better example is I can't think of any desktop manufacturer offering two HDDs as standard. A small percentage of users may have a second drive for back up which for most users is a better solution than RAID 1. So IMO RAID is the domain of users who need lots of drives and want reliability; the trade off is in practice between RAID 1, RAID 5 and RAID 6, which is why this comparison against RAID 0 in is relevant. I suppose a comparison against a JBOD of the same number of drives might also be relevant but I think the number of data drives needs to be at least 4, maybe more, i.e. benchmarking, 4 drive JBOD, 4 drive RAID 0, 8 drive RAID 1, 5 drive RAID 5, 6 drive RAID 6 and maybe 2x4 drive RAID 10. This puts it out of the domain of desktop and personal gaming benchmarks and equipment.

U certainly are welcome to look for 1 drive comparisons and include them in the article but I do think they need to have some of the above perspective much as it was added into the RAID 0 section. Tom94022 (talk) 16:31, 23 September 2015 (UTC)

U might take a glance at DS5000 arrays and RAID levels as an example of the type of relative performance discussion that should be summarized in the article. Tom94022 (talk) 00:55, 24 September 2015 (UTC)

Those are all reasonable arguments. Hm, the more I think about it, the more I find that it would be the best to ditch separate "Performance" sections and merge them into a single "Performance" section that would also provide a comparison between different RAID levels. I've also tried to find more single-drive vs. RAID benchmarks, but they don't seem to be widely available. — Dsimic (talk | contribs) 03:47, 24 September 2015 (UTC)

I'd have no problem with a separate performance section. I do suggest we leave out RAID 2, 3 and perhaps 4 as not currently practiced. I'm happy to work with u on such a section.

I did a thought experiment and concluded that a JBOD (independent not spanned) should out perform RAID 0 for small reads and writes and the performances would converge for large reads and writes. Then I found JBOD vs RAID among a whole bunch of discussion on this subject. Many seem to think RAID 0 would beat JBOD but I suspect that should only be true in a single threaded environment or with spanned JBODs. All of this goes to show that anything we write about performance needs to be carefully written and properly sourced. Tom94022 (talk) 18:33, 24 September 2015 (UTC)

Here is another article in the same vein: Why not RAID-0? It’s about Time and Snowflakes. Tom94022 (talk) 20:56, 24 September 2015 (UTC)

As we know, RAID isn't by far the silver bullet that solves all storage-related problems. In addition to the articles you've linked above, these are also good examples:

• https://www.backblaze.com/blog/vault-cloud-storage-architecture/  – discusses data redundancy implemented on the object level
• http://www.zdnet.com/article/comprehensive-raid-performance-report/  – describes how multiple RAID 1 sets easily beat one large RAID 10 set

No automated low-level data arrangement can beat carefully laid out high-level data placement that follows the actual usage patterns. That's also an important performance aspect we should cover somehow in the performance comparison, if you agree. Oh, and I agree that discussing the performance of RAID 2, 3 and 4 wouldn't be needed, moreover there aren't many good benchmarks of those RAID levels that would make a good coverage possible. — Dsimic (talk | contribs) 07:33, 25 September 2015 (UTC)

Here's another benchmark that compares performance levels achieved by hardware RAID 0 and high-level data placement, showing rather interesting results. — Dsimic (talk | contribs) 10:42, 7 October 2015 (UTC)

I'm having severe problems with using real-world benchmarks to rate RAID levels (and such) in general. A benchmark shows how well (or not so) a specific setup runs certain workloads. Not more, not less. When comparing RAID levels in general, you need to look at the potential of the method – possibly taking into account various algorithms – and assuming a perfect setup that has no bottleneck except the one you're currently looking at. Quite a lot of real hardware is throttled by the manufacturer, either on purpose (marked down product line) or by using suboptimal hardware (budget) or because of simple ignorance. When using real-world benchmarks for validation, you need to have detail knowledge about the hardware, its limits, and algorithms used – which most probably is only possible with Linux soft RAIDs and such. --Zac67 (talk) 17:47, 7 October 2015 (UTC)

Agree with Zac67. I think the best we can do is find a simulation or theoretical analysis of relative performance and then note in real world benchmarks frequently do not achieve the capabilities do to limitations of the benchmarks. Tom94022 (talk) 07:23, 10 October 2015 (UTC)

Yeah, but theoretical analysis and simulations aren't always sufficient. If something works great in theory but doesn't perform very well in the real world, it's that (a) the theory is wrong, (b) the theory can't be applied to the real world, (c) widespread implementations of the theory are bad, or (d) the observed behavior simply proves that there's no silver bullet. Please don't get me wrong, IMHO we should cover both the theoretical analysis and benchmarks, which also adheres to the WP:NPOV guideline. — Dsimic (talk | contribs) 08:16, 10 October 2015 (UTC)

There is also e) the benchmark is inappropriate for the parameters being assessed. Absent a RS, a) thru e) are likely POV or OR.

Perhaps an example might help. Consider a benchmark of a new HDD of a given capacity against an older HDD of say 1/4 the capacity but having the same RPM and about the same average seek time. The new drive would likely have 1/4 the disks and 4 times the data rate. Any benchmark with short data transfers will show very little difference since the performance would be limited by latency and seek time. Most application benchmarks and many synthetic benchmarks have short data lengths. The new drive will in fact transfer the data 4 times faster, it just won't matter for most benchmarks and in the real world.

Don't get me wrong, If we choose to include benchmarks I suggest we have to be very careful in how we present their results. After all how would we deal with the drives in the example? The new HDD does transfer data 4 times faster than the benchmark HDD, but data rate just doesn't matter in most applications, many benchmarks and the real world. Tom94022 (talk) 17:43, 10 October 2015 (UTC)

Good, I think we're on the same page here. Benchmarks are important but they need to be taken with a grain of salt. However, while many (possibly most) workloads depend largely on latency, there are many workloads that depend heavily on data transfer rates as well, like backups and working with large audio/video files for instance. --Zac67 (talk) 18:23, 10 October 2015 (UTC)

I wouldn't agree that the sequential HDD data transfer rate isn't important at all in the real world; for example, and in addition to the examples Zac67 already provided, it's rather important for relational databases and queries going through large data sets, for reading and extracting data from various large log files, for moving virtual machines between different hosts that don't use shared storage, etc. Virtually every single MB/s counts in such use cases, which aren't that uncommon.

Speaking about the benchmarks, IMHO we should focus on presenting the relative performance levels, based solely on how each of the benchmarks rates different tested RAID levels (and, if available, single-drive configurations as well). That way, we should be rather free from the comparison troubles introduced by the changes in drive performance over time; in other words, we shouldn't cross the boundaries of single benchmarks, which would also comply with the WP:SYNTH guideline. — Dsimic (talk | contribs) 23:25, 10 October 2015 (UTC)

As it turns out there was just a webcast "Storage Performance Benchmarking: Introduction and Fundamentals" to be followed shortly by Part 2. The speakers are from NetAps, EMC and Cisco, so perhaps they are the reliable source we can use. I'm going to watch both. To be continued. Tom94022 (talk) 18:39, 16 October 2015 (UTC)

Good find, I'm halfway through the first one. — Dsimic (talk | contribs) 00:45, 17 October 2015 (UTC)

RAID 6

The article, as it stands, defines RAID 6 as any system that can survive two disk failures and then, from my reading, seems to go on to define a particular way of implementing it, implying that this is the definition of RAID 6. This is clearly inconsistent. I've added reference to proprietary RAID 6 implementaiton such as RAID-DP and RAID-Z2, but clearly more cleanup is needed in this section, since it's not currently self-consistent. In practice, many RAID 6 implementations are likely to be proprietary systems differing from the one described, even if branded as RAID 6.

Either the parity scheme described is the canonical implementation of RAID 6 (and other implementations are not strictly RAID 6) or it is just one example (perhaps the original example) of how RAID 6 may be implemented. I'm not sure which is right at the moment. Quite possibly both are, depending on what source you use for the definition of RAID 6.

Given Patterson, Gibson and Katz's paper only describes levels 1 to 5, I'm tempted to go with the generic SNIA definition of RAID 6 given in the article. But then we should reference the implementation we describe. e.g. "An early proposed RAID 6 implementation, proposed by XXX, is as follows...".

Roybadami (talk) 22:41, 18 November 2015 (UTC)

And if there is genuine controversy about what constitutes RAID 6 then we should, of course, give multiple definitions describing as far as possible which sources use those defintions. Roybadami (talk) 22:43, 18 November 2015 (UTC)

RAID 10 and the definition of 'standard'

What makes a RAID level standard? The article doesn't seem to reference any standard (and AFAIK there isn't any accepted standards document on this). But from where I'm standing RAID 10 seems to be a widely implemented RAID level across multiple vendors with broad agreement as to what it means. And despite the implication in Non-standard_RAID_levels that this is specific to Linux md, it patently isn't, with hardware RAID solutions (both controller cards and external arrays) from many vendors implementing RAID 10.

I'd be tempted to say the same about RAID 50, but perhaps that's more questionable.

Roybadami (talk) 22:58, 18 November 2015 (UTC)

Ok, perhaps I should say there's no universally accepted standard. The lead does cite an SNIA standard, but given most of these levels predate the SNIA for decades I think this is misleading - unless we were to rename this article as 'SNIA Standard RAID levels' (which I don't think would be a particularly useful approach).

Given the SNIA glossary claims there are six 'Berekely standard' RAID levels, namely 0, 1, 2, 3, 4, 5 and 6 (count them, that's seven), the SNIA is not exactly gaining credibility with me. And given the original Berekely paper only defined five levels I'm even more confused (although it's very probably there's a later paper that defines a sixth - is that RAID 0 or RAID 6 though?)

Anyway, I'm not sure any of this is useful - IMHO Standard RAID levels, Non-standard_RAID_levels and Nested RAID levels should be merged into a single 'RAID levels' article. Thoughts? Roybadami (talk) 23:16, 18 November 2015 (UTC)

Minimum number of drives

In Linux MD, the minimum number of drives for a RAID-5 or -6 is 2 or 3 repsectivily, since you only need space for data and parity. The information is based on most hwraid controllers, where they take it for granted that you want at least 3 drives in a RAID-5. With Linux MD you can start happily with two drives and add more as you go. Even though the initial RAID-5 will function somehow like a mirror, it's still, technically, a RAID-5. Rkarlsba (talk) 03:02, 25 March 2016 (UTC)

Hello! Well, that would be, basically, like having a degraded two-drive RAID 5 array, or a degraded three-drive RAiD 6 array. You can also have a degraded two-drive RAID 6 array. However, that doesn't change the fact that regular (non-degraded) RAID 5 and 6 arrays require at least three and four drives, respectively. — Dsimic (talk | contribs) 21:20, 16 April 2016 (UTC)

If a RAID 5 consisting of 2 1 TB disks is degraded, it means that it's supposed to consist of 3 disks (+ optional hot spares). It offers 2 TB of usable storage space. A RAID 5 consisting of 2 1 TB disks isn't necessarily degraded. It can be designed like that. It then only offers 1 TB of usable storage space, just like a RAID 1 consisting of 2 1 TB disks would. It behaves like a RAID 1, it's just more complicated. McGucket (talk) 09:50, 4 March 2017 (UTC)

Incorrect failure rates in "Comparison" section

The failure rates in the "Comparison" section should be much lower in practice, as these assume that the drives that fail don't get replaced. For instance, in the RAID 3 example the failure rate, 1 - (1-r)^n - nr(1-r)^(n-1), is basically the number of times that two or more drives fail over an n-year period. However, if drive A fails after 3 months and drive B fails after 2 years, one can assume that drive A would have been replaced in the meantime and no data would've been lost by the time drive B failed. A reasonable way to estimate the failure rate would be to put in the formula the average time it takes to replace a failed drive, not the entire lifetime of the operation. 78.0.209.217 (talk) 16:36, 15 February 2016 (UTC)

Just checking, did you read the footnote in "Array failure rate" column, which says that it "assumes independent, identical rate of failure amongst drives"? In other words, those aren't true statistical failure rates, which wouldn't be too useful anyway. — Dsimic (talk | contribs) 17:14, 15 February 2016 (UTC)

@Amirsegal, Zac67, and Dsimic: After reading the second of Amirsegal's references I realized the formulae provided in the column "Array Failure Rate" was as near as I can tell without a reliable source and more or less meaningless gross failure rate with out rebuilding. A more meaningful measure might be MTTDL (Mean Time To Data Loss) which might have reliable sources but the second reference, ""Mean time to meaningless: MTTDL, Markov models, and storage system reliability," suggests that too is flawed. At this point I think it is original research to come up with a meaningful comparison in terms of a set of formulae or numbers so I took the whole column out. Since Amirsegal seems to know quite a bit on this subject perhaps he can find us an RS. Tom94022 (talk) 22:23, 1 September 2017 (UTC)

Explanation of "x"?

I'm by no means a computer expert, but I was trying to understand something here:
https://wiki.riteme.site/wiki/Standard_RAID_levels#Comparison
In the chart that is shown there, there are references to expressions like "(n − 1)×" and "(n − 2)×" etc.
I tried to find an explanation of what "x" represents in these equations, but could find none. I understand it's a variable, but what values could be put in for "x" and where do these values come from?
I apologize if there's an answer in the article somewhere, but if there is I couldn't find it.
Should this also be explained in the article?
Thanks!
Skitar (talk) 02:13, 4 January 2018 (UTC)

• I'm unfortunately a computer expert in some areas (compiler engineer). Ignore my bad math anyway, it's late. But for x... I'm not sure what this is either. Superscript like that indicates an exponential aside from the lack of mention of the variable, but those aren't involved here. For RAID-0, for example, we really see the optimal to be the read speed multiplied by the number of drives, assuming only one sector or block can be read at a time which holds for normal HDD and SSDs. (Yeah, I know) I'd expect, and have witnessed, an approximate speed of N * R where R is the number of drives, for both read and write. RAID-1 implementations would be N * R read (but most software-raid messes this up and uses one drive for read rather than distributing them).
• I won't speak for RAID-2 through 4 since I haven't tested them, but RAID-5 is close to the speed of RAID-0 on reads; the actual parity bit just determines whether a group of bits is even or odd. The RAID-5 read listing in the article is probably wrong. Let's say we choose 8 bit chunks for the parity bit to be inserted after. We'll end up with about 89% of the data in any given block, with the rest being parity. There's nothing preventing this from being read from all drives at once, so ignoring parity checking (which is cheap, you're dedicating less than two nanoseconds per 8 bits to parity checking on a modern desktop, which isn't even specialized for this... heck, you can probably offload it to a GPU and get full speed writes. this is nothing compared to the speed of reading everything, even multiplied by the amount of data, so I'd toss it as insignificant. The article missed the point that the parity bit is distributed on purpose to avoid the slowdown on reads
• Writes are the slower part and depend hugely on having a good dedicated processor to make all chunks even or all odd. Even then, we're picking a parity, doign a xor of itself with every chunk of data (not like xor is slow, but now we're modifying memory instead of just examining a stream which could be done as the data is read from the drives, and determining the new size / ordering of data to be written across the drives. On soft-raid you'll get poor results at this stage, and on cheap hardware raid you won't do much better. Maybe a hardware card somewhere can keep up with 3+ SSDs in RAID-0, but Intel soft-raid, despite having improved over the years, could only manage about 200MB/s write tops, with 3% CPU use. Read was close to 1.5MB/s. RAID-0 read was 1.7, write was 1.6. I'd test on hardware RAID some time, but I'm waiting for the RAID manufacturer's supply of 233MHz PowerPCs to run out so I can see if RAID-5 writes are still slow on a 2GHz 8 core off-the-shelf ARM chip. (I guess I could test with in-memory data & multithreading on the 8 core ARM in my phone and the 6/12 on the PC and see what wins in bandwidth adjusted for memory and clock speeds).
• Honestly this was the least of my issues with this article.

1. In the chart for RAID-4, where read speed is simple (read data, wasting the time that could be used reading one more drive), I'd expect something like N-1. We have to read 4 drives, we only get data from 3. Instead an unexplained formula appears.
2. Although I appreciate reading it for the memories, I'm not quite sure that a Galois field is the best way to explain anything as simple as this anywhere, especially in mathematical notation. They didn't start teaching that until 4th year mathematics in college, and it caused about 25% of the class to leave immediately. If the professor had given the explanation on the Galois Field page https://wiki.riteme.site/wiki/Finite_field#Definitions,_first_examples,_and_basic_properties it would have been 90%.
3. I wish I could help with the math section, but I explain things by giving proofs of them when on the topic of math, and that requires a formal language of logic and knowledge of proof types that wouldn't fit. Your average person off the street would have far less trouble computing 7^8 in base 7 in their head than 95% of the populace would reading that section. That isn't an insult to anyone, it's a measure of how complicated the section is.

• 0E8h 00h 00h 00h 00h EdityMcEditorson (talk) 11:07, 4 January 2018 (UTC)

Parity Details

I would suggest to include some information about parity in raid levels. As raid 3 and raid 4 are discouraged, this should go inside of the raid5 section. The word parity can mean a number of different things, so it's good to mention a simple operation (xor $\oplus$) early on and not keep it vague. I suggest to include a few lines like:

The parity is calculated using the xor ($\oplus$) operation which can be chained over $n-1$ drives. With four drives, that is $A_{12}^{p}=A_{1}\oplus A_{2}$ and $A_{p}=A_{12}^{p}\oplus A_{3}=(A_{1}\oplus A_{2})\oplus A_{3}$. So, $A_{12}^{p}$ can be reconstructed from either $A_{1}$ and $A_{2}$ or from $A_{3}$ and $A_{p}$. Consequently, when the drive containing $A_{2}$ fails it can be reconstructed as $A_{2}=(A_{p}\oplus A_{3})\oplus A_{1}$. This extends to any number of drives.

That would make it clearer why there is a minimum number of drives but no maximum or even reasonable number of drives for the algorithm to work.

Shall I go ahead and include this? Yeorasae (talk) 14:59, 24 January 2017 (UTC)

• I find this clearer, but I'd just explain it without the xor notation which is kind of specific to having read certain math material and use an explanation that includes xor. This is a technical topic, not particularly a math or CS one. I came here looking at the differences between RAID 3 & 4 and ended up with Galois fields. EdityMcEditorson (talk) 11:12, 4 January 2018 (UTC)

Raid-2 claims unsupported by cited sources

Specifically, the sentence "Extremely high data transfer rates are possible" appears to be unsupported by the two cited sources.

1. Vadala, Derek (2003). Managing RAID on Linux. O'Reilly Series (illustrated ed.). O'Reilly. p. 6. ISBN 9781565927308.
2. Marcus, Evan; Stern, Hal (2003). Blueprints for high availability (2, illustrated ed.). John Wiley and Sons. p. 167. ISBN 9780471430261.
   

I'm unable to find a supporting reference. Nacarlson (talk) 18:58, 2 November 2018 (UTC)

Given the dates of the sources it is not surprising to find no support for high data rates in RAID2. However, the only implementation of RAID2 did transfer 32 bits in parallel which is a pretty high rate so I rewrote the sentance accordingly. Tom94022 (talk) 20:54, 2 November 2018 (UTC)

"Extremely high data transfer rates" is a very relative term. It might have been extremely fast in the 1980s but today this kind of parallelism is bog standard. In theory, the RAID 2 from the diagram could ideally transfer user data at four times the speed of a single disk, provided there's no congestion and all data streams to/from the disks can run at media speed. A RAID 6 with seven disks can (also in theory) read with seven times the speed of a single disk and write with five times single disk speed (assuming best case, full stripes, no write amplification and on-the-fly parity generation without delay). Where is RAID 2 extremely fast now? As it is, the claim is still unsourced. --Zac67 (talk) 22:41, 2 November 2018 (UTC)

Doesn't 32 drives in parallel exceed by a large margin any other known RAID configuration? I suppose u can build a RAID 10 with 64 drives and also achieve "extremely high data rates" but no one has ever done that. There are also some HIPPI historical examples of extremely high data rates achieved either by many parallel drives (or parallel heads). The point being RAID2 with a high rate Hamming Code inherently has many drives in parallel leading to very high data rates. Such a fact doesn't need sourcing beyond the multiple drives in parallel which is supported by both the link to the Hamming Code and the link to the Connection Machine. Maybe the sentance could be better worsmithed but I leave that to other editors. Would you prefer "very high data rates?" Tom94022 (talk) 06:17, 3 November 2018 (UTC)

"Doesn't 32 drives in parallel ..." – no, it doesn't. 32 drives in parallel is quite possible with RAID 6 and commonly exceeded with nested RAID, esp. RAID 60. Those "extreme high data transfer rates" without context (and without any figure) would also need to stand up against today's flash storage which is ridiculous. Do you think RAID 2 is used with flash? Why isn't it? I think this claim needs at least more (temporary) context or a decent, current source. Anything that's challenged requires a source, and I challenge this claim as it is. --Zac67 (talk) 10:53, 3 November 2018 (UTC)

Anything is possible and a SSD RAID2 would have very high data transfer rates when compared to any other RAID configuration other than perhaps RAID3. RAID2 isn't used by anyone any more and that is made clear in the article. Anyhow, I added a ref, so maybe we can end this discussion. Tom94022 (talk) 17:44, 3 November 2018 (UTC)

Is the "uniquely determine the lost data" statement true in RAID 6 "Simplified parity example"?

It states the following:

" $$A=\mathbf {P} \oplus D_{1}\oplus D_{2}=D_{0}\oplus D_{3}$$$$B=\mathbf {Q} \oplus \mathrm {shift} (D_{1})\oplus \mathrm {shift} ^{2}(D_{2})=D_{0}\oplus \mathrm {shift} ^{3}(D_{3})$$On a bitwise level, this represents a system of $2k$ equations in $2k$ unknowns which uniquely determine the lost data. "

If we XOR the two formulas, we get $D_{3}\oplus \mathrm {shift} ^{3}(D_{3})$ value, which seems to leave a freedom due to dependence in these equations? --188.146.160.41 (talk) 09:25, 27 April 2019 (UTC)

Raid 3 and Obsolescence

• Short story is that it isn't obsolete judging by Areca products. Areca sells controllers with RAID 3 and 30 support, and solid state drives eliminate the need for spindle sync. That's on a controller that supports 256 SAS 12Gb/s drives, so those levels likely still see use. They probably lost popularity 30 years ago, but someone is finding a use case. A thousand dollars is cheap for a server card but Areca spends more by including it and hardware manufacturers don't like doing that. The section doesn't match modern applications or list any (video editing can either be done on non-raid drives or on specialty hardware. Prices are lowering enough most of it can be done on a ramdisk, and a medium-high end desktop can encode 8k lossless JPEG-2000. Low power arm processors exist that run so quickly that you don't really need to store the data, just buffer it while it writes out. Blade servers with 12TB of ram have been announced. Point is, for now, solid state is superior to any form of raid that isn't solid state, and a reference from 2003 might not be the best judge of whether a storage method is good. Thanks to whoever cleaned up the tables and shortened the wall-of-Galois math. I'll try to track down someone using a 256 drive RAID 3 config (possible with expanders and a single card) and find out why they chose that over another raid level. The fact that some of the most expandable cards are being sold with support is good enough for me to question that particular paragraph.
• Going to a different page, the nested raid levels listed on https://wiki.riteme.site/wiki/Nested_RAID_levels are also all supported by Areca controllers, so it seems odd to differentiate them from the traditional "standard" levels, out of which 2 and 4 aren't really used anywhere I can find.
• I'm pedantic. This is all. EdityMcEditorson (talk) 01:56, 7 January 2018 (UTC)

The citation from Sun listing RAID 3 as one of the most popular levels along with 1 and 5 tempts me to retract my previous agreement with said statement, but it is there in the references.

EdityMcEditorson (talk) 02:05, 7 January 2018 (UTC)

A reference from a 17 year old book about managing RAID on Linux seems kind of bizarre when talking about obsolescence... and the reference links to a search for "raid 2 implementation". All the book said about it is:

Remember that RAID-2 and RAID-3 are now obsolete and therefore will not be covered in this book.

I was under the impression that RAID-3 was still used reasonably often for video applications. Apparently manufacturers of external NAS / RAID storage enclosures didn't read this book either and tend to implement it in devices for the video market. Granted with modern disks the speed improvement over RAID-5 is much less, but this article isn't about whether or not people should use it. Also of note is that the Linux book on the subject is probably talking about obsolescence in terms of software RAID in Linux (go figure), where it is obsolete. --A Shortfall Of Gravitas (talk) 23:56, 17 March 2020 (UTC)

References

Is the RAID 5 sector order correct?

I would have expected that not only the parity data "rotates" between the discs, also the other data as well.

So instead of this scheme given in the graphics

RPL,RQL=05,02_hex

A1	A2	A3	Ap
B1	B2	Bp	B3
C1	Cp	C2	C3
Dp	D1	D2	D3

I'd expect the data is distributed this way:

RPL,RQL=05,03_hex

A1	A2	A3	Ap
B2	B3	Bp	B1
C3	Cp	C1	C2
Dp	D1	D2	D3

Can anyone provide reputable sources for the one or the other scheme? --RokerHRO (talk) 16:17, 5 February 2021 (UTC)

There are various ways to do it. For a reputable source, see the SNIA ref. For sequential throughput optimization, your scheme is better. --Zac67 (talk) 16:53, 5 February 2021 (UTC)

Hm, interesting. I've just read the SNIA PDF. I've never heard of RPL, RQL etc. before. Perhaps these terms should be mentioned in the article, too. What do you think?

But I just asked my favoured search engine: No webpage that explains RAID mentions these terms, so I don't know which of these different "RQL" variants are used in real-world RAID implementations. :-/

--RokerHRO (talk) 16:52, 8 February 2021 (UTC)

My thoughts exactly, a few years back... But feel free to add some reference anyway. --Zac67 (talk) 17:27, 8 February 2021 (UTC)

It doesn't look like there is any definitive source for variants in RAID 5 layouts or for that matter RAID 6. There was a RAID Advisory Board which went defunct circa 2001 and it published The RAID book which in its 6th and last edition (February 1997) only discloses the embodiment shown in this article, namely "RAID-5 Rotating Parity N with Data Restart," RPL:RQL == 05:02_hex. I suspect this is a fairly common construct because it is "inherent" that one starts by writing the N blocks to then calculates and writes the parity block and has the historical support of The RAID book.

SNIA actually defines three versions of RAID 5; I suppose there is at least a 4th, namely "RAID-5 Rotating Parity 0 with Data Continuation". I suspect that many are "RAID-5 Rotating Parity N with Data Continuation," RPL:RQL == 05:03_hex for its performance advantage with large contiguous blocks but inertia and simplicity has a lot of "RAID-5 Rotating Parity N with Data Restart" but I doubt if there is any RS as to distribution of RAID layouts in the market.

The SNIA Spec called DDF Specification was submitted to INCITS for standardization (see: RAID Compatibility Puzzle) but I don't know how that turned out. It's not clear to me that the SNIA document by itself is a RS nor is it clear to me what we should do about the sector order issue in the article. Your thoughts? Tom94022 (talk) 21:10, 11 February 2021 (UTC)

The four types of RAID5 are well discussed using terms left/right, synchronous/asynchronous, see, e.g [1]. None of the search results seem to be a particularly good RS but the collective consistency should allow us to pick one ref, e.g. RAIDs utilizing parity function(s) and explain in the article that the image is one of the four types of RAID5. Interestingly I have not yet found such a taxonomy for RAID6 which should be even more complex. Tom94022 (talk) 19:34, 16 February 2021 (UTC)

BTW it looks to me like RAIDs utilizing parity function(s) has an incorrect figure for the Right-Synchronous layout so it's not an RS. Tom94022 (talk) 20:21, 16 February 2021 (UTC)

RAID 6 Simplified parity scheme is wrong?

I'm no expert on erasure codes, but I'm pretty sure the Simplified Parity Example section is wrong.

Consider the case where all data bits are zero. In that case, P and Q will also be all zeros, and if we lose disks $D_{0}$ and $D_{3}$ (or any two data disks), A and B will also be all zeros. But if we use the given equations for A and B to try and recover $D_{0}$ and $D_{3}$:$$0=D_{0}\oplus D_{3}$$$$0=D_{0}\oplus \mathrm {shift} ^{3}(D_{3})$$ We can easily see that these equations are true when $D_{0}$ and $D_{3}$ are all zeros, or all ones. So there is no unique solution for $D_{0}$ or $D_{3}$, and we cannot recover the lost data.

This problem is not limited to the case where the data bits are all zeros (or all ones). I think there are always two solutions to the equations, regardless of the inputs.

Also, I haven't found any other source that describes this algorithm. It seems to only exist on Wikipedia.

I did find some RAID-6 coding schemes which are similar to this one, but not identical: EVENODD and RDP. They are described nicely in [The RAID-6 Liberation Codes].

PromisesPromises (talk) 22:46, 18 January 2022 (UTC)

Yea, I recently read through this when talking to a friend and went... wait a sec, these 2k equations aren't actually linearly independent. If you view Di and Dj as vectors in Z2^n, and use a simple Shift(Dj), you end up with a system like [100 100, 010 010, 001 001, 100 010, 010 001, 001 100] = v, for k=3, n=3. Using any shift function on Dj for any size system will give you one redundant equation, and *not* a well-defined solution for your system.

However, to convince myself I did a little more work, and used (as a simple *working* example) the polynomial field Z2[x]/<x^3+x+1>, which is a 8-element irreducible polynomial field over {0,1}, since the only nonzero element, 1, isn't a root. You can see that x has order 8 in this field (keep multiplying 1 by x and reducing via the relation x^3+x+1=0), and thus can be used as a generator per the general case in the next section. Finally, use multiplication by g=x as our "shift" operator instead. And Dj=y0 + y1*x + y2*x^2, we have g*Dj = y0*x + y1*x^2 + y2*(x^3 -> x^2 + 1) = y2 + (y0+y2)*x + y1*x^2. Using these in our linear system instead from above gives [100 100, 010 010, 001 001, 100 001, 010 101, 001 010]. A little work shows that these equations ARE independent, and thus any parity blocks P and Q can uniquely reconstruct Di and Dj.

For a little more abstract explanation (sorry, I'm actually figuring this out as I go), we need Di+Dj=0 and Di+g*Dj=0 to have the unique solution Di=Dj=0. Combining these gives Dj+g*Dj=(I+g*)Dj=0 requiring Dj=0, thus I+g* must be invertible as a linear operator, 1 is *not* an eigenvalue of the operator g*, and g(v)=v has no nonzero solution. As all nonzero elements of our field are invertible, we have g*v=v -> g=1, which was an invalid choice for our (2nd) operator. Thus finally, (I+g*) is invertible, and our system implies Dj=0, which then by our original system Di+Dj=0 gives Di=0. This solution is actually general, I just used the easiest choice of size, suitable finite field, and generator I came up with.

I'm sure there's a MUCH easier way of explaining this, but the upshot here is that (I+Shift^n) is NOT an invertible operator ever in our setup, since as you've seen Di = {11111111} is an eigenvalue, satisfying v=Shift^n(v). However, in an irreducible polynomial field of degree n over Z2, any generator g of the aforementioned field gives a proper set of equations, and any power of g up to 2^n-1 gives a unique operator to apply to each drive, hence D0 + g*D1 + g^2*D2 +... that you see in the next wiki section. Since g (and thus any power of g not equal to 1) is cyclic and thus loops back to 1 and itself by applying powers of g, the above proof works with any g^i and g^j, requiring that (g^(i-j)-I) is invertible for each power, which checks out by the same method. Ok, I apologize, but I figured I might as well post this instead of erasing it. 174.134.36.149 (talk) 07:01, 18 April 2022 (UTC)

At the end of the example: A xor B = D3 xor (D3 rotate-shift 3). So, will be a 4 bit HDD: the bits are abcd. So, we known a xor d, b xor a, c xor b, d xor c. The xor of the starting 3 is the last: d xor c. So, we have only 3 bit information, but the capacity of the HDD is 4 bits. By the way, the example can not work on 2+2 and 3+2 drives so. So, the example is wrong, must be deleted. X00000 (talk) 09:07, 7 July 2022 (UTC)

At the end of the example: A xor B = D3 xor (D3 rotate-shift 3). So, will be a 4 bit HDD: the bits are abcd. So, we known a xor d, b xor a, c xor b, d xor c. The xor of the starting 3 is the last: d xor c. So, we have only 3 bit information, but the capacity of the HDD is 4 bits. By the way, the example can not work on 2+2 and 3+2 drives so. So, the example is wrong, must be deleted. X00000 (talk) 17:51, 14 July 2022 (UTC)

why was my article section removal reverted? i thought the result of this talk section was, that the section in the article shalt be deleted... --2003:D7:DF0B:AA00:76D4:35FF:FE54:CF0D (talk) 20:31, 10 September 2022 (UTC)

I agree with all of the above. A simple bit-shift (actually a rotate) cannot be used as the generator. The statement in the article "On a bitwise level, this represents a system of $2k$ equations in $2k$ unknowns which uniquely determine the lost data" is flawed in that it does not consider the possibility that the $2k$ equations are not independent, which is in fact the case for a simple bit-shift/rotate.

For example, for k = 8 and lost data $D_{0}$ and $D_{1}$ and calculation A and B as described, you get the following equations:$${\begin{aligned}A&=D_{0}\oplus D_{1}\\B&=shift^{0}(D_{0})\oplus shift^{1}(D_{1})\end{aligned}}$$

At the bit level where:$${\begin{aligned}D_{0}&=[x_{0},x_{1},x_{2},x_{3},x_{4},x_{5},x_{6},x_{7}]\\D_{1}&=[y_{0},y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7}]\end{aligned}}$$

This expands to something like:$${\begin{aligned}x_{0}\oplus y_{0}=a_{0}\\x_{1}\oplus y_{1}=a_{1}\\x_{2}\oplus y_{2}=a_{2}\\x_{3}\oplus y_{3}=a_{3}\\x_{4}\oplus y_{4}=a_{4}\\x_{5}\oplus y_{5}=a_{5}\\x_{6}\oplus y_{6}=a_{6}\\x_{7}\oplus y_{7}=a_{7}\\x_{0}\oplus y_{1}=b_{0}\\x_{1}\oplus y_{2}=b_{1}\\x_{2}\oplus y_{3}=b_{2}\\x_{3}\oplus y_{4}=b_{3}\\x_{4}\oplus y_{5}=b_{4}\\x_{5}\oplus y_{6}=b_{5}\\x_{6}\oplus y_{7}=b_{6}\\x_{7}\oplus y_{0}=b_{7}\end{aligned}}$$

If you combine the first 15 equations, the result matches the last. Thus the equations are not independent and cannot be uniquely solved. A more suitable generator such as a Linear-feedback shift register mentioned in the general section avoids this issue.

Since the section does not reference any sources supporting the use of simple shift/rotate operation as the generator, I expect this may have been some original research which was both incomplete and flawed, and I support its removal. -- Tom N talk/contrib 05:29, 11 September 2022 (UTC)

@User:Tcncv: ok... do we need user:zac67's written consent? or can we just delete it a 2nd time? --2003:D7:DF0B:AA00:E6DF:7080:942A:9711 (talk) 17:32, 11 September 2022 (UTC)

it started there: [2] --2003:D7:DF0B:AA00:E6DF:7080:942A:9711 (talk) 18:05, 11 September 2022 (UTC)

You can delete it again. Be sure to reference this talk page discussion and apparent consensus in your edit cummary. -- Tom N talk/contrib 22:34, 11 September 2022 (UTC)