Talk:3 GB barrier

Archives (Index)

2017/November

This page is archived by ClueBot III.

Suggestion

This article looks rather about memory limitation in general.

I suggest renaming the article, making the 3Gb barrier a section in it, or removing the content, which is not directly about the 3Gb limitation. As far I know, this 3Gb memory limitation exists only for 32-bit MS Windows OS (am I right?). If it is so, I suggest making this clear in the article. — Preceding unsigned comment added by AlexanderVK (talk • contribs) 08:37, 26 December 2016 (UTC)

Yes to it needing a rewrite and/or the scope/title should be reconsidered or just merged into a broader scope. Yes just 32-bit OSes but not just MS. User:Jeh Rather than obscuring the individual aspects with language like "It is commonly claimed.." it should say the obvious that 32-bit addressing goes up to 4GB, 36-bit like PAE etc to... but keeping this "commonly claimed" just obscures those hard facts [1]. The article still conflates rather than separates and explains the different aspects of this hardware address space issue. Widefox; talk 20:50, 12 December 2018 (UTC)

The word "memory" as switched in by Guy Macon is misleading. "Memory" could mean virtual memory. A wording such as "physical address space" is necessary for precision.

The claims in the references are false, because they blatantly say that ~"32-bit processors can only address 4 GB RAM", without qualification (e.g. "unless PAE is in use"), even though they were written years after PAE was introduced on the Pentium Pro and later CPUs. I could find a dozen more such references without looking hard.

In any case, the "bit width" of a processor has no direct relationship to the physical address space it supports. The industry history includes a great many machines for which the "bit width" does not match the width of physical addresses. I wonder if the myth persists only among those who have experience only with x86?

On the other hand, anyone who wants to add text concerning other operating systems, or find a "broader scope" to merge this into, is of course welcome to do so. But it'll take more than "someone should do this". Jeh (talk) 22:11, 12 December 2018 (UTC)

I agree with @Widefox. Also with @Guy Macon. Memory is RAM aka Primary Storage. Virtual Memory is pagefiles on a hard drive, AKA Secondary Storage. There is a clear & distinct difference between those two mediums. @Jeh - what is misleading here is to lump in Virtual Memory with Memory so that there appears to be no distinction between the two to an average reader. I.e non technical.

Also, the "Pushing the Limits": blog on Technet which is so often sourced is in itself misleading & not a reliable source. That blog uses two different chipsets to illustrate memory limitations of Windows. First screenshot is of a an Intel P965 laptop with an IA-32 CPU with a a physical limitation of 4GB - Mark's old laptop. The second screenshot below,is an AlienWare Mobile WorkStation with a Xeon installed. And I believe it was Alex Ionescu's machine. I really doubt Mark R even wrote the blog it's so full of holes..

Xeon up until ~2010 was the only CPU Intel had which could access over 4GB of main memory. Claiming all x86 CPU's since "Pentium Pro" had PAE capability is very vague reference. Also wholly incorrect. Xeon was the only one - it had a 36bit EXTERNAL address bus. Also regarding PAE - Intel's official definition from a very old whitepaper, is PAE is any CPU with more than 32 line EXTERNAL address bus. Only Xeon had this. The other in Intel's stable had 32 bit EXTERNAL bus for addressing, though a few had 64 internal.... These were known as IA-32e. IA-32e supported paging, therefore could count to over 4GB if hard drive space was included (pagefile). IA-32 on the other hand DOES NOT support paging.

Also, videocard on Mark's laptop was onboard, meaning the VRAM was sliced of the main memory. That's where the extra 512MB went. A discrete GPU with discrete VRAM does not play any part in how much memory is left over for an OS because GPU & CPU have separate address spaces. At least at the time the Technet blog which is often refernced here was written.

So, there are my reasons for supporting both Widefox & Guy Macon's edits.

Maybe we should just delete this page altogether?? Thoughts? — Preceding unsigned comment added by 122.61.44.242 (talk) 15:08, 21 July 2019 (UTC)

"IA-32 does not support paging"? Wrong. Paging was introduced in the first IA-32 processor, the 80386. Paging OSes such as SunOS, Linux, and Windows NT ran on 386-based machines. What "IA-32e" - Intel's version of the originally-AMD-designed x86-64, now known as "Intel 64" - aadded was 64-bit registers and 64-bit virtual addresses (although not all bits could be used), and support for more than 36 bbits of physical address. The 64-bit virtual addresses are why IA-32e processors could have more than 44GB of virtual address; IA-32 processors could have at most 4GB of virtual address. An IA-32 processor with more than 32 bits oof physical address (requiring PAE) could have more than 4GB of physical memory, but a given process's virtual address can aat most refer to 4GB of that memory at any given instant (although, if the OS permits that, it could map regions into and out oof that virtual address space over time). Guy Harris (talk) 18:13, 21 July 2019 (UTC)

@Guy Harris. You are correct in that IA-32 does indeed support paging. Allow me to clarify: Some IA-32 processors don't support pagefile for 64bit addressing (starting w/ W2K) - just 32bit flat mode, therefore won't run Windows x64 at all. It depends on register count & internal data width & so on... In my opinion it's important to note this..

EM64T is not the original name for Intel64. Intel64 is the name of Intel's 64bit processor architecture from Core 2 Duo onward. IA-32 is the name of Intel's 32bit processor architecture prior to Core 2 Duo.

EM64T is simply IA-32 in extended mode (32e) - Long Mode....which allows 64bit addressing. The increased address space doesn't have to be in RAM, it could be (and often is) the pagefile. Xeon boards supported 8GB RAM. I would say PAE in this case is Physical Address Extension IA-32e running on hardware limited to 4GB RAM can run in extended mode - if the page file was enabled. In this case, PAE would be Page Address Extension. I feel a clear distinction should be made between the two in articles covering PAE & x86 etc.....

In this Dell whitepaper on EM64T from 2004 Page PAE & Physical PAE/EM64T are used in two contexts. https://www.dell.com/downloads/global/vectors/2004_em64t.pdf (page 3) What do you think? Onzite. (talk) 03:43, 17 May 2020 (UTC)

Unless by "pagefile" you mean something other than the file used by various versions of Windows to store anonymous pages, e.g. pagefile.sys, that's purely a software construct, so it makes no sense to say that "Some IA-32 processors don't support pagefile for 64bit addressing". What you mean is "32-bit x86 processors don't support 64-bit addressing".

The only processors that support 64-bit addressing are processors that support x86-64; those processors also support the 32-bit version of x86, and even the 16-bit version, all the way back to real mode. I have no idea what you mean by "IA-32 processor", but no 32-bit x86 processor, whatever you want to call those processors, supports 64-bit addressing at all.

What is "IA-32 in extended mode", and in what fashion does it differ from x86-64? As Intel's FAQ on EM64T explains, "Since Q4 of 2006, all mobile, desktop, and server processors based on the Intel® Core™ Microarchitecture have supported Intel® EM64T." That translates to "starting with the Core 2 processors, and server processors that use the Intel Core microarchitecture, our processors support EM64T", so processors that support EM64T support Intel64. (Note that the first "Intel Core" microprocessors didn't have the Intel Core microarchitecture.)

"The increased address space doesn't have to be in RAM, it could be (and often is) the pagefile." Yes, that's called "demand paging", and dates back to at least the Atlas machine of the early 1960's. The backing store might be a file, or it might be multiple files (as on macOS), or it might be a "swap partition", or....

What PAE does is expand the size of the page table entry and add another level to the page table hierarchy. Most of the added bits in the page table entry are either used to extend the physical address of the page or reserved (must be zero). The uppermost bit, however, is used to prohibit code fetches from the page, to support W^X.

If you have a 32-bit x86 processor that supports PAE but has only 32 address pins, the only reasons to enable PAE would be 1) if you want to simplify the OS paging code and omit support for non-PAE processors or 2) to support W^X on processors that support it (not all processors that support PAE support the no-execute bit).

Paging in long mode has similar page table entries to paging in PAE mode, but adds yet another level to the page table hierarchy, and, at least in newer processors, uses some of the reserved bits below the no-execute bit for a protection key (and marks bits 58:52 as "ignored" rather than "reserved").

5-level paging, which is apparently implemented on some recent processors, although it's not yet documented in the main Intel architecture documentation, adds another level to the page table.

The Dell white paper only talks about "Legacy PAE mode", and doesn't speak of "page PAE" and "physical PAE", it just speaks of "legacy PAE", referring to the physical address extension supported by some 32-bit processors and by 64-bit processors in legacy mode. They talk about the Windows "Address Windowing Extensions", which were a hack in Windows to support more than 4GB of physical memory on 32-bit machines, by making the additional memory a pool of non-pageable memory, parts of which could be mapped into or out of the address space as needed.

However, if you're not running Windows, there are other ways to use that memory on 32-bit processors with PAE. Unix-like operating systems with PAE support on 32-bit processors generally just treated the additional memory as part of the pool of page frames for paging, although they may have had to use special tricks, such as copying data to and from those pages into bounce buffers in the first 4GB of physical memory, to do I/O to those pages, if a peripheral didn't support 36-bit physical addresses when doing DMA. just refers to a technique that can be used on OSes that allow user-mode code to map regions into or out of the address space to access more than 4GB of data by mapping parts of that data into the address space when needed and, if that data is not currently needed and some other data must be mapped into the address space, mapping the not-currently-needed data out of the address space. Similar techniques were used, for example, in PDP-11 operating systems to support more than 64KB of code or data.

The remapping stuff, however, is independent of PAE. You can do that on a machine with an 80386, as long as the OS lets you map data in and out, as Windows NT and several Unix-like OSes did. You won't be able to keep all the data in main memory, so you may run the risk of thrashing.

The additional memory can also be used to support more processes fitting into main memory, even if those processes use no more than 4GB of code+data. Guy Harris (talk) 06:44, 17 May 2020 (UTC)

@Guy Harris: "The increased address space doesn't have to be in RAM, it could be (and often is) the pagefile." Yes, that's called "demand paging",

Wrong.

Demand Paging describes how data needed by a process is paged in from the HDD to RAM to CPU L1/2/3 on an as needed JIT basis. Demand Paging is not the Page File. A PF is an area on the drive which is used as RAM - it's used so the OS can load even though available RAM may not meet the min required to run it. Hence why the CPU traditionally boots in protected mode and hence why by default Windows initial installations have the pagefile enabled.....to prevent against boot failure. Ever try installing or loading x64 windows on a 32bit CPU or platform? It won't run. Same/similar concept as above.

But ofc rarely a problem these days anyway with 64bit hardware.

The aforementioned Page File & Demand Paging are not related to each other in any way shape or form. — Preceding unsigned comment added by 151.210.226.161 (talk) 23:05, 18 September 2021 (UTC)

The address space is the range of addresses that are considered "valid" for the code that's currently running. Some or all of those addresses refer to data in RAM; the ones that don't refer to data in RAM either refer to data in some form of secondary storage or to all-zero pages (zero-fill-on-demand).

If the CPU refers to a not-in-RAM address, a page fault occurs, and the operating system will try to find some unused RAM to store the page containing that address (it may have to remove an in-memory page to make it available for that purpose) and either read from secondary storage into that page or to zero that page out (for zero-fill-on-demand).

This is demand paging.

The pagefile (or swap partition or swap file or whatever the particular OS calls it) is one place in secondary storage from which those pages can be read. It's not "used as RAM", it's used to store pages that aren't currently in RAM; the hardware has no idea that there's any such thing as a "pagefile", it just knows that a page table entry has the valid bit set (meaning it refers to a page that's in RAM) or not set (meaning it refers to a page that's not in RAM). Pages may also be directly read from a file; for example, program executable images and shared libraries may be directly mapped into the address space, so that the pages are read from the file.

Demand paging is the mechanism that arranges that "a program can execute even though available RAM may not meet the min required to run said program." The pagefile (or whatever the OS has) is part of that mechanism; it's used to store pages that aren't in a file mapped into the address space.

And that may be less of a problem with machines with more *physical* memory, but 64-bit hardware is neither necessary nor sufficient to fix the problem. PAE (and its equivalent in some other machines, e.g. the 36-bit physical addresses provided by the SPARC Reference MMU on some 32-bit SPARC processors) allows a 32-bit machine to have more than 2^32 bytes of *physical* memory, even if an address space can't be larger than 2^32 bytes. And while, right now, my 64-bit machine with 64 GB of memory has, according to Activity Monitor, only 55 or so GB of memory in use, 64-bit machines I had earlier, with only 16 GB of memory, definitely were under memory pressure when heavily used.

So, for the claims that I was challenging:

"IA-32e supported paging, therefore could count to over 4GB if hard drive space was included (pagefile)." My machine, with an Intel64 (IA-32e) processor, can have over 4 GB of data in main memory (given that it has 64 GB of main memory), so its swap files aren't necessary to have over 4 GB of data in main memory, if that's what "could count to over 4GB" means.

"IA-32 on the other hand DOES NOT support paging." IA-32 processors can take page faults and read in pages from secondary storage, so they do support demand paging. What they don't support is having more than 4 GB in an address space; however, code running on an IA-32 processor, or any other processor with a 32-bit virtual address space, can, if the operating system supports it, map regions of files into and out of its address space, so it can awkwardly work around that issue if it needs to access more than 4 GB of data, and in an operating system that gives each process its own address space, it can have multiple processes running, with those processes collectively having more than 4 GB of address space. In both cases, that machine could make use of more than 4 GB of physical memory. Guy Harris (talk) 23:36, 18 September 2021 (UTC)

"A PF is an area on the drive which is used as RAM - it's used so the OS can load even though available RAM may not meet the min required to run it." No, the pagefile is used to store anonymous pages.

The true minimum of memory required to run an OS at all is the amount of memory required to hold all the wired pages - pages that must remain in memory at all times - of the OS, and still have enough memory to support paging. As those pages are wired, they don't ever reside in the pagefile. (For example, the code that handles a page fault has to be wired - if it weren't wired in memory, and required a page fault to read it in, that page fault couldn't be handled, because handling that page fault would require a page fault, and so on....)

That true minimum, however, would produce thrashing, so that, while the OS would run, it would be so painfully slow that you can't use it, as software would constantly be taking page faults, writing dirty pages out to mapped files or the pagefile and reading other pages from mapped files or the pagefile.

"Hence why the CPU traditionally boots in protected mode" "Boots in protected mode" in what sense? x86 CPUs start up in real mode, and some code in the booting process switches to protected mode. They switch to protected mode because the OS depends on protected mode in a number of ways, not all of which have anything to do with demand paging (or demand segment-loading - OSes running on 80286's, which were the first x86 processors to support protected mode, didn't do demand paging because the 286 didn't support paging).

"Ever try installing or loading x64 windows on a 32bit CPU or platform?" No, because I'm clueful enough to know that x86-64 OSes require that the CPU support the x86-64 instruction set, and 32-bit CPUs don't support that. The same applies to running SPARCv9 OSes on a SPARCv7/SPARCv8 platform, or ARM64 OSes on a 32-bit ARM platform, or....

"Same/similar concept as above." That would even apply to an OS that required all code and data to be wired in memory and that, therefore, had no pagefile/swap partition/swap files/etc., so it's not the "same/similar concept" as anything having to do with demand paging, including pagefuls/swap partitions/swap files/etc.. Guy Harris (talk) 00:17, 19 September 2021 (UTC)

And, to quote “Windows Internals Seventh Edition”, Part 1, from "Page files", in chapter 5 "Memory Management":

Page files store modified pages that are still in use by some process but have had to be written to disk because they were unmapped or memory pressure resulted in a trim. Page file space is reserved when the pages are initially committed, but the actual optimally clustered page file locations cannot be chosen until pages are written out to disk.

Guy Harris (talk) 21:49, 19 September 2021 (UTC)

"Undue weight" tag

So... what are the " ideas, incidents, or controversies" to which this article "may" lend "undue weight"?

Nobody has come up with specifics on this in nearly five years. If nobody can come up with specifics then the tag should be removed. Jeh (talk) 09:56, 7 September 2017 (UTC)

Remove it. --Guy Macon (talk) 13:42, 7 September 2017 (UTC)