Data erasure

Data erasure (sometimes referred to as data clearing, data wiping, or data destruction) is a software-based method of data sanitization that aims to completely destroy all electronic data residing on a hard disk drive or other digital media by overwriting data onto all sectors of the device in an irreversible process. By overwriting the data on the storage device, the data is rendered irrecoverable.

Ideally, software designed for data erasure should:

1. Allow for selection of a specific standard, based on unique needs, and
2. Verify the overwriting method has been successful and removed data across the entire device.

Permanent data erasure goes beyond basic file deletion commands, which only remove direct pointers to the data disk sectors and make the data recovery possible with common software tools. Unlike degaussing and physical destruction, which render the storage media unusable, data erasure removes all information while leaving the disk operable. New flash memory-based media implementations, such as solid-state drives or USB flash drives, can cause data erasure techniques to fail allowing remnant data to be recoverable.

Software-based overwriting uses a software application to write a stream of zeros, ones or meaningless pseudorandom data onto all sectors of a hard disk drive. There are key differentiators between data erasure and other overwriting methods, which can leave data intact and raise the risk of data breach, identity theft or failure to achieve regulatory compliance. Many data eradication programs also provide multiple overwrites so that they support recognized government and industry standards, though a single-pass overwrite is widely considered to be sufficient for modern hard disk drives. Good software should provide verification of data removal, which is necessary for meeting certain standards.

To protect the data on lost or stolen media, some data erasure applications remotely destroy the data if the password is incorrectly entered. Data erasure tools can also target specific data on a disk for routine erasure, providing a hacking protection method that is less time-consuming than software encryption. Hardware/firmware encryption built into the drive itself or integrated controllers is a popular solution with no degradation in performance at all.

Encryption

When encryption is in place, data erasure acts as a complement to crypto-shredding, or the practice of 'deleting' data by (only) deleting or overwriting the encryption keys.

Presently, dedicated hardware/firmware encryption solutions can perform a 256-bit full AES encryption faster than the drive electronics can write the data. Drives with this capability are known as self-encrypting drives (SEDs); they are present on most modern enterprise-level laptops and are increasingly used in the enterprise to protect the data. Changing the encryption key renders inaccessible all data stored on a SED, which is an easy and very fast method for achieving a 100% data erasure. Theft of an SED results in a physical asset loss, but the stored data is inaccessible without the decryption key that is not stored on a SED, assuming there are no effective attacks against AES or its implementation in the drive hardware.

Importance

Information technology assets commonly hold large volumes of confidential data. Social security numbers, credit card numbers, bank details, medical history and classified information are often stored on computer hard drives or servers. These can inadvertently or intentionally make their way onto other media such as printers, USB, flash, Zip, Jaz, and REV drives.

Data breach

Increased storage of sensitive data, combined with rapid technological change and the shorter lifespan of IT assets, has driven the need for permanent data erasure of electronic devices as they are retired or refurbished. Also, compromised networks and laptop theft and loss, as well as that of other portable media, are increasingly common sources of data breaches.

If data erasure does not occur when a disk is retired or lost, an organization or user faces a possibility that the data will be stolen and compromised, leading to identity theft, loss of corporate reputation, threats to regulatory compliance and financial impacts. Companies spend large amounts of money to make sure their data is erased when they discard disks. High-profile incidents of data theft include:

• CardSystems Solutions (2005-06-19): Credit card breach exposes 40 million accounts.
• Lifeblood (2008-02-13): Missing laptops contain personal information including dates of birth and some Social Security numbers of 321,000.
• Hannaford (2008-03-17): Breach exposes 4.2 million credit, debit cards.
• Compass Bank (2008-03-21): Stolen hard drive contains 1,000,000 customer records.
• University of Florida College of Medicine, Jacksonville (2008-05-20): Photographs and identifying information of 1,900 on improperly disposed computer.
• Oklahoma Corporation Commission (2008-05-21): Server sold at auction compromises more than 5,000 Social Security numbers.
• Department of Finance, the Australian Electoral Commission and National Disability Insurance Agency (2017-11-02) - 50,000 Australians and 5000 Federal Public servant records.

Regulatory compliance

Strict industry standards and government regulations are in place that force organizations to mitigate the risk of unauthorized exposure of confidential corporate and government data. Regulations in the United States include HIPAA (Health Insurance Portability and Accountability Act); FACTA (The Fair and Accurate Credit Transactions Act of 2003); GLB (Gramm-Leach Bliley); Sarbanes-Oxley Act (SOx); and Payment Card Industry Data Security Standards (PCI DSS) and the Data Protection Act in the United Kingdom. Failure to comply can result in fines and damage to company reputation, as well as civil and criminal liability.

Preserving assets and the environment

Data erasure offers an alternative to physical destruction and degaussing for secure removal of all the disk data. Physical destruction and degaussing destroy the digital media, requiring disposal and contributing to electronic waste while negatively impacting the carbon footprint of individuals and companies. Hard drives are nearly 100% recyclable and can be collected at no charge from a variety of hard drive recyclers after they have been sanitized.

Limitations

Data erasure may not work completely on flash based media, such as Solid State Drives and USB Flash Drives, as these devices can store remnant data which is inaccessible to the erasure technique, and data can be retrieved from the individual flash memory chips inside the device. Data erasure through overwriting only works on hard drives that are functioning and writing to all sectors. Bad sectors cannot usually be overwritten, but may contain recoverable information. Bad sectors, however, may be invisible to the host system and thus to the erasing software. Disk encryption before use prevents this problem. Software-driven data erasure could also be compromised by malicious code.

Differentiators

Software-based data erasure uses a disk accessible application to write a combination of ones, zeroes and any other alpha numeric character also known as the "mask" onto each hard disk drive sector. The level of security when using software data destruction tools is increased dramatically by pre-testing hard drives for sector abnormalities and ensuring that the drive is 100% in working order. The number of wipes has become obsolete with the more recent inclusion of a "verify pass" which scans all sectors of the disk and checks against what character should be there, i.e., one pass of AA has to fill every writable sector of the hard disk. This makes any more than one pass an unnecessary and certainly a more damaging act, especially in the case of large multi-terabyte drives.

Full disk overwriting

While there are many overwriting programs, only those capable of complete data erasure offer full security by destroying the data on all areas of a hard drive. Disk overwriting programs that cannot access the entire hard drive, including hidden/locked areas like the host protected area (HPA), device configuration overlay (DCO), and remapped sectors, perform an incomplete erasure, leaving some of the data intact. By accessing the entire hard drive, data erasure eliminates the risk of data remanence.

Data erasure can also bypass the Operating System (OS). Overwriting programs that operate through the OS will not always perform a complete erasure because they cannot modify the contents of the hard drive that are actively in use by that OS. Because of this, many data erasure programs are provided in a bootable format, where you run off a live CD that has all of the necessary software to erase the disk.

Hardware support

Data erasure can be deployed over a network to target multiple PCs rather than having to erase each one sequentially. In contrast with DOS-based overwriting programs that may not detect all network hardware, Linux-based data erasure software supports high-end server and storage area network (SAN) environments with hardware support for Serial ATA, Serial Attached SCSI (SAS) and Fibre Channel disks and remapped sectors. It operates directly with sector sizes such as 520, 524, and 528, removing the need to first reformat back to 512 sector size. WinPE has now overtaken Linux as the environment of choice since drivers can be added with little effort. This also helps with data destruction of tablets and other handheld devices that require pure UEFI environments without hardware NIC's installed and/or are lacking UEFI network stack support.

Standards

Many government and industry standards exist for software-based overwriting that removes the data. A key factor in meeting these standards is the number of times the data is overwritten. Also, some standards require a method to verify that all the data have been removed from the entire hard drive and to view the overwrite pattern. Complete data erasure should account for hidden areas, typically DCO, HPA and remapped sectors.

The 1995 edition of the National Industrial Security Program Operating Manual (DoD 5220.22-M) permitted the use of overwriting techniques to sanitize some types of media by writing all addressable locations with a character, its complement, and then a random character. This provision was removed in a 2001 change to the manual and was never permitted for Top Secret media, but it is still listed as a technique by many providers of the data erasure software.

Data erasure software should provide the user with a validation certificate indicating that the overwriting procedure was completed properly. Data erasure software should also comply with requirements to erase hidden areas, provide a defects log list and list bad sectors that could not be overwritten.

Overwriting Standard	Date	Overwriting Rounds	Pattern	Notes
U.S. Navy Staff Office Publication NAVSO P-5239-26	1993	3	A character, its complement, random	Verification is mandatory
U.S. Air Force System Security Instruction 5020	1996	3	All zeros, all ones, any character	Verification is mandatory
Peter Gutmann's Algorithm	1996	1 to 35	Various, including all of the other listed methods	Originally intended for MFM and RLL disks, which are now obsolete
Bruce Schneier's Algorithm	1996	7	All ones, all zeros, pseudo-random sequence five times
Standard VSITR of Germany Federal Office for Information Security	1999	7	The disk is filling with sequences 0x00 and 0xFF, and on the last pass - 0xAA.
U.S. DoD Unclassified Computer Hard Drive Disposition	2001	3	A character, its complement, another pattern
German Federal Office for Information Security	2004	2 to 3	Non-uniform pattern, its complement
Communications Security Establishment Canada ITSG-06	2006	3	All ones or zeros, its complement, a pseudo-random pattern	For unclassified media
NIST SP-800-88	2006	1	?
U.S. National Industrial Security Program Operating Manual (DoD 5220.22-M)	2006	3	?	No longer specifies any method.
NSA/CSS Storage Device Declassification Manual (SDDM)	2007	0	?	Degauss or destroy only
New Zealand Government Communications Security Bureau NZSIT 402	2008	1	?	For data up to Confidential
Australian Government ICT Security Manual 2014 – Controls	2014	1	Random pattern (only for disks larger than 15 GB)	Degauss magnetic media or destroy Top Secret media
NIST SP-800-88 Rev. 1	2014	1	All zeros	Outlines solutions based on media type.
British HMG Infosec Standard 5, Baseline Standard	?	1	Random Pattern	Verification is mandatory
British HMG Infosec Standard 5, Enhanced Standard	?	3	All ones, all zeros, random	Verification is mandatory

Data can sometimes be recovered from a broken hard drive. However, if the platters on a hard drive are damaged, such as by drilling a hole through the drive (and the platters inside), then the data can only theoretically be recovered by bit-by-bit analysis of each platter with advanced forensic technology.

Number of overwrites needed

Data on floppy disks can sometimes be recovered by forensic analysis even after the disks have been overwritten once with zeros (or random zeros and ones).

This is not the case with modern hard drives:

• According to the 2014 NIST Special Publication 800-88 Rev. 1, Section 2.4 (p. 7): "For storage devices containing magnetic media, a single overwrite pass with a fixed pattern such as binary zeros typically hinders recovery of data even if state of the art laboratory techniques are applied to attempt to retrieve the data." It recommends cryptographic erase as a more general mechanism.
• According to the University of California, San Diego Center for Magnetic Recording Research's (now its Center for Memory and Recording Research) "Tutorial on Disk Drive Data Sanitization" (p. 8): "Secure erase does a single on-track erasure of the data on the disk drive. The U.S. National Security Agency published an Information Assurance Approval of single-pass overwrite, after technical testing at CMRR showed that multiple on-track overwrite passes gave no additional erasure." Secure erase is a feature built into modern hard drives and solid-state drives that overwrites all data on a disk, including remapped (error) sectors.
• Further analysis by Wright et al. seems to also indicate that one overwrite is all that is generally required.

Even the possibility of recovering floppy disk data after overwrite is disputed. Gutmann's famous article cites a non-existent source and sources that do not actually demonstrate recovery, only partially-successful observations. Gutmann's article also contains many assumptions that indicate his insufficient understanding of how hard drives work, especially the data processing and encoding process. The definition of "random" is also quite different from the usual one used: Gutmann expects the use of pseudorandom data with sequences known to the recovering side, not an unpredictable one such as a cryptographically secure pseudorandom number generator.

E-waste and information security

The e-waste centre of Agbogbloshie, Ghana.

E-waste presents a potential security threat to individuals and exporting countries. Hard drives that are not properly erased before the computer is disposed of can be reopened, exposing sensitive information. Credit card numbers, private financial data, account information and records of online transactions can be accessed by most willing individuals. Organized criminals in Ghana commonly search the drives for information to use in local scams.

Government contracts have been discovered on hard drives found in Agbogbloshie.

See also

• Anti-computer forensics
• Crypto-shredding
• Data recovery
• Data remanence
• Electronic waste
• File deletion
• Gutmann method
• List of data-erasing software
• Physical information security
• Sanitization (classified information)
• Security, more generally