Data execution prevention

DEP marks certain memory regions as non-executable, so that if an attacker attempts to inject and run code in those areas, the CPU raises an exception and the process is terminated. On hardware-enforced DEP, the processor's NX bit (or equivalent) is used to set the execute permission on each page of memory. Software-enforced DEP, also called SafeSEH or DEP emulation, is used on CPUs without NX support and works by preventing exceptions from being handled in a way that could lead to code execution.

DEP is applied at the process level, and can be enabled for all processes, or selectively for specific executables. It is controlled by the /NXCOMPAT linker option and can be configured via the System Properties dialog or using the bcdedit command-line tool. By default, DEP is enabled for essential Windows services and for programs that opt in.

DEP was introduced with Windows XP Service Pack 2 in 2004, as part of a broader security push that also included the Windows Firewall and Automatic Updates. It was a response to the increasing prevalence of buffer overflow exploits, which had been a major vector for worms like Code Red and Blaster.

Initially, DEP was enabled only for system processes and a few critical applications, but it was later extended to all processes by default in Windows Vista and later versions. The adoption of DEP was accelerated by the widespread availability of NX-capable processors, and by 2008, most consumer CPUs supported hardware DEP. However, some legacy applications had compatibility issues, leading Microsoft to provide per-application exceptions.

DEP is not a complete defense; attackers have developed techniques to bypass it, such as Return-Oriented Programming (ROP) and Return-to-libc attacks, which reuse existing executable code rather than injecting new code. These techniques chain together small sequences of instructions (gadgets) already present in memory to perform malicious actions.

To counter ROP, modern Windows versions include additional mitigations like ASLR, Control Flow Guard (CFG), and Arbitrary Code Guard (ACG). DEP can also be disabled by an attacker if they can exploit a vulnerability to modify the DEP settings for a process. Furthermore, DEP is ineffective against attacks that do not require code execution, such as data-only attacks or those that manipulate existing code paths.

DEP has several lesser-known facets. For instance, it was originally developed as part of the Palladium initiative, which later evolved into the Trusted Platform Module (TPM) and BitLocker. The NX bit itself was first implemented in the AMD64 architecture, and Intel adopted it as XD (Execute Disable) bit.

DEP also has a subtle interaction with Just-In-Time (JIT) compilers, which need to generate executable code at runtime; they must explicitly allocate memory with execute permissions, which can be a security risk. Additionally, DEP can be bypassed on some ARM processors due to differences in how they handle execute permissions. Finally, the Windows kernel itself uses a form of DEP called Kernel DEP to protect its own memory, and it is always enabled.

NX bit

A CPU feature that marks memory pages as non-executable, used by hardware-enforced DEP.

Buffer overflow

A type of vulnerability where data is written beyond a buffer's boundary, potentially allowing code execution.

Return-Oriented Programming (ROP)

An exploit technique that chains existing code snippets to perform malicious actions without injecting new code.