Kernel (operating system)

The kernel manages process scheduling, memory allocation, and inter-process communication, ensuring that multiple applications can run concurrently without conflict. It also handles device drivers, file systems, and network protocols, abstracting hardware details from user programs. The kernel operates in a privileged CPU mode (e.g., ring 0 in x86), allowing it to execute privileged instructions that user processes cannot. This separation is enforced by hardware, preventing user programs from directly accessing hardware or corrupting the system.

Kernel designs vary: monolithic kernels (e.g., Linux, BSD) include most services in a single large binary, offering high performance but requiring careful maintenance. Microkernels (e.g., Minix, QNX) minimize the kernel to basic IPC and scheduling, moving other services to user space, which improves modularity and fault isolation but can incur performance overhead. Hybrid kernels (e.g., Windows NT, macOS) blend both approaches, running some drivers in kernel space for speed while keeping others in user space for stability.

The first kernels emerged in the 1960s with systems like the Atlas supervisor and IBM's OS/360, which introduced multiprogramming and memory protection. The UNIX kernel, developed by Ken Thompson and Dennis Ritchie at Bell Labs in 1969, established a clean separation between kernel and user space, influencing virtually all subsequent designs. The 1980s saw the rise of microkernel research, with Mach and its successors, though performance issues limited adoption. Linux, created by Linus Torvalds in 1991, became the dominant monolithic kernel for servers and embedded devices, while Windows NT's hybrid kernel, designed by Dave Cutler, became the basis for modern Windows versions.

Real-time kernels, such as those in VxWorks and FreeRTOS, prioritize deterministic response times for embedded systems, often using preemptive scheduling and priority inheritance. Exokernels and unikernels represent more radical approaches, exposing hardware directly to applications or compiling the kernel with the application for cloud deployment.

The kernel is the primary trust anchor of an operating system; a compromise of the kernel grants an attacker full control over the system. Modern kernels implement numerous security mechanisms: address space layout randomization (ASLR), stack canaries, and control-flow integrity to mitigate exploits. Capability-based security, as in seL4, provides fine-grained access control, while mandatory access control (MAC) systems like SELinux and AppArmor restrict process privileges even for root. Kernel modules, which extend functionality, are a common attack vector, leading to signed module enforcement in many systems.

Microkernels offer better isolation because a failure in a user-space service does not crash the whole system, and they have been formally verified in the case of seL4, proving correctness of its implementation. However, the performance cost of IPC in microkernels has historically limited their use, though modern implementations like seL4 and L4 have reduced overhead significantly.

Kernel development has its own culture and practices: the Linux kernel mailing list (LKML) is a notoriously high-volume forum where patches are reviewed and merged by maintainers. The kernel also includes a built-in test framework, KUnit, and a debugger, KGDB. Some kernels are designed for specific niches: the L4 microkernel family is used in mobile phones (e.g., in Apple's iOS as a hypervisor), and the seL4 kernel is used in military and aviation systems due to its formal verification. The concept of a 'kernel' also extends to other domains, such as database kernels and GPU kernels, but in operating systems, it remains the essential layer that makes computing possible.

Notable edge cases include the 'kernel panic'—a fatal error state—and the 'blue screen of death' in Windows. The first kernel to be formally verified was seL4 in 2009, a milestone in systems security. Additionally, some kernels run in user space, such as the 'User-Mode Linux' variant, which allows a Linux kernel to run as a process on another Linux system, useful for debugging and virtualization.

Monolithic kernel

A kernel where all OS services run in kernel space, offering high performance but larger code base.

Microkernel

A minimal kernel that provides only essential services, with others in user space, enhancing modularity and fault isolation.

Hybrid kernel

A kernel that combines aspects of monolithic and microkernel designs, balancing performance and stability.

Privileged mode

A CPU execution mode that allows access to all hardware instructions, used by the kernel.

System call

An interface through which user programs request services from the kernel.