Object storage

Object storage treats data as discrete units called objects, each with a unique identifier and rich metadata. Unlike file systems, objects are stored in a flat address space, often called a bucket or container, eliminating the need for hierarchical directories. This design enables horizontal scaling across distributed clusters, as objects can be spread across many nodes without the constraints of a single namespace. Metadata can include custom attributes, such as access policies and retention rules, which are stored alongside the data, allowing for efficient indexing and management. The architecture is particularly suited for unstructured data, such as multimedia files, backups, and IoT sensor data, where the volume and variety exceed the capabilities of traditional storage.

The concept of object storage emerged from research in the 1970s, with early systems like the UCLA LIS and the Carnegie Mellon University's Coda file system exploring object-based approaches. In 1995, the Object Management Group published the Object Database Standard, but it was the rise of cloud computing that propelled object storage into the mainstream. Amazon S3, launched in 2006, became the de facto standard for cloud object storage, offering a simple RESTful API and virtually unlimited scalability. Since then, open-source implementations like OpenStack Swift and Ceph have made object storage accessible for private clouds and on-premises deployments. The development of the S3 API as a common interface has led to widespread compatibility across vendors, from AWS to MinIO and Google Cloud Storage.

Object storage is widely used for backup and archiving, where its durability and cost-effectiveness are critical. It is also the backbone of big data analytics, enabling the storage of massive datasets for processing by frameworks like Apache Hadoop and Spark. Media repositories, such as video streaming platforms, rely on object storage to serve content at scale. In healthcare, object storage supports the long-term retention of medical imaging, such as MRI scans, with compliance requirements like HIPAA. The financial sector uses object storage for regulatory data retention, and scientific research institutions store petabytes of experimental data. The technology's ability to handle immutable data and support versioning makes it ideal for compliance and audit trails.

Object storage has several niche applications and technical nuances. For example, it supports erasure coding, a data protection method that uses less overhead than traditional replication, making it efficient for large-scale systems. Some object storage systems offer S3-compatible interfaces but with unique features, such as MinIO's support for single-node deployments. The term 'object' can also refer to the Object Management Group's CORBA, but in storage, it is distinct. A lesser-known fact is that the first object storage system was developed at the University of California, Berkeley in the 1980s, called the Object-Based Storage Device (OSD). Additionally, object storage is often used for cold storage, where data is accessed rarely but must be preserved for years, leading to the development of tape-based object storage solutions. The S3 API has become so ubiquitous that it is now a standard for interoperability, with many vendors offering S3-compatible endpoints.

Object

A discrete unit of data in object storage, consisting of the data itself, metadata, and a unique identifier.

Bucket

A container or namespace for objects in object storage, similar to a directory but flat.

Metadata

Descriptive information about an object, such as creation date, size, and custom attributes.

Erasure coding

A data protection method that splits data into fragments and distributes them across nodes, allowing reconstruction even if some nodes fail.

S3 API

A RESTful API for object storage, originally defined by Amazon S3, now widely adopted as a standard interface.