File synchronization

File synchronization relies on detecting differences between file versions. The simplest approach compares modification times and file sizes, but this can miss changes when clocks are skewed or when files are modified without updating timestamps. More robust methods use cryptographic hashes, such as MD5 or SHA-1, to compare content, though hashing entire files is expensive for large datasets. The rsync algorithm, developed by Andrew Tridgell and Paul Mackerras in 1996, uses a rolling checksum to identify blocks that have not changed, allowing efficient transfer of only the differing portions over a network¹. For bidirectional synchronization, tools often maintain a local database of file states, as seen in Unison, which uses a replica structure to track versions and resolve conflicts based on user-defined policies².

When the same file is modified in two locations, synchronization must decide which version prevails. Simple tools adopt a last-writer-wins policy based on timestamps, but this can lose updates. More sophisticated systems use version vectors or operational transformation to merge changes, as in collaborative editing tools like Google Docs. Unison, a bidirectional file synchronizer, detects conflicts and prompts the user to choose, or can be configured to prefer one replica. Cloud services like Dropbox create conflict copies (e.g., "file (conflicted copy)") to preserve both versions³. Consistency models range from eventual consistency, where replicas converge over time, to strong consistency, where all replicas are updated atomically, the latter being harder to achieve in distributed systems.

File synchronization is integral to many domains. In software development, version control systems like Git synchronize repositories across machines, though they focus on changesets rather than raw files. In enterprise settings, tools like Microsoft's SyncToy and rsync are used for backup and mirroring. Cloud storage services—Dropbox, Google Drive, OneDrive, and iCloud—provide automatic synchronization across devices, often using proprietary protocols. For peer-to-peer synchronization, Resilio Sync (formerly BitTorrent Sync) uses a distributed hash table to sync files without a central server⁴. Mobile operating systems synchronize contacts, photos, and app data via vendor-specific frameworks, such as Apple's CloudKit.

Beyond mainstream tools, file synchronization has niche applications. In scientific computing, tools like Globus synchronize large datasets across supercomputing centers, using GridFTP for high-performance transfers. In embedded systems, synchronization protocols like SyncML (now OMA DS) were developed for mobile devices, predating modern cloud sync. A notable edge case is the synchronization of files with extended attributes or sparse files, which many tools mishandle. Also, the rsync algorithm's rolling checksum is based on the Adler-32 checksum, which is weaker than cryptographic hashes but faster. Another obscure fact: the term "sync" in Unix's sync command refers to flushing filesystem buffers, not file synchronization, a common confusion. Finally, some tools like FreeFileSync use a database of previous syncs to detect deletions and avoid data loss.

rsync

A command-line utility for efficiently transferring and synchronizing files across systems, using a delta algorithm.

Unison

A bidirectional file synchronization tool that works on multiple platforms and handles conflicts explicitly.

Conflict copy

A duplicate file created by cloud sync services when two versions of a file cannot be automatically merged.

Rolling checksum

A checksum that can be updated incrementally as a window slides over data, used in rsync's delta algorithm.