Data deduplication (backups)

Backing up ten nearly identical virtual machines does not have to cost ten times the space. Deduplication keeps one copy and a lot of pointers.

The short answer

Data deduplication replaces multiple copies of data with references to one shared copy, which saves storage space, transferred data or both [1]. Proxmox Backup Server (PBS) does this by cutting backups into chunks, identifying each chunk by the SHA-256 checksum of its content and storing a given chunk once, however many snapshots use it [3]. Our assessment: deduplication is a great fit for backup targets, because backups of the same systems are full of repeats, but it is not a backup by itself and it is no free lunch on other file systems.

How it works

1. Chunks and checksums. PBS splits backup data into fixed-size or dynamically sized chunks. The same content gets the same checksum, and the checksum names the chunk file in the datastore [3].
2. Fixed or dynamic. VM disk images use fixed-size chunks, typically 4 MiB. For file backups, fixed sizes would break as soon as a file grows, so PBS uses a rolling hash (a Buzhash variant) to place chunk boundaries and keep chunks reusable [3].
3. Incremental upload, full snapshot. The client downloads the chunk list of the previous snapshot and sends only the checksum for chunks the server already has. Each snapshot still references all its chunks, so each one counts as a full backup [3][4].
4. Encryption changes the maths. For encrypted chunks, PBS hashes the plain text together with the key. Identical data under two different keys gets two different checksums and is not deduplicated across keys [3][4].
5. Pruning does not free space. Prune removes only snapshot metadata. Garbage collection (GC) then removes chunks that no snapshot references. It works in two phases, mark and sweep, and spares chunks newer than a cutoff, which is 24 hours and 5 minutes before the start if no older backup is still running. Proxmox suggests starting with a weekly schedule [5].
6. Shared chunks need checking. Because many snapshots can depend on one chunk, PBS offers verify jobs and recommends re-verifying all backups at least monthly to catch bit rot [5]. Our reading: a damaged chunk can hurt every snapshot that references it.
7. Other places, other costs. In the file system ZFS, deduplication is synchronous and the OpenZFS documentation warns against enabling it without a specific reason. It plans for at least 1.25 GiB of RAM per TiB of stored data, suggests compression first, and says the savings are not worth it unless the simulated ratio is comfortably above 2x [6]. How much you save always depends on the data [2].

A practical example

Take a 100 GiB virtual disk in 4 MiB chunks: that is 25,600 chunks. Suppose only 120 chunks change overnight (invented numbers for illustration). The next backup uploads about 480 MiB of new chunks, yet the new snapshot still lists all 25,600 chunks and restores like a full backup [3][4]. We have not documented a PBS setup in our inventory (14 to 16 Oct 2026), so this is a worked example, not a measurement.

Checklist

• Treat the dedup ratio you read as a result of your data, not as a promise [2].
• Schedule garbage collection after pruning, or space never comes back [5].
• Use one encryption key for backups that should deduplicate against each other, because different keys do not [3].
• Schedule verify jobs and re-verify old backups regularly [5].
• Simulate before you switch dedup on in ZFS, and try compression first [6].
• Keep a second copy elsewhere; deduplication stores less data, it does not protect it, so follow the 3-2-1 rule.