Scope/Description
This article will walk through the process of recovering a ZFS storage pool from a backup and re-enabling replication task afterwards
Scenario:
- Initial setup of two servers, one "primary" and one "backup"
- primary -> 192.168.123.121
- ZFS auto-replication setup between the two server such that "backup" is a copy of "primary"
- "primary" experiences massive failure where pool is unrecoverable
- Users rebuilds an empty pool on "primary" and wants to restore from latest snapshot stored on "backup"
Prerequisites
- A backup server that has the latest snapshots sent from "primary"
- A new zpool created on "primary"
Steps
Disable Znapzend on Primary Server
- Temporarily disable znapzend service on "primary" until pool has been restored from backup
Recovering Snapshot on Backup Server
- Find the latest backup snapshot on the "backup" server. Snapshots are organized per-dataset and the time stamp will indicate which is the latest
- Make sure "backup" can ssh into the "primary" without prompting for a password
ssh-keygen (You'll be prompted with 3 questions, just press enter on all 3 until you're returned to the command prompt)
ssh-copy-id root@primary (You will be asked if you want to continue connecting, say "yes" and then enter the root password.)
- In a terminal use zfs send/receive to send snapshot back to the "primary" from the "backup"
zfs send esxitank/backup@2021-04-19-120000 | mbuffer -s 128k -m 2G | ssh 192.168.123.121 zfs recv tank/storage
- This will output a progress bar to get an idea at how fast your data is transferring back to primary
# zfs send esxitank/backup@2021-04-19-120000 | mbuffer -s 128k -m 2G | ssh 192.168.123.121 zfs recv tank/storage
in @ 112 MiB/s, out @ 112 MiB/s, 8994 MiB total, buffer 100% full
- Go back to primary and re-enable replication job
Restarting Snapshot Replication on Primary Server
- If you have the "znapzend" config saved you can re-import with:
znapzendzetup import --write tank/storage znapzend-tank-storage.backup
- If not recreate a new znapzend replication task
znapzendzetup create --recursive --mbuffer=/usr/bin/mbuffer --mbuffersize=5G SRC '7d=>1h' tank/storage DST:a '7d=>1h' root@192.168.185.10:esxitank/backup
- Re-enable znapzend service
Verification
- Once transfer has begun, go back to primary and verify the dataset was created and the Used Space is increasing
- Once transfer is complete, verify Used capacity and navigate in the dataset that all files are as expected
Troubleshooting
- If the zfs send | zfs recv command fails, ensure that the dataset you are recreating IS NOT created already on the primary end
