I use ZFS on all hard drives. The current setup, is that I have 2 hard drives in no raid or redundancy, just in two VDEVs in a ZFS Pool because I have more data than the capacity of one hard drive. This is then backed up irregularly to another system, where the same situation is set up, 2 VDEVs. That means, I have no redundancy on either of these systems.

Now with the addition of the new hard drives, I’m able to make the primary system at least redundant (and expand the capacity), while expanding the backup capacity too (however, still no redundancy there). As the newly purchased hard drives will go to the backup system as well as I’m changing VDEVs, I need to shuffle data around. I have a ZFS pool, called storage on the primary system.

Please note, that I’m showing the path of all hard drives below, but I actually use WWNs to make sure, that even if the path changes, I use the same hard drives everywhere.

The plan

is the following:

  1. Create a new pool, named storage_tmp with the new hard drives in mirror mode (they are large enough to accommodate all data and I’ll gain redundancy immediately) using:
    • sdh
    • sdi
  2. Create the snapshot transferon the old pool storage
  3. Send the snapshot storage@transferto the pool storage_tmp
  4. Once prepared for the change, create a new snapshot transfer_final on storage pool
  5. Stop all processes that would access data on storage
  6. Send the incremental snapshot storage@transfer_finalto storage_tmp
  7. Double check if all data is copied to storage_tmp
  8. Destroy the storage pool
  9. Create the storage pool with RaidZ2, using
    • sde
    • sdd
    • sda
    • sdf
  10. Move back the snapshot storage_tmp@transfer_final to the newly created pool storage
  11. Restart applications
  12. export the temporary pool storage_tmp

Preparation

The state of the storage pool, before starting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

tetragir@nas:~$ zpool status -L storage
  pool: storage
 state: ONLINE
config:

NAME        STATE     READ WRITE CKSUM
storage     ONLINE       0     0     0
  sde       ONLINE       0     0     0
  sdd       ONLINE       0     0     0

errors: No known data errors

Create Temporary Pool

Create the pool storage_tmp

1

tetragir@nas:~# zpool create -O atime=off storage_tmp mirror /dev/sdh /dev/sdi

Confirm that the new pool is created correctly as mirror

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

tetragir@nas:~$ zpool status -L storage_tmp
  pool: storage_tmp
 state: ONLINE
config:

NAME         STATE     READ WRITE CKSUM
storage_tmp  ONLINE       0     0     0
  mirror-0   ONLINE       0     0     0
    sdh      ONLINE       0     0     0
    sdi      ONLINE       0     0     0

errors: No known data errors

Transfer data from storage to storage_tmp

Create the transfer snapshot

1

tetragir@nas:~# zfs snap -r storage@transfer

Start the copy of the storage@stransfer snapshot to storage_tmp

1

root@nas:~# zfs send -R storage@transfer | zfs recv -Fvu storage_tmp

I’m using the flags:

  • zfs send -R to recursively send all snapshots before this snapshot as well as sending all snapshots recursively
  • zfs recv -F to force a rollback on the destination (this is unnecessary in this case. However, if there would be an older snapshot received on the destination and the data woul’ve been changed, this would roll back the dataset)
  • zfs recv -v to show the progress of the transfer
  • zfs recv -u to not mount the filesystem after receiving it

Once I’m ready to the last data transfer, stop all applications that might use the data. Then create a new snapshot and send an incremental backup with ZFS send.

1
2

root@nas:~# zfs umount storage
root@nas:~# zfs snap -r storage@transfer_last

Send the incremental

1

root@nas:~# zfs send -RI storage@transfer storage@transfer_last | zfs recv -Fvu storage_tmp

The flag -I (capital i) defines the 2 snapshots, in between all snapshots will be replicated. In the above command, ZFS will send all snapshots that were created since the creation of the @transfer snapshot until (and including) @transfer_last.

Make sure, that the received data is correct

1

root@nas:~# zpool scrub storage_tmp

Prepare the new storage pool

Double check that you have all data, then destroy the old pool. This is irreversible and will delete ALL data on the old pool! Always triple check!

1

root@nas:~# zpool destroy storage

Create the new pool

1

zpool create -f -O atime=off -O compression=zstd -o autoexpand=on storage raidz2 /dev/sda /dev/sdd /dev/sde /dev/sdf

I use the flag -f, because not all my disks are the same size

Send the data back from storage_tmp to storage

Send back all the data to the storage pool

1

root@nas:~# zfs send -R storage_tmp@transfer_last | zfs recv -Fv storage

Once the transfer is complete, double-check if the is copied back. The filesystems should be mounted, but if not, mount them.

1

root@nas:~# zfs mount storage

Export storage_tmp to make sure, that it’s not used anymore.

1

root@nas:~# zpool export storage_tmp

Restart applications and check if they are up and running.

Summary

With this method, I was able to move data temporarily to a different pool, change my primary pool and move data back.

ZFS is awesome.