I'm going to try XFS deduplication for Synapse and Peertube and hopefully it'll save a bit of space

@matrix You can do

`duperemove -hdr --hashfile=/dedupe-hash.db --dedupe-options=same,block /mnt/storage`

First pass will be slow af but it'll build an sqlite database at /dedupe-hash.db so in the future only new files will need to get hashed and deduped.

Replace /mnt/storage with your mount point.

@sjw Thanks. I know, I followed the article you linked me a while. Now I'm waiting for Peertube rsync to transfer the files. Servers have 1Gbit so it should be done soon. The RAID still hasn't synced yet though so that's gonna slow down the dedupe.

@matrix here's about 30 hours of savings. Took about 45 seconds to run.
I still need to setup a cronjob for it...
Screenshot_20220103-133648.png

@sjw
Deduplicating juice Peertube saved 12.6GB out of 1.4TB, but that's to be expected. I'm expecting a bigger impact once I move Synapse

@matrix Yeah, if you do transcoding the audio tracks will get deduped at least

@sjw With Synapse it saved 35GB from 400GB, which is quite decent.