I have backups on a backup hard drive and also synced to B2, but I am thinking about backing up to some format to put in the cupboard.
The issue I see is that if I don’t have a catastrophic failure and instead just accidentally delete some files one day while organising and don’t realise, at some point the oldest backup state is removed and the files are gone.
The other thing is if I get hit by a bus and no one can work out how to decrypt a backup or whatever.
So I’m thinking of a plain old unencrypted copy of photos etc that anyone could find and use. Bonus points if I can just do a new CD or whatever each year with additions.
I have about 700GB of photos and videos which is the main content I’m concerned about. Do people use DVDs for this or is there something bigger? I am adding 60GB or more each year, would be nice to do one annual addition or something like that.
Hmm damn. I don’t really think cloud is the right answer for what I’m trying to do.
I disagree that formats like JPEG won’t be readable in 50 years. I feel like there would be big demand for being able to read the format even if it’s been superceded, on account of all the JPEGs that still living people have.
Maybe I get a big drive. Each year I copy over files from the last year. Every X years I swap the hard drive for a new one, copy all data.
How can I tell if individual files get corrupted? Like the hard drive failed in that section, then I copy the corrupted file to the new drive, and I’d never know. Can I test in bulk? 50k+ photos and videos so far.
Checksums. A good filesystem will do this for you, but you can do it yourself if you want.
If you sync a drive with rsync or something periodically, it’ll replace files with different checksums, fixing any corruption as you go. Then smart tests should tell you if there’s any corruption the drive is aware of. I’m sure automated backup tools have options for this.
I specifically don’t want to be touching previous files on the drive, it should be addition only. So I may need to write a script to do the checks, or compare against a mirror drive. I can do this with the right filesystem, but I’m worried that if I use a filesystem not readable by Windows then it may not be layman-proof enough.
Then I’d go with FAT on a USB, which should be plenty portable into the future. You’ll want to replace it every 5-10 years, and check on it every other year or so.
That’s about as easy to use as I can think of. Decades down the road, physical media like DVDs and tapes may be difficult to find readers for, but USB is versatile enough that someone is bound to have access. Micro SD cards may also be a good option, as long as you keep a couple USB readers around.
I have a terrible track record with USB sticks, including completely losing a stack of photos because of a USB stick.
I’m now thinking the benefits of a nice error-correcting file system probably outweigh the benefits of using a widely supported one. So I might use a pair of mirrored hard drives with SATA->USB cable, then include instructions along the lines of “plug into my linux laptop to access, or take to a computer repair show if you can’t work it out”.
I’m not talking about USB sticks, I’m talking about USB drives, like a HDD or SSD. If you want to go with flash memory, I recommend SD cards because they’re small and cheap, so keeping a few copies isn’t particularly burdensome.
I wouldn’t trust any of these options to last a long time on a shelf though. Check them every year or two and replace every 5-10 years, maybe a little longer if you buy higher quality.
That’s basically what I’m planning too. But my use case is disaster recovery, basically as a cheaper alternative to paying for hosted backup for important, but recoverable information (e.g. ripped media). Everything truly important (pictures and documents) goes to hosted backup as well.
I’m largely relying on documents explaining how to access the backups. If I pass, I expect my survivors to either figure it out themselves or hire someone who can figure it out from my documentation.
Everyone is saying to avoid flash memory. It doesn’t store well.
Another suggestion given is M-Disc, which might be a better option because then anyone should be able to throw it in a CD drive and load it without having to worry about the format of the drive and things like that. And even if CDs are not that common anymore, I think people will still know what they are and be able to find a way to load it even in 50 years. Like if someone found a cassette tape today (I know they were common much less than 50 years ago, but it’s hard to find an example since records came back into fashion). Plus M-Discs are designed for long term storage, so I could worry less about bitrot and files getting corrupt etc. They are write once so I’m not going to write over existing content.
Yeah, flash memory doesn’t store well, hence the recommendation to keep checking on it. This article claims 10 years, but I think checking on it every year or two is a good idea.
And yeah, M-Disk looks like a good option, especially if you never need to read from it. I would make multiple copies though, because disks break, get lost, etc. And get an extra drive so your successors don’t need to go find one; people are lazy and you want as few obstacles to them using it as possible.
I personally don’t like using optical media because players can be finicky and storage can be annoying. But it’s probably a good solution for your stated needs.
Seems the only M-Disc capable writer I can find locally is a portable USB-C connecting one, so if I go with it I’ll probably just store it with the discs. In theory M-Disc is supposed to be resistant to the kinds of things that destroy regular CDs, but making a second copy does sound like a good idea. I could even store the second copy somewhere else (another house) to protect against fire. I have cloud backup but you never know what’s going to happen over 50 years. Or if I die in the fire and no one knows I have the cloud backup.
Any file systems Windows can read out-of-the-box are no good file systems. What Windows read? FAT and NTFS. Former is so basic it has no mechanisms to detect errors and bitrot and the later one is a mess.
You should stick to ext4, btrfs and zfs.
If you want to make if fool-proof then add a sticker with ‘bring me to a computer shop to access my content’.
I have considered that exact message. It does seem making it easily plug and play may be out of the question if I want the error correction capabilities.
Btrfs and zfs are self-healing.
You can make a script to check for errors and autocorrection yourself but that needs at least a second hdd. On both drives are the same data and a file or database with the checksums of the data. The script then compares the actual checksums of the two copies and the db checksum. If they match -> perfect. If they don’t match the file where there are two matching checksum is the good one and replaces the faulty one or corrects the db entry, whichever is defect. That’s it. It doesn’t have to be more complicated.
Yip I think this is the setup I will want (probably both - zfs + a custom script for validation, just to be sure). Two mirrored drives. I do need to read up some about zfs mirroring to understand it a bit more but I think I have a path to follow now.
The format is the tape in the drive, or the disk or whatever.
Tape existed 50 years ago: nothing modern and in production can read those tapes.
The problem is, given a big enough time window, the literal drives to read it will simply no longer exist, and you won’t be able to access even non-rotted media because of that.
As for data integrity, there’s a lot of options: you can make a md5 sum of each file, and then do it again and see if anything is different.
The only caveat here is you have to make sure whatever you’re using to make the checksums gets stored somewhere that’s not JUST on the drive because if the drive DOES corrupt itself, and your only record of the “good” hashes is on the drive, well, you can’t necessarily trust those hashes either.
Ah good thinking. I am thinking a spare drive that I update once a year with new content and replace every few years with a new drive is a good idea.
That could probably work.
Were it me, I’d build a script that would re-hash and compare all the data to the previous hash as the first step of adding more files, and if the data comes out consistent, I’d copy the files over, hash everything again, save the hash results elsewhere and then repeat as needed.
Yeah I think I should do something like this. I really want to make sure the files are not getting corrupted in storage without me knowing.
Cloud can be surprisingly cost effective, as part of a 3-2-1 backup.
Check out storj.io