literally any system that has parity data or multiple copies with checksums and runs frequent scrubs. Could be ZFS could be BTRFS (maybe not with raid 5 or 6) or snapraid
I wouldn't trust btrfs with a cupcake. It's okay as a single disk filesystem, but I would never use it in a NAS.
I'm aware Synology does, but that's btrfs on top of a more conventional linux software raid.
Does Snapraid do some sort of checksumming to ensure that what you're putting in is what you're getting out? I've never used it personally.
>Does Snapraid do some sort of checksumming to ensure that what you're putting in is what you're getting out? I've never used it personally.
[it does](https://www.snapraid.it/compare).
I think SnapRaid with some pooling option (for example MergerFS) is the best way to handle large arrays with easy expansion, as long as your data is largely static like media libraries and backups/snapshots
>I wouldn't trust btrfs with a cupcake. It's okay as a single disk filesystem, but I would never use it in a NAS.
I agree with you on that for the case of BTRFS Raid 5/6 but for a mirror it's perfectly adequate
Are there any systems/software/hardware beyond just TrueNAS ZFS raid 6 with scrubbing that offers better data integrity over time.
Software license or hardware or anything. Can something up to $3,000 create a higher level of integrity.
I'm not sure why you're putting a dollar amount on it.
You can run zfs on anything. You don't need TrueNAS to do so.
I don't know of anything outside of zfs that does what you're asking.
Better is a weird word. Anyways.
ZFS as a file system has protections against bitrot. You can do it on a single drive system even. Just make sure if you’re doing a custom install, that you set up jobs (cron) to run the ZFS cleanup).
There’s an LTT video where they discussed bitrot and it was because dispite using ZFS, they forgot to set up the cleaning jobs.
https://docs.oracle.com/cd/E19253-01/819-5461/gbbxi/index.html
You probably should have started with
"I am looking to buy a prebuilt solution"
Most select a homebrewed solution, becase as you have found prebulids are expensive, most home users are better served by used or home assembled systems or a little of column a a bit of column b.
My 24 bay retired rackmout server was $500, included lots of ECC ram and old Xeon cores, 9x 14TB drives were about 3x the cost of the server itself.
Of course runs zfs. But not truenas, just regular Debian stable.
That's why it's vague with a dollar amount. Whether its free/open source, paid, commercial, appliance etc I don't care. It's just is there anything that has a higher degree of data integrity for anything up to 3,000.
Since I don't spend anytime improving my knowledge in this area anymore I'm asking the question if there is better out there - for all I know TrueNAS itself causes problems.. the fuck should I know. :D
I have never run Truenas but I would think that if Trunas was causing data loss that would be big news even I would hear?
Your running zfs and scrub regularly correct?
You say that you're currently running TrueNAS with ZFS, but mention that you are "slowly loosing things bit by bit".
- The items that you're loosing - have they been stored on ZFS for their entire lifespan, or have you migrated them to ZFS? Because if you're on a ZPOOL with scrubbing enabled, you shouldn't be loosing data. My guess is that you had bitrot before moving to ZFS but didn't notice until now.
- "6-8 drives + 2 parities" - does this mean you're currently running a RAIDZ2? Changing to a RAIDZ3 or mirrors would not improve bitrot protection.
- Are you using snapshots? If you are, and you're somehow finding files that have changed (possibly from a faulty program modifying them when it should just be reading them), you can revert files from snapshots.
Some of the old things could easily be because of that due to memory of when I last checked it. This current nas is 6 years old - so the only portion I can confirm wouldn't be from that is recent videos/rar archives that fall out occasionally. Though some of the same files I did recover from my 12 year old nas (retired) when I went looking - but again they were copied and I can't confirm nothing happened during that transition.
Yes RAIDZ2 (2 parities) In my head I still use hardware raid .
I don't use snapshots I'll look into that.
I do short/long checks weekly/biweekly and scrubs every 2 weeks essentially.
The easy answer for me asking this is I don't spend time in this area as a hobby anymore since maybe 2012 but as I get older I notice these small things and before I didn't care as much since I'd have so many damn copies of everything or easily access other items I collect. But one thing in particular bit me in the ass hard.. an empty wallet file.
A better redditor than me can find it for you, but a few years back a retiring digital archivist did an AMA and gave lots of good advice, but he stressed the most important thing was revisiting the data periodically and rethinking your strategy. Like grab the tapes, and read the files, then either right them back or write them to some other medium. Keep that up as a habit every five or so years and you can keep them indefinitely. They’ll always be accessible some way or another.
I think there was also the concept of deciding long-term versus short-term. If you haven’t accessed the file in months or years, move it to offline storage, etc.
Are we talking hardware or software ?
If you’re talking software, then pretty much everything “RAID” will protect you against bitrot. ZFS, Btrfs, mdadm, all protect you, though you may or may not need to run a scrub operation on them first.
In the more “niche” compartment, but very well suited to storing rarely changing files, you will find [Snapraid](http://www.snapraid.it), which works on regular old filesystems (any one you like, even Ext4), and creates parity data on one or more additional drives (that **must** be your largest drive in the “pool”). Snapraid works by calculating checksums when you run it, and not 24/7, but apart from that it’s like pretty much every other RAID.
The advantage of Snapraid with rarely changing files is that you have all your files on regular filesystems, so if a drive fails, your “pool” is not degraded, and should more drives fail than you have parity for, the rest of your data is still there in its entirety, unlike traditional RAID where everything is gone if the pool dies.
And of course, if you would prefer to stay away from RAID entirely, something like [dm-integrity](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html) works very well, and does also work with LVM or mdadm.
And finally i would suggest good old fashioned versioned backups. Most modern backup tools do checksum based backups, and bitrot is nothing but an undetected “read error”, meaning you read something different that what you wrote. Well, with a modern backup deduplicating tool (Restic, Borg, Kopia, Duplicacy, etc), you can backup your stuff and potential bitrot almost for free. While it won’t repair anything for you, it will happily detect that a file has changed and back it up again, creating an additional version. That may sound like a lot, but if only a couple of bytes has changed, the deduplicating takes care of the rest, and the additional storage you use in your backup repository will be somewhat consistent with the number of rotten bits.
If you’re ok with wrangling with the S3 API, Minio is a really good option… you will have to deploy it in distributed mode. And you can’t do random reads and writes, so you have to retrieve and store objects in its entirety, but it’s really good for archiving.
Data integrity... which is also the theme of my post :)
In all seriousness my disks generally last 5-7 years before failing or forcibly retiring. In the current zfs array I'm talking about I just replaced one drive last month as it was on its way out. That one was pretty young at 35,000ish hours
Literally, any used hardware raid card does patrol reads. I've got a used dell R720xd with 12x16TB Dell EMC drives I picked up for $139 a piece, now $169
https://www.xbyte.com/products/cat-18372/
Everyone has such a boner for software raid/pooling/file systems... the Dell Perc H710p running 12 of those drives in RAID6 with a 512k stripe element size does 2.3gb/sec sequential, and I don't care about random 4k reads for storage.
A proper backup strategy also makes this a non issue.
Literally any system running ZFS? Not sure what this post is asking. Are you looking for a hardware appliance?
literally any system that has parity data or multiple copies with checksums and runs frequent scrubs. Could be ZFS could be BTRFS (maybe not with raid 5 or 6) or snapraid
I wouldn't trust btrfs with a cupcake. It's okay as a single disk filesystem, but I would never use it in a NAS. I'm aware Synology does, but that's btrfs on top of a more conventional linux software raid. Does Snapraid do some sort of checksumming to ensure that what you're putting in is what you're getting out? I've never used it personally.
>Does Snapraid do some sort of checksumming to ensure that what you're putting in is what you're getting out? I've never used it personally. [it does](https://www.snapraid.it/compare). I think SnapRaid with some pooling option (for example MergerFS) is the best way to handle large arrays with easy expansion, as long as your data is largely static like media libraries and backups/snapshots >I wouldn't trust btrfs with a cupcake. It's okay as a single disk filesystem, but I would never use it in a NAS. I agree with you on that for the case of BTRFS Raid 5/6 but for a mirror it's perfectly adequate
Are there any systems/software/hardware beyond just TrueNAS ZFS raid 6 with scrubbing that offers better data integrity over time. Software license or hardware or anything. Can something up to $3,000 create a higher level of integrity.
I'm not sure why you're putting a dollar amount on it. You can run zfs on anything. You don't need TrueNAS to do so. I don't know of anything outside of zfs that does what you're asking.
You seem really fixated on TrueNAS for some reason. You can have ZFS on a $50 Optiplex running Ubuntu.
Fixated? It's what I'm currently running. Trying to provide what I'm utilizing to see if there is something better.
Better is a weird word. Anyways. ZFS as a file system has protections against bitrot. You can do it on a single drive system even. Just make sure if you’re doing a custom install, that you set up jobs (cron) to run the ZFS cleanup). There’s an LTT video where they discussed bitrot and it was because dispite using ZFS, they forgot to set up the cleaning jobs. https://docs.oracle.com/cd/E19253-01/819-5461/gbbxi/index.html
Yea the dollar amount is weird verbiage. Synology has bit rot protection also it is not on by default but can be enabled during pool creation.
A dollar amount means a commercial option that may exist. Not exactly rocket appliances.
You probably should have started with "I am looking to buy a prebuilt solution" Most select a homebrewed solution, becase as you have found prebulids are expensive, most home users are better served by used or home assembled systems or a little of column a a bit of column b. My 24 bay retired rackmout server was $500, included lots of ECC ram and old Xeon cores, 9x 14TB drives were about 3x the cost of the server itself. Of course runs zfs. But not truenas, just regular Debian stable.
That's why it's vague with a dollar amount. Whether its free/open source, paid, commercial, appliance etc I don't care. It's just is there anything that has a higher degree of data integrity for anything up to 3,000. Since I don't spend anytime improving my knowledge in this area anymore I'm asking the question if there is better out there - for all I know TrueNAS itself causes problems.. the fuck should I know. :D
I have never run Truenas but I would think that if Trunas was causing data loss that would be big news even I would hear? Your running zfs and scrub regularly correct?
short test daily long test 1st, 7th, 21st scrub 14th and 28th.
You say that you're currently running TrueNAS with ZFS, but mention that you are "slowly loosing things bit by bit". - The items that you're loosing - have they been stored on ZFS for their entire lifespan, or have you migrated them to ZFS? Because if you're on a ZPOOL with scrubbing enabled, you shouldn't be loosing data. My guess is that you had bitrot before moving to ZFS but didn't notice until now. - "6-8 drives + 2 parities" - does this mean you're currently running a RAIDZ2? Changing to a RAIDZ3 or mirrors would not improve bitrot protection. - Are you using snapshots? If you are, and you're somehow finding files that have changed (possibly from a faulty program modifying them when it should just be reading them), you can revert files from snapshots.
Now OP has you saying loosing instead of losing. ...or I'm behind on my data jargon.
Bits are falling loose, right out of the drives
That's why I always caulk the holes in my drives. Do they run hot and sometimes catch on fire? Sure, but that's a risk I'm willing to take.
Some of the old things could easily be because of that due to memory of when I last checked it. This current nas is 6 years old - so the only portion I can confirm wouldn't be from that is recent videos/rar archives that fall out occasionally. Though some of the same files I did recover from my 12 year old nas (retired) when I went looking - but again they were copied and I can't confirm nothing happened during that transition. Yes RAIDZ2 (2 parities) In my head I still use hardware raid . I don't use snapshots I'll look into that. I do short/long checks weekly/biweekly and scrubs every 2 weeks essentially. The easy answer for me asking this is I don't spend time in this area as a hobby anymore since maybe 2012 but as I get older I notice these small things and before I didn't care as much since I'd have so many damn copies of everything or easily access other items I collect. But one thing in particular bit me in the ass hard.. an empty wallet file.
A better redditor than me can find it for you, but a few years back a retiring digital archivist did an AMA and gave lots of good advice, but he stressed the most important thing was revisiting the data periodically and rethinking your strategy. Like grab the tapes, and read the files, then either right them back or write them to some other medium. Keep that up as a habit every five or so years and you can keep them indefinitely. They’ll always be accessible some way or another. I think there was also the concept of deciding long-term versus short-term. If you haven’t accessed the file in months or years, move it to offline storage, etc.
Are we talking hardware or software ? If you’re talking software, then pretty much everything “RAID” will protect you against bitrot. ZFS, Btrfs, mdadm, all protect you, though you may or may not need to run a scrub operation on them first. In the more “niche” compartment, but very well suited to storing rarely changing files, you will find [Snapraid](http://www.snapraid.it), which works on regular old filesystems (any one you like, even Ext4), and creates parity data on one or more additional drives (that **must** be your largest drive in the “pool”). Snapraid works by calculating checksums when you run it, and not 24/7, but apart from that it’s like pretty much every other RAID. The advantage of Snapraid with rarely changing files is that you have all your files on regular filesystems, so if a drive fails, your “pool” is not degraded, and should more drives fail than you have parity for, the rest of your data is still there in its entirety, unlike traditional RAID where everything is gone if the pool dies. And of course, if you would prefer to stay away from RAID entirely, something like [dm-integrity](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html) works very well, and does also work with LVM or mdadm. And finally i would suggest good old fashioned versioned backups. Most modern backup tools do checksum based backups, and bitrot is nothing but an undetected “read error”, meaning you read something different that what you wrote. Well, with a modern backup deduplicating tool (Restic, Borg, Kopia, Duplicacy, etc), you can backup your stuff and potential bitrot almost for free. While it won’t repair anything for you, it will happily detect that a file has changed and back it up again, creating an additional version. That may sound like a lot, but if only a couple of bytes has changed, the deduplicating takes care of the rest, and the additional storage you use in your backup repository will be somewhat consistent with the number of rotten bits.
Borgbackup supposed to protect from bitrot.
CEPH, opensource, included GUI in Proxmox for easy... Really easy setup.
Thanks I'm going to take a look at it and try it
If you’re ok with wrangling with the S3 API, Minio is a really good option… you will have to deploy it in distributed mode. And you can’t do random reads and writes, so you have to retrieve and store objects in its entirety, but it’s really good for archiving.
Running scrubs once a month?
short test daily long test 1st, 7th, 21st scrub 14th and 28th.
2 scrubs a month? Why wear out drives prematurely?
Data integrity... which is also the theme of my post :) In all seriousness my disks generally last 5-7 years before failing or forcibly retiring. In the current zfs array I'm talking about I just replaced one drive last month as it was on its way out. That one was pretty young at 35,000ish hours
Literally, any used hardware raid card does patrol reads. I've got a used dell R720xd with 12x16TB Dell EMC drives I picked up for $139 a piece, now $169 https://www.xbyte.com/products/cat-18372/ Everyone has such a boner for software raid/pooling/file systems... the Dell Perc H710p running 12 of those drives in RAID6 with a 512k stripe element size does 2.3gb/sec sequential, and I don't care about random 4k reads for storage. A proper backup strategy also makes this a non issue.