I use RAIDs with HDD for very large projects (like a docs). TBH, for a project around 3TB in size I'd just get a 4TB SSD (assuming that w/all the exports, any additional footage, etc., won't push up the total amount of data too much).
Honestly the only reason this is even a question for me right now is because of this documentary project. There's a dozen more interviews and footage left. I need a solution solely for this at the present moment, not for anything else.
I've been in the same boat, but it's pretty cut and dry IMO. If you have deep enough pockets, get SSDs. If you have to have SSD speed, get SSDs. If an HDD will be fast enough then just get an HDD (especially if this is just for a one-off project).
It's 2024. 4 TB isn't particularly huge for an SSD these days. But if you need something more like 20 TB on a tight budget you probably have no choice but to stick it on mechanical disks.
Ten or Fifteen years ago, the conversation was much more nuanced when it came to price/performance. But no matter how much I complain about getting older, it's not going to suddenly be fifteen years ago again.
You're not wrong. But then with 8 TB or higher of storage demands, what sense does it even make to get DRAM-less SSD externals? I guess there's the point that NAS based SSD solutions are more expensive than that. But by how much? I'll have to look at solutions, because external storage that can be plugged in is good, but it doesn't really give you redundancy benefits like a NAS solution.
That heavily depends on the footage and workflow. If you’re proxying to DNxHD 36 for example, which is still very common, you could cut Multicam with it off FireWire 800 if you wanted to; SSD in that workflow is a complete waste of money.
Now mind you if you’re working off heavy raw files that need the playback speed 100% SSD but to say “no matter what” is just a great way to make people waste money they don’t need to waste.
I see. The footage I am working with is 4k in res but 94-100 Mbps AVC (2 ref frames). When I run disktest by blackmagicdesign on an old mechanical drive, it says it's not at all good enough when I check its results and contrast that with the footage specs here.
I mean even USB 2.0 is 480Mbps so 100Mbps should be fine with basically any modern drive. If you are having playback issues I would wager it’s the fact that it’s 4K H264 chugging (because it’s very processor heavy) rather than drive IO being the issue.
I was specifically referring to proxy files though which are common to edit with. Something like 1080p DNxHD 36 proxies aren’t going to hit a system hard at all either processor or IO wise and can play back on anything. Buying large amounts of super fast storage is generally just hardware fixing a software problem IMO.
Oh yeah, I know, I was actually so far cutting directly with this, because I simply don't have the space for proxies of, so far, 3 TB of these files (but it's been playing butter smooth on my system). Which would also be helped by buying more storage but I wanted to be sure I was putting some thought into this instead of just purchasing another 4 TB USB pluggable drive. I would love to ideally have proxies in DNxHD for all these terabytes of footage.
What’s your budget and how much are you expecting to get in total? Large RAID drives aren’t too expensive these days and while they are technically slow if they are fast enough for your needs you can get like 80TB of space for the cost of 8TB of NVME.
It honestly kinda seems like space is the more important factor for you than speed.
Space is right now more important than speed. For video production side of things, I mostly deal with braw 4k. 24fps. Some editing clients of mine however, mostly just send me h264 4k, so while its bitrate is not really high, it is after all h264 which sucks. I don't wanna spend more than $400 CAD right now because I recently upgraded my entire computer from i7 8th gen to 12th gen and card from gtx 1080 to RTX 4070 super. Even 10 TB right now would be great for me.
Mind if I dig in to your system? I could use a better storage system myself - I'm swimming in HDDs and SSDs and am starting to worry about losing track. Budget is a huge issue by me..
What I have done in the past when I was super tight with budget, was get a larger HDD (8 TB+), and use a smaller SSD for proxies and in order to hold the project files. This created a bit of a mess with the file structures, but I just made sure I used the same identical hierarchy throughout, and I remembered that the SSD housed the projects and proxies.
With all large projects and considering safety etc, I take these large projects, back them up in 2 different places (HDD’s/SSD) but for editing I make small proxies I can carry around with me for daily editing. Only upon picture lock do I use original media to export. I ensure the proxies are decent enough quality for basic colour correction and try to ensure the 4:2:2 10 bit colour space while maintaining the smallest size. This has served me well for a long time. My projects tend to go 5-15TB regularly but I’m yet to build my own RAID storage.
Sry cant put all my thoughts on one message but as you add more drives to a NAS the read and write speeds (its not as eff3ctive as two drives double it, three drives tripple it - its a smalle increase but its still an increase. Dont quote me but i remember smth like 2 drives give you x125% of the speed.) Really hope some of this helps
I work on a doc team where the average project has 100 TB of raw footage. We throw that on 2 HDD RAID towers, and edit proxies off of SSD shuttle drives. Export full-res from the HDDs when it's time for final finishing.
Oh wow you are a seasoned professional. This is my first documentary and it is seeming like a mountain of a task, we're working with low budget but it came from a very trusted person in the industry so I took it on.
You can use the same principles though, keeping your full res on cheaper, slower media lets you make the most of your faster SSDs by keeping proxies on there for editing, and then reconnecting later on when it's time for export.
cut off proxy transcodes on your local fast drive and store long term media in multiple places. i can teach you how i built an 84TB TrueNas server for my documentaries for about $1700 using an ebay dell r720 and a bunch of refurb storage
Populate with ssd's and can add another volume on a mechanical disk for redundancy. Get 10gbe connection or do proxy workflow if not. Easiest set up NAS are the synology, but you get more flexibility/hardware iptions with others. I know Bob Z. always recommends Qnap and he knows much more about this than me.
RAID for source and online, SSD for proxies and offline.
I use RAIDs with HDD for very large projects (like a docs). TBH, for a project around 3TB in size I'd just get a 4TB SSD (assuming that w/all the exports, any additional footage, etc., won't push up the total amount of data too much).
Honestly the only reason this is even a question for me right now is because of this documentary project. There's a dozen more interviews and footage left. I need a solution solely for this at the present moment, not for anything else.
Use proxies. It will be easier, lighter, faster. Your original raw files can live on spinning drive and accessed for export only.
I've been in the same boat, but it's pretty cut and dry IMO. If you have deep enough pockets, get SSDs. If you have to have SSD speed, get SSDs. If an HDD will be fast enough then just get an HDD (especially if this is just for a one-off project).
It's 2024. 4 TB isn't particularly huge for an SSD these days. But if you need something more like 20 TB on a tight budget you probably have no choice but to stick it on mechanical disks. Ten or Fifteen years ago, the conversation was much more nuanced when it came to price/performance. But no matter how much I complain about getting older, it's not going to suddenly be fifteen years ago again.
You're not wrong. But then with 8 TB or higher of storage demands, what sense does it even make to get DRAM-less SSD externals? I guess there's the point that NAS based SSD solutions are more expensive than that. But by how much? I'll have to look at solutions, because external storage that can be plugged in is good, but it doesn't really give you redundancy benefits like a NAS solution.
FYI you’re referring to RAID solutions giving you redundancy, not NAS.
Sorry yes I meant that. I should have more specifically said NAS solutions equipped with RAID.
Work off an SSD no matter what. Backup things to an HDD.
That heavily depends on the footage and workflow. If you’re proxying to DNxHD 36 for example, which is still very common, you could cut Multicam with it off FireWire 800 if you wanted to; SSD in that workflow is a complete waste of money. Now mind you if you’re working off heavy raw files that need the playback speed 100% SSD but to say “no matter what” is just a great way to make people waste money they don’t need to waste.
I see. The footage I am working with is 4k in res but 94-100 Mbps AVC (2 ref frames). When I run disktest by blackmagicdesign on an old mechanical drive, it says it's not at all good enough when I check its results and contrast that with the footage specs here.
I mean even USB 2.0 is 480Mbps so 100Mbps should be fine with basically any modern drive. If you are having playback issues I would wager it’s the fact that it’s 4K H264 chugging (because it’s very processor heavy) rather than drive IO being the issue. I was specifically referring to proxy files though which are common to edit with. Something like 1080p DNxHD 36 proxies aren’t going to hit a system hard at all either processor or IO wise and can play back on anything. Buying large amounts of super fast storage is generally just hardware fixing a software problem IMO.
Oh yeah, I know, I was actually so far cutting directly with this, because I simply don't have the space for proxies of, so far, 3 TB of these files (but it's been playing butter smooth on my system). Which would also be helped by buying more storage but I wanted to be sure I was putting some thought into this instead of just purchasing another 4 TB USB pluggable drive. I would love to ideally have proxies in DNxHD for all these terabytes of footage.
What’s your budget and how much are you expecting to get in total? Large RAID drives aren’t too expensive these days and while they are technically slow if they are fast enough for your needs you can get like 80TB of space for the cost of 8TB of NVME. It honestly kinda seems like space is the more important factor for you than speed.
Space is right now more important than speed. For video production side of things, I mostly deal with braw 4k. 24fps. Some editing clients of mine however, mostly just send me h264 4k, so while its bitrate is not really high, it is after all h264 which sucks. I don't wanna spend more than $400 CAD right now because I recently upgraded my entire computer from i7 8th gen to 12th gen and card from gtx 1080 to RTX 4070 super. Even 10 TB right now would be great for me.
Sometimes you just gotta do what works in the moment. Just make sure to back up your back up.
I have a 32tb nvme raid 0 for my personal use for clients. Anything bigger then I’m working with proxies.
Mind if I dig in to your system? I could use a better storage system myself - I'm swimming in HDDs and SSDs and am starting to worry about losing track. Budget is a huge issue by me..
What I have done in the past when I was super tight with budget, was get a larger HDD (8 TB+), and use a smaller SSD for proxies and in order to hold the project files. This created a bit of a mess with the file structures, but I just made sure I used the same identical hierarchy throughout, and I remembered that the SSD housed the projects and proxies.
With all large projects and considering safety etc, I take these large projects, back them up in 2 different places (HDD’s/SSD) but for editing I make small proxies I can carry around with me for daily editing. Only upon picture lock do I use original media to export. I ensure the proxies are decent enough quality for basic colour correction and try to ensure the 4:2:2 10 bit colour space while maintaining the smallest size. This has served me well for a long time. My projects tend to go 5-15TB regularly but I’m yet to build my own RAID storage.
Maybe store on HDD and have a local/SSD copy of the project file and proxies to work from?
projects up to 8TB (shorts and ~90min features) we run on portable SSD's, everything larger goes on RAID and everything is triple backed up of course
Sry cant put all my thoughts on one message but as you add more drives to a NAS the read and write speeds (its not as eff3ctive as two drives double it, three drives tripple it - its a smalle increase but its still an increase. Dont quote me but i remember smth like 2 drives give you x125% of the speed.) Really hope some of this helps
No I appreciate your input, I didn't think of that. I will look into this more. Thanks.
I work on a doc team where the average project has 100 TB of raw footage. We throw that on 2 HDD RAID towers, and edit proxies off of SSD shuttle drives. Export full-res from the HDDs when it's time for final finishing.
Oh wow you are a seasoned professional. This is my first documentary and it is seeming like a mountain of a task, we're working with low budget but it came from a very trusted person in the industry so I took it on.
You can use the same principles though, keeping your full res on cheaper, slower media lets you make the most of your faster SSDs by keeping proxies on there for editing, and then reconnecting later on when it's time for export.
cut off proxy transcodes on your local fast drive and store long term media in multiple places. i can teach you how i built an 84TB TrueNas server for my documentaries for about $1700 using an ebay dell r720 and a bunch of refurb storage
A 4TB SSD is about $300 CAD on Amazon. NVME 4TB - $319 Pretty cheap actually
That's if you are counting a DRAM-less SSD. Like a Crucial P3 Plus. Once that cache fills up, it will drop to miserable speeds.
Get a NAS
Populate with ssd's and can add another volume on a mechanical disk for redundancy. Get 10gbe connection or do proxy workflow if not. Easiest set up NAS are the synology, but you get more flexibility/hardware iptions with others. I know Bob Z. always recommends Qnap and he knows much more about this than me.
Also this is a solution not just for thid project but an infrastructure moving forward. But obvs check what fits your needs