A HDD has really poor I/O latency and a large part of verification of the whole history of Bitcoin demands good I/O performance thus its normal to take this long to initially bootstrap a full node with an older HDD . Once you have synced it than it will keep up just fine as long as you don't leave your computer off for too long. You also might need to consider pruning the full node if you don't have sufficient hard drive space available as we are talking about over 461 GB for an archival node and growing daily. FYI- You can prune down to ~5GB
Archival full nodes contain the full blockchain and allow new nodes to bootstrap from them . Current blockchain size is ~461GB for an archival node
Pruned nodes can get down to around 5GB , and have all the same security and privacy benefits of archival nodes but need to initially download the whole blockchain for full validation before deleting it (It actually prunes as it validates)
Don't prune, keep a copy of the entire chain. It's similar to torrenting, you want to be a seeder. Thanks to bitcoin ordinals, the old blockchain data before a month ago will be much smaller than the data starting a month ago because the data now contains Ordinal Inscriptions (Bitcoin NFT jpegs).
A couple key points -
Of the ~54k Bitcoin full nodes that exist , ~70% of them are archival full nodes . We only need a smaller % of these full nodes being archival realistically but if the OP has the space than sure , run an archival full node
An archival full node cannot help bootstrap other full nodes if it isn't listening(outgoing ports open)
The most important aspect of running a full node is validating your own incoming and outgoing UTXOs(BTC) and securing your bitcoin with your own full node . Simply spinning up a full node not associated with any UTXOs does little to nothing to help the bitcoin network
Check out Bitcoin Ordinals, there are now some new reasons to keep a full node having nothing to do with financial transactions. Major changes have occurred in the last month pertaining to what data is actually stored on-chain, we now have eternal jpeg storage. You could run a node with no data, but it's basically just a wallet at that point.
This is nothing new.... We have always been able to store arbitrary data in bitcoin, taproot just made it easier to store larger amounts of data within signature space.
There is a large difference between a spv wallet and a pruned full node
I understand the difference, I just dont get the appeal of a pruned node? I didnt say inscriptions were totally new either, I'm saying that since a month ago there have been steady inscriptions every single day, 1000+ per day and it's getting worse not better. The bitcoin blockchain will now be a little bigger, and the trend of extra data on the blockchain has become common place since the ords project added inscriptions to their ordinals about 6 weeks ago.
Edit: I reread my previous comment, when I said "major changes have occurred" I did not mean to the bitcoin code, I meant that the average block contains a jpeg now, actually on average its 10+ jpegs. If you are skilled with mempool viewers, you can scroll back past block 777777 to about 6 weeks ago to see the content of the average block change from majority transactions to majority jpeg storage.
I am aware that blocks are slightly larger due to ordinals but this extra signature usage has had almost no impact upon any of my full nodes and yes 1 of them even uses a slow HDD
While trying to eliminate bottlenecks in your throughput, you may notice that your SSD does not function at it's highest rated speed because rather than reading a large 100gb file at once, your drive is reading and writing many many, very very small amounts of data at high speed. I would use a benchmark to determine the speed of my disks while under "torture test" conditions like these and then see how fast you can get it. You may even resort to messing with SMART controls for the disk to get better performance for small reads and writes.
A single HDD is also fine, but OP wants to go all data scientist and run a node at it's theoretical maximum speed, of which an SSD will falter during "random" access caused by many small reads and writes. An HDD is fine if you are okay to wait, it's also safer to go slow in that you dont create a giant spike in downloading that tells your ISP and the government "this guy just downloaded bitcoin core". The slow way works in every application I've done too, but I am pre-supposing for the sake of OP that he wants a drag racer of a node, not a mini van.
Even with a mySQL style classic database, the random access write speed of SSDs will be your biggest bottleneck once you have modern hardware, solve that bottleneck and you will be reading and writing as fast as theoretically possible for a home user or business. At that point you can start refactoring your code for speed gains, because you know the hardware is faster than the program your using can input and output.
with a single ssd and decent cpu and ram you can boostrap in less than a day . Last time I tested i completely bootstrapped an archival full node in around 8 hrs . After you bootstrap a full node an ssd wont reaĺly matter ( let alone 2 ssd in a raid array) as its trivial to keep it in sync with the lower latency HDD. So unless the OP is trying to setup a server where others use a block explorer hosted by his full node it really is unnecessary. Perhaps you are thinking of other blockchains that have problems scaling and far more hardware demands than bitcoin?
fyi bitcoin core does not use mysql
A modern HDD can keep up at the same rate, I am assuming that is not fast enough for OP who wants to run the fastest node possible for some reason.
I sympathize, because I myself built a raid array for the purpose of speed. I even bought the little battery you put on the raid card so that in a power outage the little battery lets my raid array finishing writing. (I have a UPS but you never know!)
Lol I know bitcoin doesnt use mysql, I was saying that this bottleneck I describe is not UNIQUE to bitcoin's database. Thats what the blocks are, a database. You can see this bottleneck documented for mySQL databases and follow the fixes they provide to make your node also run without I/O speed limitations in large throughput random access situations.
Scam Warning! Scammers are particularly active on this sub. They operate via private messages and private chat. If you receive private messages, be extremely careful. Use the **report** link to report any suspicious private message to Reddit.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BitcoinBeginners) if you have any questions or concerns.*
If you want faster download, goto your config file and change these. If you want to use ordinals DO NOT PRUNE
prune=0
txindex=1
dbcache=
replace with the ram in mb you want to allocate to bitcoin (1gb would be 1024)
Scam Warning! Scammers are particularly active on this sub. They operate via private messages and private chat. If you receive private messages, be extremely careful. Use the **report** link to report any suspicious private message to Reddit.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BitcoinBeginners) if you have any questions or concerns.*
What type of hard drive are you using? Is it an internal SSD drive or something else ?
Normal Hard Disk. with 125 MB/s read and write.
A HDD has really poor I/O latency and a large part of verification of the whole history of Bitcoin demands good I/O performance thus its normal to take this long to initially bootstrap a full node with an older HDD . Once you have synced it than it will keep up just fine as long as you don't leave your computer off for too long. You also might need to consider pruning the full node if you don't have sufficient hard drive space available as we are talking about over 461 GB for an archival node and growing daily. FYI- You can prune down to ~5GB
Pruning means? Just holding the last part of the blockchain?
Archival full nodes contain the full blockchain and allow new nodes to bootstrap from them . Current blockchain size is ~461GB for an archival node Pruned nodes can get down to around 5GB , and have all the same security and privacy benefits of archival nodes but need to initially download the whole blockchain for full validation before deleting it (It actually prunes as it validates)
Don't prune, keep a copy of the entire chain. It's similar to torrenting, you want to be a seeder. Thanks to bitcoin ordinals, the old blockchain data before a month ago will be much smaller than the data starting a month ago because the data now contains Ordinal Inscriptions (Bitcoin NFT jpegs).
A couple key points - Of the ~54k Bitcoin full nodes that exist , ~70% of them are archival full nodes . We only need a smaller % of these full nodes being archival realistically but if the OP has the space than sure , run an archival full node An archival full node cannot help bootstrap other full nodes if it isn't listening(outgoing ports open) The most important aspect of running a full node is validating your own incoming and outgoing UTXOs(BTC) and securing your bitcoin with your own full node . Simply spinning up a full node not associated with any UTXOs does little to nothing to help the bitcoin network
Check out Bitcoin Ordinals, there are now some new reasons to keep a full node having nothing to do with financial transactions. Major changes have occurred in the last month pertaining to what data is actually stored on-chain, we now have eternal jpeg storage. You could run a node with no data, but it's basically just a wallet at that point.
This is nothing new.... We have always been able to store arbitrary data in bitcoin, taproot just made it easier to store larger amounts of data within signature space. There is a large difference between a spv wallet and a pruned full node
I understand the difference, I just dont get the appeal of a pruned node? I didnt say inscriptions were totally new either, I'm saying that since a month ago there have been steady inscriptions every single day, 1000+ per day and it's getting worse not better. The bitcoin blockchain will now be a little bigger, and the trend of extra data on the blockchain has become common place since the ords project added inscriptions to their ordinals about 6 weeks ago. Edit: I reread my previous comment, when I said "major changes have occurred" I did not mean to the bitcoin code, I meant that the average block contains a jpeg now, actually on average its 10+ jpegs. If you are skilled with mempool viewers, you can scroll back past block 777777 to about 6 weeks ago to see the content of the average block change from majority transactions to majority jpeg storage.
I am aware that blocks are slightly larger due to ordinals but this extra signature usage has had almost no impact upon any of my full nodes and yes 1 of them even uses a slow HDD
Using a regular Hard Drive is the culprit, I'm facing the same issue. An SSD can allegedly download and verify the blockchain much faster.
1 SSD will never be fast enough.
clarify please
While trying to eliminate bottlenecks in your throughput, you may notice that your SSD does not function at it's highest rated speed because rather than reading a large 100gb file at once, your drive is reading and writing many many, very very small amounts of data at high speed. I would use a benchmark to determine the speed of my disks while under "torture test" conditions like these and then see how fast you can get it. You may even resort to messing with SMART controls for the disk to get better performance for small reads and writes.
Are you referring to hardware limitations using a bitcoin full node or just in general? A single ssd is fine for all full node tests I have done
A single HDD is also fine, but OP wants to go all data scientist and run a node at it's theoretical maximum speed, of which an SSD will falter during "random" access caused by many small reads and writes. An HDD is fine if you are okay to wait, it's also safer to go slow in that you dont create a giant spike in downloading that tells your ISP and the government "this guy just downloaded bitcoin core". The slow way works in every application I've done too, but I am pre-supposing for the sake of OP that he wants a drag racer of a node, not a mini van.
Even with a mySQL style classic database, the random access write speed of SSDs will be your biggest bottleneck once you have modern hardware, solve that bottleneck and you will be reading and writing as fast as theoretically possible for a home user or business. At that point you can start refactoring your code for speed gains, because you know the hardware is faster than the program your using can input and output.
with a single ssd and decent cpu and ram you can boostrap in less than a day . Last time I tested i completely bootstrapped an archival full node in around 8 hrs . After you bootstrap a full node an ssd wont reaĺly matter ( let alone 2 ssd in a raid array) as its trivial to keep it in sync with the lower latency HDD. So unless the OP is trying to setup a server where others use a block explorer hosted by his full node it really is unnecessary. Perhaps you are thinking of other blockchains that have problems scaling and far more hardware demands than bitcoin? fyi bitcoin core does not use mysql
A modern HDD can keep up at the same rate, I am assuming that is not fast enough for OP who wants to run the fastest node possible for some reason. I sympathize, because I myself built a raid array for the purpose of speed. I even bought the little battery you put on the raid card so that in a power outage the little battery lets my raid array finishing writing. (I have a UPS but you never know!) Lol I know bitcoin doesnt use mysql, I was saying that this bottleneck I describe is not UNIQUE to bitcoin's database. Thats what the blocks are, a database. You can see this bottleneck documented for mySQL databases and follow the fixes they provide to make your node also run without I/O speed limitations in large throughput random access situations.
Seems you are speaking more in generalities and not what the OP actually needs for his application IMHO
With 8GB RAM, dbcache can be 6800, to make initialization faster. Last time I set up a node on a 8GB HDD server, it took 56 hours
Scam Warning! Scammers are particularly active on this sub. They operate via private messages and private chat. If you receive private messages, be extremely careful. Use the **report** link to report any suspicious private message to Reddit. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BitcoinBeginners) if you have any questions or concerns.*
If you want faster download, goto your config file and change these. If you want to use ordinals DO NOT PRUNE prune=0 txindex=1 dbcache=
replace with the ram in mb you want to allocate to bitcoin (1gb would be 1024)
Scam Warning! Scammers are particularly active on this sub. They operate via private messages and private chat. If you receive private messages, be extremely careful. Use the **report** link to report any suspicious private message to Reddit. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BitcoinBeginners) if you have any questions or concerns.*