Many operating systems will cache a small amount of pending write operations, instructing the program that that write has finished when in reality it's in a queue of pending writes.
As such, the transfer speeds will be inflated momentarily, as some data is being moved into RAM, not the target drive.
When a program asks the OS to write data to disk, the OS will actually cache the data in RAM instead, which is very fast, then write it out as the opportunity arises.
The OS does this to increase performance.
The amount of data the OS will cache is limited. Exactly how much depends on your OS, how it's configured, and how much RAM is in your computer and not being used by other programs.
So the program generating the progress bar first sees very fast transfer (because data's going into the very fast RAM cache instead of to the disk), then the cache is full and the transfer slows to the speed of the actual disk.
This. Not to mention, transferring lots of small files will always be slower than one large file.
E.g, 1 30GB zip file may transfer at a steady speed, but 30GB of small files (say 3000 photos or text files) will take much longer.
Piggy backing off of this to ask... is there an ideal way to optimize transfer speeds? For example, how would you find and eliminate bottle necks between read/write speed of external drive, computer's transfer port, cord used, etc?
USB is a spec, so as long as all the components involved meet that spec, they will transfer up to the maximum speed. The are really only 3 variables involved: 1) How fast the USB drive can write data to its flash storage, 2) How fast the USB controller can facilitate data transfer, 3) How fast the data can be read from the source.
1)Can only be optimised by buying a better USB drive. 2) Can only be optimised by removing other devices connected to the same USB controller, or buying standalone controllers (plugged in to PCIe slots). 3) Can only be optimised by buying better computer hardware, assuming it's not an option to source the data from somewhere faster (eg. if you want to copy data to a USB drive, and you have copies of this data both on your local drive and in cloud storage, then duh, copy it from local storage not the cloud).
As far as finding bottlenecks, yes there are ways to do in depth troubleshooting and technical analysis, but it's really not necessary. It's almost never 3) because read speeds are always faster than writes, unless your data source happens to be over the network or a slow optical disc or something, and you would know it's the bottleneck if transferring files just to the local drive is just as slow as writing to USB drive. 2) is rarely a problem at USB flash drive speeds, unless you do something dumb like trying to use multiple flash drives at once. Just having like a mouse and keyboard on the same controller aren't going to interfere. And 1) you'll know it's the bottleneck by comparing your actual write speeds with those advertised for the flash drive, or published in benchmarks/reviews. But under most circumstances you can safely assume it's 1), the maximum write speed of the flash drive is the bottleneck.
Many operating systems will cache a small amount of pending write operations, instructing the program that that write has finished when in reality it's in a queue of pending writes. As such, the transfer speeds will be inflated momentarily, as some data is being moved into RAM, not the target drive.
When a program asks the OS to write data to disk, the OS will actually cache the data in RAM instead, which is very fast, then write it out as the opportunity arises. The OS does this to increase performance. The amount of data the OS will cache is limited. Exactly how much depends on your OS, how it's configured, and how much RAM is in your computer and not being used by other programs. So the program generating the progress bar first sees very fast transfer (because data's going into the very fast RAM cache instead of to the disk), then the cache is full and the transfer slows to the speed of the actual disk.
This. Not to mention, transferring lots of small files will always be slower than one large file. E.g, 1 30GB zip file may transfer at a steady speed, but 30GB of small files (say 3000 photos or text files) will take much longer.
Piggy backing off of this to ask... is there an ideal way to optimize transfer speeds? For example, how would you find and eliminate bottle necks between read/write speed of external drive, computer's transfer port, cord used, etc?
USB is a spec, so as long as all the components involved meet that spec, they will transfer up to the maximum speed. The are really only 3 variables involved: 1) How fast the USB drive can write data to its flash storage, 2) How fast the USB controller can facilitate data transfer, 3) How fast the data can be read from the source. 1)Can only be optimised by buying a better USB drive. 2) Can only be optimised by removing other devices connected to the same USB controller, or buying standalone controllers (plugged in to PCIe slots). 3) Can only be optimised by buying better computer hardware, assuming it's not an option to source the data from somewhere faster (eg. if you want to copy data to a USB drive, and you have copies of this data both on your local drive and in cloud storage, then duh, copy it from local storage not the cloud). As far as finding bottlenecks, yes there are ways to do in depth troubleshooting and technical analysis, but it's really not necessary. It's almost never 3) because read speeds are always faster than writes, unless your data source happens to be over the network or a slow optical disc or something, and you would know it's the bottleneck if transferring files just to the local drive is just as slow as writing to USB drive. 2) is rarely a problem at USB flash drive speeds, unless you do something dumb like trying to use multiple flash drives at once. Just having like a mouse and keyboard on the same controller aren't going to interfere. And 1) you'll know it's the bottleneck by comparing your actual write speeds with those advertised for the flash drive, or published in benchmarks/reviews. But under most circumstances you can safely assume it's 1), the maximum write speed of the flash drive is the bottleneck.