Thanks David. Yeah, it's megabytes MB - tried to keep all the numbers in the same format so the math wouldn't hurt my head. We have residential 1Gbps fibre in Singapore and I consistently get 400mbps up and down to a local server at any hour. So that's why I said a 50MB connection. No bottleneck there.
Aha, when I try speedtest.net
to various Oregon servers (e.g. Comcast), then I'm getting only 8Mbps (1MBps) up, and only slightly better to San Francisco. Cloudberry is currently showing 2-4MBps - perhaps the AWS servers are better located.
So, no surprise there that the SG-US connection is a bottleneck. But weird, while cloudberry is working on the same 6 or so files, all 6-10GB each, for a long time, the upload speed is usually around 2-3MBps, but might go up to 10MBps for a little while, then back down. Concurrently, I keep running the speedtest.net
and my connection is consistently 1MBps, which makes it really odd that my effective connection speed is 1MB and my upload speed can get up to 10MB.
Time of day doesn't appear to matter. Internet connection appears to be stable, but I don't have a good test on that (general web surfing doesn't cut out).
I went through the Cloudberry options and it seems only the thread count and chunk size would make a difference??? There's no proxy needed, I have speeds set to no-limit, etc.
Thread count from 5-20 doesn't seem to make a difference, but the queue shows only 9 active uploads at the most so maybe this option doesn't really change in real time?
For Glacier multipart upload chunks, I've tried 10-128-256-512 MB chunks ("upload chunks in parallel threads" is checked). The larger numbers are a little better, but the variability mentioned above seems to trump everything.
AWS Transfer Acceleration SG-US shows 2000% increase but as you mentioned, it is not available for Glacier and is a bit pricey. Alternatively, I could upload to Singapore and then transfer to US within AWS, but that too incurs a fee (not sure if available for Glacier anyway). I store in the US because I want it there longterm and the storage rate is a little lower than Asia.
Once I get my current 2TB uploaded, then incremental uploads will only be 200GB/week so this won't be as much of a problem.