finally finished - took 2 day and 8 hours to restore 102,000 files totally 1.2tb of data. But over a day of that time was it just doing nothing since it kept pausing every 10,000 files.

I was just looking through the history for this restore and noticed something odd. Last night at 9pm is was restoring a batch of files that ranged from under 1 meg to 50meg. At the start it was restoring maybe 2-3 files a minute. And around 9:20pm it suddenly started going much faster, with more then 1 file a second. And then shortly after 10pm it suddenly went back down to 2-3 files a minute. The file sizes stayed roughly the same the whole time. This seems more like it would be an issue with amazon and not cb, since if it was a problem with the software I'd expect the transfer rate would stay roughly the same the whole time.

So, this 2nd restore which I thought would go faster since there should be less data/files to restore, seems to be going even slower then my first one, i assume due to this 3.5 hour stop for every 10,000 files. It's currently been running for 25 hours and has restored 56,000 files 70gb. I looked through the history of the first restore and I don't see any pauses like this in that one. The only difference I can think of between the two is this 2nd one is encrypted. And I did not restore every file in the vault, i select a set of files. In the first one i restored the entire vault.

More then happy to send you any log files or anything if that helps figure out why it's doing this.

Left it going overnight, and just checked on it now and it's stopped at 30,000

decided to wait it out - about 3.5 hours passed with nothing downloading and then it started again. I guess CB only requests 10k files at a time?

Sorry for the misunderstanding - i wasn't expecting to get 1000mbps throughput. But I think getting half that would be a pretty reasonable expectation, maybe even a little more then half. And I was getting more like 40mbps when you average out the size of the files and how long it took.

I'm really not that concerned about the download speeds using glacier - i'm basically just trying to figure out if it's normal. My initial understanding was that there was a delay in starting the download from glacier, but that once it started there wasn't any sort of throttling and it should download at close to whatever my maximum network speed is. So basically, i'm trying to figure out if the slow download was due to something with amazon, or something with cloudberry.

well I was a little off - it finished just over 24 hours. It finished with an error though, error code 1003. And according to the file count, it looks like just one file didn't download. And from the file size, it looks like a lot wasn't downloaded since it says it downloaded 216 out of 992gb. But it looks like it's all there on my drive now since there's 992gb worth of files there.

noticed one more odd thing - the download status says it's downloaded 198gb of 992gb. But this is incorrect. Because there's just over 900gb in the folder I am restoring the files into. Not sure if the files are compressed or something before download, but they can't be compressed very much since almost all of them are photos - either raw files or tiff/jpg files.

Looks like this one restore job should be finished up tonight and it will probably have taken around 28-30 hours to finish (it's at 23 hours now).

Just for some additional info - i looked at the history tab to see how quickly it's downloading files. The last 10 or so files range in size from 26-30MB. This is how many seconds passed between each download of the files: 11, 7, 12, 10, 2, 11, 6, 2, 13, 1 - so that averages out to 7.5 seconds per file.

That's pretty slow - if it was downloading at 100mb/s it should be able to download each file in about 2-3 seconds. So in effect, i'm getting download speeds something like 30-50mb/s

Oh and to answer your previous question - i'm using the ohio location, and I am in michigan, so it's right next door to me.

If i had to take a guess right now - my guess would be that CB is only downloading a single file at a time. I'm not sure if that's because glacier only allows one download thread or if there's some other limitation.

I didnt run the calculator to calculate costs to restore since I need these files so regardless of the cost, i have to restore them. The cost of backing them up is definitely one of the cheapest. And since this is the first time I've had to do a full restore in probably 7-8 years, cost-wise it was worth it.

I don't need these files urgently. It's just more of a convenience issue since my laptop is basically tied up while this is going on since I can't pause it. I thought once I was past the initial 1-5 hour delay, the restore process would be a little faster then this. Does CB only request/only allowed to request 1 file at a time?

If i did have to stop this mid-restore, will CB pick up from where it left off or start over? I know there's an option to tell CB not to over-write existing files. Just not sure if it does that intelligently - i.e. does it skip requesting the file for download if it already exists?

14+ hours and its probably not even halfway done. Right now it says 20100 of 30100 files copied, 114gb downloaded, 801gb total files to be downloaded. I really didn't expect it to take this long. At this rate it's not how many hours, but how many days it'll take.

I'm using the most current version of cb to do the restore and it is the full version. But the backup was made with an older version, but I can't tell you what version.

I guess unless you have any suggestions I'll just have to let it keep running as is. After this is all done I'll look into creating new backups with the new format.

Just to be clear - i'm only using expedited to restore a few small files just to test that everything is working. I plan on doing the bulk of the restore using a standard request. I did finally manage to get the file downloaded with the correct password. So i don't need to do any expedited restores now.

Also, i can't reupload anything since I dont have the files, which is why i'm trying to do the restore :)

It seems like it's working now - assuming the standard restore doesn't run into whatever issue the expedited restore was running into. I just hope I don't waste hours waiting for it to start and nothing happens.

Thanks for your help again. Hopefully I wont need to bother you with more questions, assuming the actual restore works.

I read your previous reply about how to get an encrypted file without first decrypting it. it was a bit over my head - i'm not super familiar with using aws storage options, other then using CB to backup files.

Is there no way to force CB to download a file in it's encrypted state and then decrypt it after? The main thing I am trying to do right now is determine if i have the correct password for this particular vault. If thought I could just do a couple of expedited requests on a single file until I found the password I used. But if it's going to keep failing like this, that's going to make this process a lot harder.

I believe I run into an issue like this once before and someone from support said I could download the individual file using CB explorer to download the file. Although I can't remember how I would then decrypt it if I did manage to download it.

This is really frustrating. I just tried it again, this time using a password i think I might have used. And this time it was running for over and hour before i cancelled it.

I don't mind if it takes a few minutes longer then what's expected. But this is kinda nuts. That last one was running for over 9 hours. The docs on amazon's site says an expedited request is supposed to process in 1-5 minutes.

Is there any way to determine where the problem is? I don't feel like waiting several hours doing this, especially since right now I'm just trying to figure out if i have the right password. Having to wait hours in between each attempt would be a pain.

Thanks for your help

I cancelled that restore and tried doing it again with just a single file and it failed after 2 minutes due to an incorrect decryption password (i didn't specify one). Now I've got to hope I can remember what password i used. But why did that first restore run for over 9 hours? I used the same exact restore plan, i just edited the file selection to a single file.

Ok - of course 5 minutes after I posted this it finished. Not sure why it took 36 minutes, but it finished downloading, and i can read the file so looks like there's no encryption on that backup. Still have one other one that I need to restore but I'll try that one later.

When you say it will try the password immediately - since this is a glacier restore, doesn't that mean it would have to wait a few hours for the files to become available before it could try decrypting them? Or is it able to test the decryption before it actually starts downloading files?

In order to test this I decided to try just restoring a single file using the expediated glacier restore option. According to amazon's description for an expediated request, the data should be ready within 1-5 minutes. I've been running my restore now for 30 minutes and the file still hasn't downloaded. It's just a 1meg file. I'm running the restore without trying to decrypt it since I don't think I backed it up with encryption, although I'm not sure. If I did encrypt the files and tried to restore them without the correct password, would it result in the restore taking forever to complete (or just not completing) ?

Nevermind - i left CB running for a while and it completed.

Thanks, that seems to have fixed it for me. Going forward i assume cloudberry plans to no longer support glacier storage? Is there some reason why this was dropped? I like to use it since it's the cheapest storage and the items I store rarely need to be accessed so the long restore times aren't an issue for me.