My experiences on using AWS S3 Glacier IR for MSP360 backups
(prices shown are for AWS USA regions, might vary on other regions, I haven't checked all)
While not new anymore, it was made available on Nov 30, 2021, so almost 2 (two) years on the exact date i'm writing this, Glacier Instant Retrieval (Glacier IR) is a great AWS S3 Storage Class for storing backups. While S3 Standard costs USD 0,023 per GB, Glacier IR costs USD 0,004 per GB. Yes, about 20% the cost of S3 Standard, while still being a "hot" storage class, there's absolutely no delay on restoring data, despite the "Glacier" in its name. Previously, all Glacier storage classes were cold ones, but Glacier IR is actually a hot storage class.
But it has some caveats, some I learned the hard $$$$ way, and some that can be clearly found on the AWS S3 pricing pages. And i'd like to share some of my learnings with you.
Initially, I'd like to say that at this point i've already switched all the backups, from all my customers, mainly to Glacier IR. I manage, as of today, about 70TB of S3 data (about 68,5TB S3 Glacier IR and 1,5TB S3 Standard) of my customers backups. While others might have much bigger usages, I'd say 70TB is not a small usage whatsoever
With Glacier IR, you'll commit to pay the space usage for a minimal of 90 days, no matter how long you use it. Used for 1 month? You'll pay 90 days. Used for 2 months? You'll pay for 90 days. 1 week? 90 days. Half an hour? 90 days! Used for MORE than 90 days? Then no problem, you can delete it when you want, and pay prorated values from the 4th month on. With S3 Standard, you have no commitment like that, you'll always pay prorated for the time you used it. With Glacier IR, if using for 1 month, for example, you'll be charged 0,004 (per GB) on the first month and, as soon you delete the data, the remaining (to 0,012) will be immediately charged. On the 1 month example, you'll pay 0,004 on the first month and the remaining 0,008 on the second, as soon data is deleted.
One of the other important differences from S3 Standard to S3 Glacier IR is that you'll pay for retrieving data. Cost is not negligeable, USD 0,03 per GB. I got this information, it's clear on the Pricing pages, but I (now I know, very incorrectly) imagined that by "retrieving" data, AWS meant restoring data to my MSP360 instance, taking data *OUT* of AWS datacenters. And oh boy, I was wrong on that one.
How did I find that? Because of the Synthetic Backup feature from MSP360. While a great feature, no doubt on that, doing a Synthetic Backup causes all data from the previous backup to be read from the S3 Glacier IR to create the new backup chain. And by "retrieving data", AWS really means "READING" data from the Glacier IR, not taking it out to the world. Luckly I was following my billing daily, to quickly find something abnormal with my S3 Storage Classes changes, and quickly caught that one. A Synthetic Backup was made on a backup chain of about 3,5TB and I was charged 3500 * 0,03 = US$ 105 for "retrieval" fees. PANIC took me by storm, and I quickly find out that my interpretation that "retrieving data" means taking data OUT of AWS datacenters was very wrong and it really meant READING data from the Glacier IR. And, thus, Synthetic Backup and Glacier IR became real enemies.
I had to stop some backup plans I had already migrated, luckly just a couple ones for testing, to find out how to workaround that. Luckly, I got the answer from MSP360 Support Team, and was able to disable Synthetic Backups. While not able to directly disable it via MSP360 GUI, it can be easily acchieavable by editing the XML-like file for the backup plans (extension .cbb), located at "C:\ProgramData\CloudBerryLab\Cloudberry Backup", search for the "SyntheticFull" parameter and change from "true" to "false" (screenshots are attached to show exactly how to do it). Change will immediately presents on MSP360 GUI, there's no need to restart service or refresh anything. Just change the XML, save it, and you're done. And funny is that AFTER manually changing the parameter on the cbb file, the option to reenable Synthetic Full appears on the MSP360 GUI. The value won't revert back to true by itself. And upgrading MSP360 won't change that neither. You'll only need to remember that after creating a new backup plan that stores on the Glacier IR. It's a one-in-a-lifetime change for each new backup plan, and you're done.
The obvious downside is that I'm now forcing MSP360 to really upload everything again when a full backup happens, and not using the Synthetic Backup beneficts for that, which would certainly increase backup times by a lot. While really uploading everything again was not a problem for my scenarios, it might not be OK for some other scenarios. As usual, YMMV.
Remember that 90-day commitment? That wasn't a problem at all for most backup plans and, in fact, if I watch my billing closely, I'll find that i'm paying A LOT of prorated charges for data deleted before that 90-day commitment time. And that's just fine! Some simple math explains that.
S3 Standard = 0,023 per GB
S3 Glacier IR = 0,004 per GB with 90-day month commitment, so MINIMUM 0,012 per GB for the 90-days period and really 0,004 after 90-days
So, it I keep some data on the Glacier IR for 1 month, for example, I'll pay 0,012 per GB. If I was using S3 Standard, that would be 0,023! So, even if I'm not storing data for the minimum required period (90 days), and will always pay for 90 days if used for less than that, in lots of scenario it still makes sense to use Glacier IR for storing backups for 1 or 2 months, for example, according to your retention settings. It won't make sense to use Glacier IR for short periods, maxing on the 14-15 days range, which I also uses for some other backup plans. For those periods, maxing at 14-15 days, I already switched back to S3 Standard, as pricing for 15 days of S3 Standard is about the same for the Glacier IR and, if I need to restore data from that S3 Standard-stored data, no retrieval fees will apply. So, for very short storages, max 14-15 days, S3 Standard still makes more sense than S3 Glacier IR.
And speaking on them, the retrieval fees ... as explained, restoring data from backups stored on the S3 Glacier IR will get you charged twice: once for Glacier IR retrieval fees (USD 0,03 per GB) and also AWS-Data-Out charges for the Internet-Out traffic (this one would also apply to S3 Standard data). I completly accepted that, that in the case of restoring something from Glacier IR, i'll pay for that. The savings from switching from S3 Standard to S3 Glacier IR, even paying prorated for data not stored for the required 90 days, were HUGE (more than 50%). And most of the times, luckly, we don't do lots of restores anyway. So accepting those charges, when (and if) I need to restore something, was OK to me. For cases in which is known that lots of RESTORES will be needed beforehand, those retrieval fees from the Glacier IR might add very fast, thus maybe making using Glacier IR a BAD idea. YMMV.
That's my experience on using MSP360 with AWS S3 Glacier IR storage class. It might bring a HUGE costs decrease for lots of scenarios, while still being a hot storage class, but some caveats needs to be understood. Unfortunelly, disabling Synthetic Backups is one of them. If you're not willing to disable them, I'd say completly forget about Glacier IR. BUt if that's OK to you, making the change might be interesting and might save you A LOT of money!
Hope that helps someone!
path in which MSP360 store the backup plans as plaintext, XML-like files, with cbb extensions. Open the different cbb files until you find the one that fully matches your backup-plan-name on the "<name></name>" parameter
value to be changed from true to false, to disable Synthetic Full backups
as soon as the file is saved, Synthetic Full will be shown as disabled on the MSP360 GUI
funny enough, the option to reenable Synthetic Fulls will appear on the GUI after the manual change to disable it
Thanks Leonardo for the answer of my question and pointing me to this post.
This is a temporary fix because in my opinion that option to enable or disable the synthetic should always be there. I'm having a lot of time outs that i didn't have when Synthetic was not a thing, so It would be much simpler not to have to edit the xml, maybe msp360 are reading this and make the option always there in future builds.
Sign in or register to add a comment.
Add a Comment
Welcome to MSP360 Forum!
MSP360 Managed Products
Managed Backup - General
Managed Backup Windows
Managed Backup Mac
Managed Backup Linux
Managed Backup SQL Server
Managed Backup Exchange
Managed Backup Microsoft 365
Managed Backup G Workspace
Backup for Linux
Backup SQL Server
Connect Free/Pro (Remote Desktop)
Cloudberry Explorer pro (S3 browser) not listing folders completely in DELL ECS S3 bucket
Backup Local Data > USB > AWS S3 Glacier recommendation
question about aws s3 restore from glacier
Backup fails with "S3 Transfer Acceleration is not configured on this bucket" error
Terms of Service
Useful Hints and Tips
Created with PlushForums
© 2023 MSP360 Forum