Optimum S3/Cloudberry config for desktop data
I am backing up W10 desktops directly to S3 using CB Backup Desktop for Windows. I want to keep current + 2 prior, and I want to minimize storage cost. Retrievals are very infrequent (<1 year/machine). There are typically a few very large files that change daily, but otherwise most files may only have one version ever.
I'm thinking some combination of OneZone-IA and Glacier Instant, but don't know the right strategy. Should I:
1) Configure CB for incrementals with infrequent full, and then use S3 Versioning and a lifecyle rule to migrate data down to Glacier? (I assume I need synthetic turned off for the fulls in this case, as CB states synthetic doesn't work with Glacier). Or,
2) Turn S3 versioning OFF and then use GFS approximate my current+2 with the various time limits? I would I think style keep synthetic off and go directly to Glacier .
3) A better plan I haven't thought of?
Any all advice/experience appreciated.
A few questions first:
Do you plan to do image backups - I believe the newer version allows image backups of desktop OS’s
Roughly how many files and how many GB’s are you backing up?
What ISP upload/ download speed do you currently have?
Are you sure you want to “keep # of versions” vs say a 30 or 90 day retention period for versions?
Thanks Steve. I'll do image backups for the system disks and file backups for the data.
Typical system disk is 75GB and data will range from 40GB to 250GB/system. My effective upload speed is about 280 mbps.
The version vs retention question is a tough one. Retention period is OK for most things, but for that infrequently-used access database, by the time someone realizes that the current version is corrupted, the prior version might be several months old. That said, most files probably only have one version ever. And I also have one system that will update 80 mb of files every day (same files), so keeping 30 or 90 days can create some size, though I guess in the scheme of things it wouldn't be a big deal. I'm very interested in what you see as the tradeoffs between the two approaches. I do know I've found using the S3 versioning to be pretty non-transparent and backup buckets seem to creep up in size forever.
thanks for answering my questions.
This is how we would setup a backup scheme based on your requirements:
If you don’t already have one, get a 4-5 TB USB 3.x capable removable Hard Drive.
Set it up as a remote shared device and send both Image and data backups in legacy format from each of your computers to that device. If you have a standard OS build, you really do not need to image every desktop, just your standard OS build and any one offs.
This costs nothing other than the device cost (~$100) and should allow you to keep a couple for weeks of images ( daily incremental /weekly full with a 6 day retention, using legacy backup format.
We keep a year or two worth of data versions and deleted files - as long as we have the drive capacity ( hence the 5TB drive).
Cloud Image backups:
Once you have a set of local Image backups, there is no need to keep more than one or two copies of your standard image in the cloud.
We send daily Image backups to the cloud using the New Backup Format (NBF), with a synthetic full scheduled each weekend. We give it a one day retention. So we have anywhere from 2 to seven copies depending on the day of the week. ( if this is confusing, let me know and I will explain).
Now to keep costs down we use BackBlaze B2 ( not BB S3 compatible) for our Image cloud backups.
Reason #1 - the cost is only .005/GB/mo. Vs .01 for OZ- IA
Reason #2 - it supports synthetic full backups
Reason #3 - there is no minimum retention as there is with Amazon One zone IA ( 30:days).
Cloud File Backups
We would use legacy file format and backup to Amazon OZ- IA with a 90 day retention.
We run monthly fulls and daily block level incrementals.
Understand that a “full” in legacy format is only backing up files that have outstanding block level versions since the last full.
So the actual space consumed for all of the unchanged files and versions is typically not more than 10-15% more than the size of the data on the source.
File Backup Retention policies
Setup a separate daily cloud backup plan for that infrequently used access database and give it a 90 or 180 day retention period. Keep in mind you will eventually have a years worth on the local drive, but that cannot be guaranteed as the drive could fail.
Exclude those files from your normal file cloud backup plan and give it a 30 day retention in OZ IA.
Understand that with a monthly full and 29 incrementals, the previous set of fulls/ incrementals will not be purged until the last incremental of the set has aged to 30 days.
So in summary:
- Get a 4/5. TB local drive and backup files and images from all of your machines using Legacy format to it with as long a retention setting as you want.
- Send nightly images to the Cloud ( only unique ones) using NBF and weekly synthetic fulls. With your 280mbps upstream speed this will be piece of cake. Set retention to one or two days since it is for Disaster recovery not long term retention.
- setup a legzcy backup for your normal files to Amazon OZIA with a 90 day retention, monthly “incrementals” ( fulls in our language) and block level incrementals each day.
- for those infrequently updated access db files, setup a separate backup plan and set the retention to a year or whatever you like.
As for glacier, there is a significant cost to use lifecycle management to migrate from OZIA to glacier- $.05 per thousand objects. For small files, you will wind up paying more just to migrate them than you will save. . When we have a particular folder that holds large files ( over 2MB each on average ) that dont change, we will use Cloudberry Explorer to setup a lifecycle policy for that folder(s) with the large files to migrate to Glacier after 30 days.
in general, I do not recommend using the Glacier lifecycle migration. Not worth the trouble.
So I apologize for the lengthy and perhaps confusing reply, but there are a lot of factors to take into account when optimizing backup strategies.
Sign in or register to add a comment.
Add a Comment
Welcome to MSP360 Forum!
MSP360 Managed Products
Managed Backup - General
Managed Backup Windows
Managed Backup Mac
Managed Backup Linux
Managed Backup SQL Server
Managed Backup Exchange
Managed Backup Microsoft 365
Managed Backup G Workspace
Backup for Linux
Backup SQL Server
Connect Free/Pro (Remote Desktop)
Backup fails with "S3 Transfer Acceleration is not configured on this bucket" error
Amazon S3 Deleted Files and Cloudberry Backup
[Bug Report] - Cloudberry fails to transfer OneDrive ALOUM file to S3.
IAM permissions required for S3 bucket to connect to Cloudberry Drive
Terms of Service
Useful Hints and Tips
Created with PlushForums
© 2023 MSP360 Forum