Hello,
Could I get some clarification on how the backup plan retention policies work? I see that there are two options for a custom retention policy: deleting versions after a certain amount of time, and keeping a certain number of versions of each file.
1. What counts as a version of each file? Is a new version created every time a block-level incremental backup is run, or every time a full backup is run?
2. Does the "older than" option override the "number of versions" option, or vice versa?
3. What retention policies are generally used by you all or recommended? Typically, we have it set to keep 30 versions of files, and to keep backups for at least 30 days. However, we are also beginning to struggle with backups taking up an excessive amount of storage space--yet we don't want to sacrifice our backup histories.
We do not use the “ keep # of versions” as that complicates our retention scheme.
If we set the plan to keep 30 versions, A file that changes every day will have 30 versions in 30 days. For files that change once per month, it willl keep 30 months of versions.
We set our retention for file backups to 90 days, with one “full” each month and daily block level backups.
In the above example we would have 90 versions of the daily- updated file, and 3 versions of the monthly-updated file.
Because we do “full” ( what MSP360 calls incremental) backups only once per month, it takes an extra 29 days for the oldest set of full/ block-level versions to age to 90 days. If we did a full (incremental) each week, we would only have an extra week to wait until the oldest set is purged. The trade-off is that we would be storing 12 “full” versions vs 3 and if there are large, frequently updated files ( pst, Quickbooks files, etc) it can increase your storage consumption even more than keeping an extra 21 days of block level backups.
Confusing? Yes.
But it has worked well for us.
Happy to discuss further.
-Cloud Steve
Thank you for your response Steve. If you don't mind, I have some clarifying questions to help me better understand how this works.
1. It sounds like you are saying that the time needed for backups to "age" to 90 days is affected by the frequency of the full/incremental backups--is this right? Does running a full/incremental backup trigger the purge of files older than 90 days?
2. If we have both the "# of versions" and "older than" options enabled, will MSP360 keep the maximum number of versions/days? In other words, will MSP360 acquiesce to the "# of versions" option if it allows for a greater number of versions than the "delete older than" option allows?
Thank you for your assistance!
1. I admit this is confusing but if you think of your backups on storage as a set containing a "full" backup plus any incrementals based on that full, this gets easier to understand. Example:
- Yor retention period is set to 30 days
- You have a file that gets changed every day
- You run the full backup of that file each Sunday and block level incrementals on the other six days
- At the end of 30 days you would have four "sets" of 1 full/ 6 incrementals in Backup storage and one partial set with a full and one day of incrementals:
Set #1 = Day 1-7
Set #2 = Day 8-14
Set #3 = Day 15-21
Set #4 = Day 22-28
Set #5 = Day 29-30
- On day 31 you run another block level backup
- Set # 1 cannot be purged until all of the elements in the set have aged to 90 days.
- So until the day 7 block incremental is 30 days old , you cannot purge anything from Set #1
- But on Day 37, the entire Set 1 will be purged as all components have aged to 30 days (or more).
If you choose to do monthly fulls with block incrementals every other day of the month, Set #1 will not be purged until Day 60 - the point where the last incremental in the set reaches 30 days old.
2. My experience/understanding is that it is an "OR" condition, meaning that if either condition is met, the files will be purged. I agree that if it was an "AND" operation it would be more useful for the strategy that you are looking to employ.
Thank you, I believe I understand your first answer.
2. That makes sense that it is an OR condition, but are you saying that backups are kept based on whichever is greater (i.e., whichever option results in more backup versions)? Or would they be kept based on whichever resulted in fewer backup versions?
As an example, say that I set my backups to delete versions older than 1 month, and to keep 45 versions of each file.
Would MSP360 purge my backups after:
a) 30 days, since this is likely to come first?
b) I had accumulated 45 file versions, since this is likely to come second?
Since this is an OR condition, I am trying to understand under what circumstances would one of these options take precedence over the other when they conflict. Hopefully I am being clear. I am attempting to reach back to what I remember from a high school Logic class, and as you see the results are mixed.
So if I am correct, (and David G. will correct me if I am wrong), if you specify 30 days retention and 45 versions, all versions older than 30 days will be purged even if there are only 5 versions in storage.
If you had a 90 day retention and the same 45 versions setting, as soon as Version 46 is created, which could be on day 46, version #1 would get deleted, even though it is not yet 90 days old.
I don't trust/like the " #of versions" setting for this reason.
We sell 90 days of version recoverability to out clients for file/data backups. For Accounting and law firms that only touch customer files once per year we sell an extended retention period for a very small uplift that provides 15 months of versions. As you would expect, Image/VHDx backups are different.
So would it be correct to say that MSP360 will purge the backups based on which option results in fewer versions?
Also, if we were to set the "# of versions" to 1, would that literally only keep a single version of all files, even if it were also set to keep versions for a full 30 days?
If we had a file that was older than one month, but had not been deleted, and we left the "# of versions" unchecked, would this file still be saved despite its age?
I apologize for all the questions. Yet we haven't found very detailed documentation on the various retention policy options and how they interact. Is there some documentation that defines options?
Thank you once again for your assistance.
if you check the option to always keep the latest version, then even if it's beyond the 30 days that version will be kept, so you'll always have one version to restore even on those files that have aged out. If you set your versions to 10 any versions older than the 30 days will be removed. Only versions within the 30 days will be kept. After the 11th version is created within those 30 days the oldest version is removed leaving you with 10 within those 30 days. I hope that helps.
I took so long to write this that David G. beat me to the punch. My [lengthy] explanation below takes into consideration the implications of the Full/Block Backup sets I described in a prior post.
Backup Fan - Yes you are correct. It will purge based on the most aggressive purge setting, be it # of versions or days.
Keep in mind that the latest version of each file is always saved unless you deliberately uncheck that setting. There are situations where we do his but that is beyond the scope of this discussion.
If you back up a file such as a picture or pdf that never gets modified, it will remain in backup storage forever, regardless of any retention settings.
The settings only apply to files that get modified, creating multiple versions.
To continue for those hell bent on using # of versions, I will try to explain how it works using a different example. Example
- Retention set to Keep 2 versions
- Retention set to keep 30 days
- Fulls set to run weekly
Day 1 - Full backup = V1
Day 2 - Block incremental = V2
Day 3 - Block Incremental = V3
You would think that V1 could be deleted, but it cannot since you still need V2 (in order to still have 2 versions) and V2 depends on the Full (V1).
Now lets say that the file is not touched again until the Weekly Full Backup runs creating V4.
You still cannot delete any of the V1-V3 versions since they are considered a "set" and you cannot delete V3 for the same reason as above.
The next day you run a block level backup (V5)
NOW V1 - V3 will be purged, since we now have two versions (V4 & V5).
Lets say that you do not modify this file again.
The following week a new Full (V6) will get created as a full is taken for any files that have had modifications since the last full.
When V5 reaches 30 days old, can V4 and V5 be deleted? No, because V5 represents Version #2 and Since V5 is associated with V4, it cannot be deleted either.
Now if you did only fulls (by turning off block level backups), you would always have just the two most recent full backups of the file, but that will chew up more storage than an incremental does, particularly if you have files in the 100's of MB's. Summary
If you are backing up files/versions for paying clients, I strongly recommend a 30 day retention - at a minimum - and do not use # of versions. We use 90 days as our standard, with some clients paying extra for 15 month's retention to protect versions of files that are only touched once per year or so (accounting, Law Offices, etc.)
This way it does not matter how many times each month the file gets updated, you can always go back to a version from 30 or 90 days ago. For some files, it might be two versions for others there may be 30 or 90 versions but in all honesty, at the hideiously low cost per TB of Cloud storage available these days, storage costs should be the least of your concerns.
Not being able to recover a file from a month ago because you only have the last two days due to your " # of version" settings will be far more painful.
Thank you Steve and David, this is all very helpful.
However, the option to "Always keep the last version" only exists in the File Level backup plans, and not in Image Based plans. Is it by default set to keep the last version in Image backups?
For the record, I am using both 7.2.0.280 and 6.3.9.12, and both versions are configured this way.
The most recent version will stay in the Cloud until another run of the backup plan completes successfully, and the prior version meets the purge criteria. It’s the same as the data/ file plans.
We used to use # of versions for image and VHDx backups, but we found that the purges of old versions were not happening ( fixed since I suspect), so we went to a # of days approach.
We do a full Image Cloud backup once per month for Disaster Recovery purposes, and daily local Image/VHDx backups to a USB drive.
For retention settings, we use 1 week for the Cloud backups, and since it is a month between backups, the prior version gets purged and we end up with only one version in the Cloud.
Certainly you could try the # of versions approach for Cloud Image backups, but assuming you are taking daily or weekly Local Image/ VM backups, there is no real reason for more than one copy in the cloud (IMHO).
I just realized this. We were using up too much space in the cloud so we transitioned over to a remote deploy configuration where a full backup is run at the start of the month, and a block-level is done every day. It also allows for consistency with all the cloud plans (deploying local plans this way are trash). I put delete versions older then 30 days and 2 versions for each file.
Today I went and saw that once the Full Backup ran at the beginning of the month, there was nothing before that. So that means every time a full backup is run, there is no retention because lets say they deleted a file on 9/29, well I don't have it because the full backup ran on 10/01 deleted it.
So what settings should I use? I'm sure we would want 30 day retention of files. So if a full backup runs on 10/01 then I'm sure we would want the ability to restore a file from say 09/10 or something.
After talking it over, it seems like to get what I am looking for is to UNSELECT keep number of versions for each file and keep delete versions older than, set to 30 days. Once a full runs on 11/01, it will delete 10/01 backup, then when the incremental runs on 11/02, it will delete the 10/02 incremental, so on and so forth?? Right?? I also use Wasabi so it seems like I should just set the delete versions older then 90 days instead of 30 or what?
Mike S. -
Are you talking about data files or images?
For data files, we keep 90 days worth of versions .A version will never get deleted before it has aged for 90 days.
If you are using Wasabi, 90 day retention works out fine - you will not get any "early delete:" charges.
However, your statement that: "Once a full runs on 11/01, it will delete 10/01 backup, then when the incremental runs on 11/02, it will delete the 10/02 incremental",is incorrect.
You cannot delete a full until all of the dependent incrementals have reached the retention setting, in this case 30 days.
If you are running a full once per month. incremental versions that were created during October would not be deleted until the last incremental on October 31 has reached 30 days old on November 30.
At that point the October 1 full and all of the October incrementals will get purged at once.
If you were to run fulls each week, on Nov 7th the Fulls and incrementals from Oct 1-7 would get purged.
For image backups (and VHDx files), we only backup once per month and want to keep one version.
We only want to keep a full for 30 days so we do not use Wasabi as we would get charged for 90 days whether we use it or not.
We have been using BackBlaze for the once-a-month Full Image backups as they have no minimum retention period.
I am talking about images. What would be the best setting for this? We deployed "Delete versions older than" to 30 days and "Keep 2 versions for each file" and "Always keep the latest version." Unfortunately, its not working as we thought. We do a Full Backup every Month, and a Block Incremental every day. What I am looking for is basically so when my full backup runs on November 1, it doesn't wipe out all of October 1-30 until December 1.
I played around with the new format and GFS and it seems way more straight forward but still a tad confusing. Seems like that will be the way to go going forward once its in the remote management console. I settled on GFS for keep backup for 60 day and keep monthly full 1 month. Not sure if that's the right configuration but...
Version for images are restore points. Keeping 2 versions when you are running a full every 30 days may not do what you want. You would run:
1 - full
2 - incremental - now you have 2 versions
3 - incremental - need to keep it because you only want two and those two are this one and the previous incremental since incremental backups are chained - but the full is needed for those restores so it needs to stay too
4...29 - same thing
the next set runs and
1- full - now this full and the previous incremental are your 2 versions, but since that previous incremental needs all the backups in that set, everything stays
2 - incremental - now you have 2 versions - this incremental and the full before it and now the previous backup set (full + 29 incremental backups can be removed)
3 - same as previous backup set example
To do what you want, keep 30 versions. You'll end up with 60 backups before the first 30 can be removed and that cycle will repeat.
1. What if I just put "Delete versions older than 1 months" and uncheck "keep number of versions"? Is that the same thing?
2. Aare you saying to put "Delete versions older than: 1 months" and "Keep number of versions: 31"
3. What would be the equivalent settings for the GFS policy? I put "Keep backup for 60 days" and "Keep monthly full for 1 month". Not sure if that is the right configuration.
I think with GFS, you can just keep monthly backups for 1 month if you want. Or keep weekly backups for 4 weeks - although there will be more data stored in this case.
You can delete versions older than 1 month if you want, as that will mostly have the same end result. But you do not need to specify both options if you do not want. It all depends on your full and incremental backup schedules.. If you run the full monthly and run incremental backups 5 times a day, you'll end up with 150 incremental backups over the course of a month. That's more than 30 versions. If you ran a full once a month and incremental backups once a week, you'd end up with 4 incremental backups per month and if you used 30 versions in retention, then you'd be keeping a minimum of 8 months of backups.
I just realized this. We were using up too much space in the cloud so we transitioned over to a remote deploy configuration where a full backup is run at the start of the month, and a block-level is done every day.
I played around with the new format and GFS and it seems way more straight forward but still a tad confusing. Seems like that will be the way to go going forward once its in the remote management console.
Is this on your guy's radar? I know that it probably is applied correctly in the Windows application but it would make me feel better to see it on the website properly represented.