Immutability / WORM support is coming - currently scheduled for August. I'll reply back with more details once I have them.

Also very interested in immutable backups. Especially seeing other storage providers like Backblaze now support it.

I would like to add my name to the list.

The add-on question I have is, will the immutable backups be for other storage accounts like Amazon S3, Azure and Wasabi or just for MSP360 Wasabi accounts?

+1

The beta is now available for stand-alone customers and can be downloaded here. I don't have a date for Managed Backup yet.

https://get.msp360.com/cbb74beta

More information about the release is in this post:
https://forum.msp360.com/discussion/2287/cloudberry-backup-7-4-beta/p1

Any update on this?

It's available for stand-alone users in the 7.2 agent release. It's also available, by request, for Managed Backup users with Windows agent version 7.2+. Please contact Support to request the feature be enabled if you want to play with it from the agent-side. It's not yet configurable from the management console. We are planning to officially release mid-October. You can read more about the feature as implemented for stand-alone users here: https://help.msp360.com/cloudberry-backup/backup/about-backups/gfs/immutability

It seems that it only the GFS full backups are made immutable. Is there any way to apply this to the incremental backups based on a given full backup?

That's true. But the full backups are synthetic fullls using all the bits and bytes already in the cloud to create the next set. So if you create a full backup every week then you'll have a new immutable backup weekly. Does that help? If not please describe your retention settings and a little more detail and maybe I can offer an alternative solution.

I was looking at Arq and how they handled this, just as a reference. If I'm reading this correctly, they set the object lock time period to be the sum of the specified "keeping" time for the backup plus the interval between full backups or their equivalent "refresh", if I understand correctly (not certain I do perfectly). Anyway, the proposal I'd make is that you set the object lock time period to be the sum of the keeping time and the interval between equivalent backups, and for each incremental you set the object lock time to basically the keeping time for the relevant full backup plus the time to the next full backup. That way all files in the full+incremental are kept for the GFS keeping time, at least and any incremental backup is available. If one is trying to protect against ransomware by having backups, this just protects the files created between full backups. I kind assume in the ransomware case that the last backups are suspicious and I will need to go back to a restore point that precedes the attack. Protecting the incrementals with immutability means I can pick a restore point that is closer to the time of attack and rescue more files.

I haven't set up real backups with the new backup format yet. I'm not 100% clear that you recommend switching to it now. That is, is the feature fully ironed out. I know it is just off beta, was beta until fairly recently.

Given that each full backup is adds a substantial amount to my storage, I haven't settled on the right compromise between having a lot of GFS full backups to restore to and the extra storage for that. I might think that doing only say annual full backups and having incrementals provide the restore points across the year with a keeping period of just under a year would be a good solution where the backup storage would only be about twice my dataset size plus some for all the incrementals. Perhaps you might suggest weekly full backups with keeping period of say 2 weeks. I don't really know. My point is that not protecting the incrementals with immutability seems to make the feature less complete.

for reference https://www.arqbackup.com/documentation/arq7/English.lproj/objectLock.html

Here's my proposal for immutable. You know the time to purge for each full backup (it shows in the storage view), so when you do an incremental backup on top of that full backup, set the immutable time on the incremental to the time to purge of the underlying full backup.

That's useful in a lot of cases I suppose, in our case we are using MSP360 with Amazon/Glacier as a tape alternative for a series of existing SQL maintenance plans which produce full, differential and transaction-log backups, and have no need or desire to do bit-level backups of the whole system (also since VSS/shapshoting of those VMs causes too much transaction latency for production workloads) , so we end up using incremental backups in MSP360 to back up those full/diff/log files.

Basically, we have these servers that populate with backup files which are then automatically deleted after several days; between creation and deletion, they need to be backed up for long-term storage, which is where we need immutability. Currently we have one incremental 360 job for full/differential backups and a second for log backups.

Our auditors recently flagged this as an area of concern, since currently these are not immutable.

SQL Server is not yet supported by the new backup format. If you are using the SQL Server edition, then you are running native SQL Server fulls, differentials (optionally), and transaction log backups (optionally). For SQL Server, you are using the best option to protect your databases.

In your case, I assume you are either tiering your data using a lifecycle policy to back up to S3 Standard and automatically moving those backups to S3 Glacier for archival purposes on some schedule or you are targeting S3 Glacier directly. Even though we do not natively support Immutability with SQL Server, you may be able to do this:

Back up the SQL Server databases to a NAS or Local Disk using the SQL Server agent and then create a second file backup plan using the new backup format that would take those backups on the NAS / Local Disk and back them up to Amazon S3 Glacier. You could then use object lock on the cloud data.

I have added your request for Immutability support to the existing item in the system for SQL Server.

Yes, we're using the SQL native full/diff/t-log scheme as you say, then using MSP360 incremental jobs to backup the files in place of a previous VEEAM/tape arrangement.

Really that's all we want or need, just a way to punt files to Amazon on a schedule and get email alerts when it fails for whatever reason. Don't have a use case for SQL integration per se, these files could be generated by anything (e.g., mariadb).

I get the feeling there's a way to do what I'm looking for (immutability for files on the job which currently backs up the full/diff .BAKs), but I'm not sure how that translates to GFS settings. Intuitively I feel like we have a very simple use case?

I am not clear on your use case. Are you using MSP360 to back up the SQL Server databases? It sounds like you are running native SQL Server backups rather than using our SQL Server agent. We do have a SQL Server agent and run native SQL Server backups in the background and then process for retention while being built into the overall management structure of the product.

Regardless, if you want immutability, regardless of the file types you are backing up, then you can do what I said in my previous post and use a File Backup using the new backup format to Amazon S3. Use the GFS settings on the backup to define how long you want to keep backups and in what granularity (weekly, monthly, yearly) and you should be good to go. The product will protect all files being backed up using those GFS settings. If you wanted, you could also use multiple file backup plans for the various files you are protecting so each backup set is smaller and can have a customized retention.

I think my biggest stumbling block is just understanding what is meant by full vs incremental backups in this context; I understand the fulls are synthetic, but if we produce (for example) 1 TB of SQL backups per day, delete them after a week and store each on Amazon to pull individual backups as needed, then when the synthetic full runs once a month, are we going to have a 30 TB object in Amazon as a synthetic full? It would be paying for a lot of storage in an unusable form in that case.

a full, in your case, would be all of the files that are selected for backup. If you have some retention that's removing old backups from that folder then they obviously would not be a part of that. And if you had different backups for different databases, as an example, then each full backup would be for an individual database. An incremental backup in your case would just be any new files that are created as it's unlikely that any sql server backups that are already created are going to be modified in some way. Unless you're appending transaction log or other backup files to one another, which I generally do not recommend. then, as you create new transaction logs or differential or full backups of the databases they would be considered as a part of the incremental backup. At the end of say a month if you're using a 30-day cycle a new full would be created based on the data that's currently on your backup folder locally, while using the bits that are already in the cloud to help create that next full.

How much data you decide to keep in the cloud is going to be up to you, and likely based on your retention needs.

I don't know how you're managing the deletion of old backups if you're not using our agent to do the SQL Server backups. But let's just assume that you're intelligently removing old backup files that are no longer needed, and treating every backup set for SQL Server as a full plus any differentials plus transaction log backups - that way you're not removing a differential or a full while leaving transaction log backups in the backup folder. If that's the case then you would end up with backups in the cloud that contain all the files that are needed for a restore. Of course, if you're doing it that way you'd have to restore the files locally and then restore them to SQL Server using native SQL Server restore functionality. You couldn't restore directly to SQL Server from the cloud because you're not using our SQL Server agent.

If your retention needs are different based on the databases that are being backed up, then you would probably be best served by creating separate backup plans for each database (assuming you're backing them up to different folders) that match your retention needs in the cloud. That would keep each full backup smaller because each backup would only be dealing with a single database.

The number of fulls that you actually keep in the cloud is going to depend on your full backup schedule, your setting for Keep Backups For, and if you're using GFS what type of backups you're keeping (weekly, monthly, yearly) and how many of each.

Feel free to reply back with more details on retention needs and maybe we could dial in the right settings for your particular use case..

In my case, the SQL backup maintenance plan is as follows:

Full - Weekly
Differential - Daily
T-Log - Hourly

Each goes into a folder like Backups\Full\DBName, Backups\Diff\DBName, Backups\Log\DBName. Each backup goes into a unique file with a timestamp in the name.

Once per week, just before the full backup, a job runs to delete all the previous week's backups off of disk.

By such a time, the assumption is that all backups are stored on Amazon, in the case of Full/Differential the last ten years, and in the case of t-log the last 90 days.

Several times per week we need to restore a full, diff, and log from Amazon to reproduce reported problems.

With respect to the GFS settings, what would I set for retention to get closest to my goal, assumption I do weekly 'full' and hourly 'incremental' on the MSP360 side?

how are you currently restoring from S3? Do you copy the files from S3 down to local disk and then perform a restore, or do you have some way to restore directly from S3?

Using the new backup format we store things in archive files, so there won't be a list of files that match what you have locally in the cloud. The way you have things set up, you may want to have different backup plans for each database, but I think you could include the t-log backups if you wanted since most backups are removed every week.

As a side-note, if you're deleting your backups locally after the next successful full backup, then you're only leaving yourself with one days' worth of backups since you're deleting the previous week. But if that process is working for you, that's fine. I would be a little concerned about that, especially since you're backing up locally and presumably you have disk space. Restoring from the cloud is certainly going to take a lot longer, and if you find yourselves restoring a few times a week, I'd be looking to keep more backups locally to avoid the need to restore from the cloud. But more backups kept locally means larger full backups on our side, so there's a trade-off.

You can run the fulls as frequently or as infrequently as you want. I assume you don't need all backups for the last 10 years, is that correct? Or are you keeping every single full and every single differential for every single database for 10 years? I guess I would need to know that to provide some guidance on retention settings.

If you're truly keeping 10 years worth of everything, then you don't need to run the fulls very frequently at all. And you may want to run them after the cleanup happens locally to minimize the next full backup size on the S3. You could run them once a month if you wanted. Or you could run them weekly if they're aligned with the local full backup and cleanup. In that case you just going to have a new full backup for the database and the rest of the folder it sounds like it's going to be cleared out anyway. So the synthetic won't really do anything as the only thing being backed up is the new full backup because that's the only file in the folder. And in that sense, a full is no different in what data is backed up than an incremental.

Use the Keep Backups For as an absolute measure of how long you need to keep the backups. Use the GFS settings if you do not need to keep every backup and are looking for longer-term retention, but less granular in restore points - or use GFS if you need immutability.

Interested in if and when B2 cloud immutable backups might be available? From what i have read the API supports it but will it be supported by MS360 at some point? I'd like to try for some of our clients

It's coming in September. Thanks for asking. We'll post here when it's available.

Thanks David. I am testing with B2 now

How about immutable file systems hardware that is local/private cloud based. Anything on that front coming? Also how about immutible backups using Azure storage?

For both Azure and local / private, there are open feature requests. I will add your requests to the system. For local / private cloud, we do support S3 compatible Minio, but we do not yet support Minio Object Lock / Immutability. But since Minio is S3-Compatible, if the feature is added, it will may end up for S3 Compatible and Minio.

Thanks David. I will continue to test with B2. None of my clients have local/private cloud hardware that supports immutability at this time but Azure is something to be considered. My current struggle is finding the configuration settings for clients that maximize protection from Ransomware while not getting surprises with regard to storage costs. Be glad to hear any ideas you have on that. eg what are the best practices?

It's more difficult to protect local data (compared to cloud data) if there's malware running on the network. If data is exposed as a network share, then there's sufficient access to some of the backup data that can put it at risk.

If you had a data center at your MSP that was going underutilized, then you could look at using Minio. Minio exposes local disk as an S3 Compatible cloud and is accessed through the S3 APIs (as opposed to CIFS), which means access needs to use those APIs. You can run it on Linux or Windows.

if you lock down the Managed Backup agents (recommended) by unchecking Enable Backup Agent and uncheck Allow Data Deletion in Backup Agent from Settings - Global Agent Options you can help prevent someone or some malware from deleting backups. You could also uncheck Allow Edit of Backup Plans
and Allow Edit of Restore Plans in Options to ensure no changes are made to plans. You can also make these changes at the Company level in the management console.

You can assign a Master Password to the agents (from Remote Deploy or by editing an endpoint directly in Remote Management - Edit - Edit Options), if desired - if you need to keep the agents available or so the password is needed should you temporarily enable an agent.

Saving locally is fine, but we always recommend using the public cloud (or Minio at the MSP) as a secondary target for backups.

Immutability is available with the new backup format and is tied to GFS retention settings. Dial in how many backups of each Period (weekly, monthly, yearly) you need and they will be locked down with Object Lock if that feature was enabled when the bucket was created and enabled for the backup plan. Object Lock prevents deletion of the data before the GFS period expires. The key here is not to keep more backup sets than you need to satisfy your customers. Depending on the customers, you may need to adjust GFS settings accordingly. Obviously, the more backup sets you keep, the more storage is needed, but if your customer needs monthly backups for 12 months and yearly backups for 3 years, then that's what they need and you can have that conversation up front to ensure there are no surprises on storage costs as time goes by and storage grows.

The strategy we use to protect our local backups.
- Put the backup USB HD on the hyperV host and then share it out to the guest VM’s.
- Use a different admin password on the hyperV than is used on the guest VM’s ( in case someone gets into the file server guest vm where all of the data resides).
- Do not map drives to the backup drive.
- Use the agent console password protection feature including protecting the CLI).
- Turn off the ability to delete backup data and modify plans from the agent console ( company: custom options). You can always turn it on/ back off as necessary to do modifications.)
- Encrypt the local backups as well somthat if someone walks off with the drive it ismof no use to them.
We also do image/ VHDx backups to the cloud and file backups to not one, but two different public cloud platforms.

Currently we use Rundeck to call your product on the command line to request download from Amazon of the specific full/diff files that we need, so they can be restored in an automated process.

You can run the fulls as frequently or as infrequently as you want. I assume you don't need all backups for the last 10 years, is that correct? Or are you keeping every single full and every single differential for every single database for 10 years? I guess I would need to know that to provide some guidance on retention settings.

Yes, we're keeping every full and every differential of every database for 10 years. We have a similar pattern (with 6-12mo retention) with some other application or image-based server backups where in software we have weekly or monthly full backups and then weekly or daily differential backups, going to local storage.

We do not currently have any use cases with MSP360 where we need to restore an entire system to a specific point-in-time, we merely need certain specific files (which are immutable and don't change) downloaded.

The confusion has been how to set GFS settings so we can do recovery of the specific backup files we need for a given database, application or server without having to restore everything that was ever backed up, or incurring additional Amazon storage costs. I feel like this should be simple but for some reason it's just not clicking for me.

I haven't set up real backups with the new backup format yet. I'm not 100% clear that you recommend switching to it now. That is, is the feature fully ironed out. I know it is just off beta, was beta until fairly recently.

The team is constantly working to add more features to the new backup format. It's been out of beta since May 2022. I would recommend you try it out and let us know how it works for you.

As for your dilemma, what you really want (and unfortunately cannot have) is the legacy format sent to Immutable storage.
Why do you need immutable storage?
Is the customer demanding that? Or are you just trying to us Immutability to protect your data?