I am trying out MSP to do server backups, and have started an initial backup of about 400Gbytes, including many small files. I am backing up a VM from AWS east, and am using MSP with storage buckets at Backblaze. My question is how long I should expect this backup to take? The system is not heavily loaded, and very little CPU or memory is in use. But data is being processed at < 2MB/s, which implies a long, long time to complete. Subsequent backups will of course be faster.
Is there any reason why the processing is so slow? I am using encryption and compression, BTW. Another question is whether I would be better off backing up directly into buckets at AWS East, rather than going through Backblaze.
And response would be appreciated. Thanks!
Rich
[reply=“Richard G;d2088”] Performance depends entirely on bandwidth variables like: Disk read speed, CPU cores available, threads used in the backup product, network speed, data egress speed from AWS, data ingress speed at Backblaze. It’s hard to say what is causing the 16 Gbit speeds you’re hitting. Both services can support faster speeds than that, so it’s more than likely a VM performance issue.How fast can you read data in the VM? What network speed do your virtual adapters support? How many threads are you using in CloudBerry Backup (in Options | Advanced)? What about file sizes of the files being backed up? Are they small, large?
David, thanks for your response. The backup is now running slower, at 471.54KB/s, according to the web monitor application. Many of the files are, indeed small, some very small like log files and the like, but the progress should still be better.
https://drive.google.com/file/d/1c_9BELUwEoxDTQnQPvwe9wvVnWLh3Z_Z/view?usp=sharing
This Amazon instance is m5ad.2xlarge, with 8 vCPUs (AMD EPYC 7000), 32Gbyte memory and NVMe SSD storage. Here is the view from htop:
https://drive.google.com/file/d/17xiBetX3GnNHupjOk6vGPIU2lORusMTW/view?usp=sharing
As you can see, the backup software is basically the only significant running process.
Can you suggest anything that would help?
Thanks,
Rich
I should add that the file system contains over 5,000,000 files, many of them in pdf and txt format. I see that under the cloudberry directory /CBB_z.softconf.com, so far hundreds of thousands of subdirectories have been created, each one corresponding to one of the 5,000,000 files. So I assume that all of the files have to be individually encrypted and compressed, and then a directory is created for each.
Is this the cause of the slowness, and is there anything that can be done to speed up the process? I need the ability for single file restoration, so backing up an image of the filesystem isn’t the solution for me. (We already have snapshots done with AWS)
Best,
Rich
[reply=“Richard G;7488”] The issue isn’t directories and files as such. With object storage they are in fact the same thing anyway. But what you are going to see is latency issues as each file that needs to be created, whether that’s a object folder or the object file itself is going to have latency as a result of the round trip out of AWS to Backblaze. One way to reduce the effects of the latency is to increase the thread count. You didn’t reply what you have it said to in the backup product, so please reply with that number. Also keep in mind that this is a one-time trip, and even if it takes a few days that may not be a problem for you. But in the interim let’s find out what your threads are set to. You can also set the logging to off if it’s turned on. But even with it turned off I don’t suspect that’s going to make much of a difference with NVMe drives.
I really appreciate the quick response. I just started with MSP yesterday, and was not aware that I could set the thread count. I suppose I am using the default configuration, which is:
config/.default_settings.conf:MinThreadCount : 10
config/.default_settings.conf:MultiThread threads count : 5
config/.default_settings.conf:MultiThread memory usage : 104857600
Would you suggest changing those numbers? And FYI, I am comfortable with the initial backup taking a few days, as long as the subsequent daily backups are faster. If I recall from the Solar Winds MXP product I used, only about 6GBytes were processed daily, most of which consisted of binary database files, which cannot be compressed.
I like the product and will definitely buy the “ultimate” subscription, as long as file restore works the way I expect it to.
Best,
Rich
BTW, I couldn’t find much documentation on installing the product from bash in Linux, until I came across the excellent instructions provided by the Average Linux User here: Cloudberry Backup for Linux Server . You should probably link to this somewhere on product pages, since it is very useful to Linux people, and I found it invaluable.
[reply=“Richard G;7492”] Change the settings from the UI if you can (https://help.msp360.com/cloudberry-backup-mac-linux/settings/advance-settings) - should be similar in the web client.
You can try changing the threads as they appear in Settings - Advanced higher and see if that helps. I can’t tell from what you posted from the config file if it’s set at 5 or 10, but try setting to 20 and see if it helps throughput.
[reply=“David Gugick;7493”] OK, thanks for the info. I am working in a server environment with CLI only, so I will try to find the relevant setting - but not until the initial reference backup is finished. I don’t want to mess with anything while it’s in progress.
Thanks, RIch
David, I have another question. The initial reference backup is still running, which may be expected with the first backup of 7,000,000 files. But when it started a few days ago, the 1Terabyte partition was about 50% used, of which 45% of that was in the backup plan.
The backup is using the same partition for its temporary storage before uploading to the cloud. The problem is that it’s now up to 79% used, and the backup is not nearly over. I am afraid I will run out of space before it is finished.
In mid-backup, is there any way to force Cloudberry to start uploading to the cloud, and release that temporary storage? Or more generally, is there a way to set the maximum temporary storage amount in the CLI on linux. (I saw something to that effect for Windows, but I am working in a server environment.)
Thanks!
Rich
[reply=“Richard G;7514”] I am unaware we use that type of temporary space when uploading directly to the cloud, which I believe you are doing. I do not know what files you are seeing int he temp folder. How much space are we talking about? Do you happen to have logging enabled to High?
BTW, I am checking with Support.
I checked with support, and after viewing the logs, they said that I had not correlated the server’s backup agent with the backup storage selected with my storage account. And hence my error. The only option provided in the server’s backup manager is the “File System”. No cloud source is available. And yet this is what is in my account: 
This backup location is unknown to the Cloudberry agent on my server. So what am I doing wrong?
Rich
[reply=“Richard G;7517”] once you define the storage account, you need to create a backup destination on that same screen, which is really just a bucket name and a region if the cloud storage provider has multiple regions. Then you need to go into the customer in question that you’re working on, and add that storage account at the customer or company level as we call it in the product. Then you create a backup plan and for the storage options for that plan you assign it the storage destination that you just created. If on the other hand you want to use a hybrid backup that backs up to the local network and the cloud at the same time then you would have to make sure you’ve created the file system storage account for the network location where you want the backup saved, and then you would also assign that account to the customer, and finally you would select hybrid as the backup type in the backup plan and make sure that both the local network storage account and the cloud destination are selected as targets.
Thanks, David.
I have been trying, without success, to launch a backup plan. I will go through the steps I used to get there. I can’t figure out what is going wrong.
Adding a storage account:
Need to add user: 
User added: 
Now, from remote management, try to add backup plan: 
Creating backup plan, but destinations are not selectable. 
Anyway, try to run the backup, which results in failure: 
I realize I am doing something wrong, but I can’t determine what it is. I need to go through a simple test backup and restore before I can put this product into production on my various servers. I am usually pretty good at setting up software, but this seems to be particularly challenging.
I finally figured it out. I had to add a new user, and then associate him with the Backblaze account. Then I could login via port 43211 on my server, and finally was able to select the cloud backup location.
[reply=“Richard G;7523”] You don’t need to assign the storage to the user. You can do it that way and that’s the way it was traditionally was done, but as I stated above you can assign the storage to the customer directly if that is more convenient for you to do. But it sounds like you figured it out.
Thanks David. Even on a Sunday! I have one last question. From my experimentation yesterday, I learned the hard way that since I am on AWS, backing to Backblaze will be very expensive. So I tried to add S3 as a provider, but the storage account manager will only allow adding buckets at Backblaze (and local filesystem storage). Is there a way to add other providers? Thanks.
[reply=“Richard G;7528”] When you add a storage account in the managed backup UI, you would need to select Amazon S3 as the storage account. Then you can click the gear icon and add a backup destination bucket in the same region where the data already resides. Then you would add that storage account to the customer (or user as you had done previously) and create a new backup plan that uses the new storage account (or edit the existing one and simply change the storage destination).
Thanks, David. The problem was when I set up my account, I landed on your page via a link from Backblaze. According to support, that is an affiliate agreement - if you are referred from Backblaze you only have the option of using B2. It caused a lot of confusion, which I solved by installing a stand-alone version with a different username. So I’m good now, at least for testing. Thanks for all your help. Rich.
[reply=“Richard G;7532”] I am unaware of such a limitation with managed backup. Stand-alone is not managed and requires a perpetual license after trial. Are you a managed service provider? If not, and you only need a single license, then stand-alone would likely be a better option.