Hi,
I am rebuilding after a ransomware breach where the attackers got onto a server and appear to have used the cloudberry GUI or CLI software installed on the breached server to ALSO delete the backups from the cloud storage.
I am looking for advice and guidance on steps to take to ensure this is not possible in the future, both against hackers and malicious insiders who have gained admin access to a device being backed up.
[reply=“db ots;d885”] If a malicious hacker deliberately destroyed data, they would have to know about CloudBerry and where the backups were stored. Makes me think this was someone known. You can secure the agent with a master password - that feature is used to prevent end-users from using the agent. This feature can be automated from the console using the RMM - Remote Deploy - Create Configuration option. Create configurations for Windows and Mac/Linux and enable the Protect Console with Master Password option. Then create a rule for each config that deploys the settings to all customers (or a subset as needed).
The rest, I think, is security related like not storing passwords in web browsers to prevent someone from logging into the cloud storage vendor via a web browser, disabling credentials as a part of the process when employees / consultants leave, changing any shared passwords if you use them, etc.
You can also enable IP Address White Lists in the console to prevent unauthorized access via the web console from unknown IP Addresses.
And lastly, notify the authorities. If it was someone close, they can probably find out who.
We had an nearly identical situation a few months back. A hacker got into a clients server console and installed ransomware, and then proceeded to delete the backups from the Cloudberry console. We were very fortunate to have taken a separate local backup the day before that was not accessible to the hackers. It saved out ssss.
Since then we have set master passwords on all client consoles AND disabled the ability to delete backups from the console. This was done in the advanced branding options settings.
Going forward if we need to delete old backups , we use Cloudberry Explorer and run a Repository sync afterwards. It is more work, but given that it is not often we have to delete old backups, we opted to err on the side of more security.
And we also put passwords on Cloudberry Explorer for Azure, Google and Amazon, to protect against our own server being hacked.
[reply=“David Gugick;3118”]
“they would have to know about CloudBerry and where the backups were stored”
That would be trivial once once they had remote access to the server. The running process list, not to mention the start menu shortcuts would make it pretty clear it was present. Knowing that, you can get a list of the backup plans via the CLI, or via the GUI and unravel all the details.
I’m aware of the existence of a master password for the gui console.
Today I set up a test master password on the console, and confirmed I was still able to run several (non-destructive) CLI commands without obstruction. I haven’t had time to dig further, but it looks like the CLI bypasses the console password??
[reply=“Steve Putnam;3119”]
Thanks for mentioning the option to remove the delete option via the advanced rebranding options. I was not aware that was there!
Like my response above though, Do we know this definitely protect’s against deletions via the CLI.
Delete files/folders from cloud (delete)
cbb.exe delete
https://www.cloudberrylab.com/resources/documentation/cli-api/cloudberry-backup/
And this was on windows, I am now also concerned about mac and linux.
[reply=“db ots;3122”] This option can only remove account from the software, it does not affect the data in the cloud.
[reply=“Matt;3129”]
Matt, I beleive you are badly mistaken. I suspect you just looked at this command:
cbb.exe deleteAccount
If you scroll down much further, there is a separate command:
cbb.exe delete
And the description is: "Delete files/folders from cloud
[reply=“db ots;3132”] Yeah, sorry, I was only referring to account deletion.
Checked out issues tracker and the option to disable CLI commands for rebranding is already in the works.
[reply=“Matt;3135”]
Thanks for clarifying.
It sounds like you are confirming that at present, there is no way to prevent a malicious attacker from deleting the cloud backups if they have breached a system being backed up.
That’s pretty terrifying. We had backup retention and lifecyle rules in place, and it was all purged.
I am now looking at re-setting up our backup target from Backblaze B2 to Amazon S3; because S3 has a cross-region-replication service that appears to let us replicate our S3 buckets onto a separate amazon account, with lifecycle rules that will ensure that even if the first bucket is maliciously purged, the replicated bucket will be beyond the reach of the cloudberry CLI.
As B2 doesn’t seem to support this, that appears to disqualify them as a suitable cloudberry backup target right now; unless there are other strategies for ensuring the security of the backups.
Do you have any suggestions, best practices, or other guidance on this?
[reply=“db ots;3136”]
You can use file name encryption for your file backups, so that when you try to delete these files you need to provide a decryption password. Such files can’t be deleted via CLI.
As for the choice of storage, S3 would be the most optimal one in terms of security across the board.
[reply=“Matt;3137”]Thanks Matt;
I’d realized filename encryption provided a good defense against senstive data leaking through filenames, for example your tax return file named with your SSN or something. Even without being able to view the file… information i didn’t want disclosed is visible.
It’s really interesting that it provides a defense against remote deletions though.
You wrote:
“Such files can’t be deleted via CLI.”
Can you clarfiy: Files with encrypted filenames can’t be deleted via CLI period? Or they can’t be deleted via CLI without the passphrase?
This actually raises two more questions:
Does this side effect of filename encryption apply to VM backups, and block based image backups as well? Or just file based backups?
Where/how is the encryption passphrase actually stored and how secure is it from being extracted on a compromised system. I know, for example, that the file encryption feature is a solid defense against the data being accessed if say the cloud storage account itself is breached (**see below) to prevent the data being decrypted.
However, it appears the pass phrase is stored as a hash in the .cbb settings file for the backup plan.
2a) What is the nature of the hashing algorithm used? How is it secured / salted ?
2b) Can anything be done with the data if you know that hash? My main concern here is along the lines that if cloudberry just uses the hash as the “real key”, then maybe i don’t really need the real key for some types of attack as long as I have the hash. Perhaps I can modify the cbb backup plans by hand in an editor, turn off retention, change what’s being backed up… is that feasible?
** As for the potential for the cloud account storage account being breached. That risk is mitigated by having redundant separate cloud storage accounts setup. Or Cross-region replication via S3.
Thanks for your time on this. I know some of these scenarios may be a bit far fetched, but after dealing with this attack, I want to make sure i understand exactly what is and is not possible, and how to guard against future attacks.
[reply=“db ots;3138”] If you try to delete file name encrypted files via CLI the software simply will not do that, GUI is required to delete such files.
Regarding the rest of your questions:
Image-based/VM backups can’t be deleted via CLI, so you should be safe here.
You can read about our encryption mechanism here
There’s too much info to simply copy and paste everything in the thread, so that should actually answer all your questions
Not really, and brute-forcing hash is not a smart idea.
Hope that this info helps in making your setup more secure.
[reply=“Matt;3140”]
Thanks Matt for this additional information.
Hope that this info helps in making your setup more secure.
Unfortunately, no, it does not; this thread sort of went down a tangent about the CLI; since I was aware of the console master password; and my data was deleted anyway. So I speculated that perhaps I’d actually left the gui open on the breached server, or that they’d done the damage via CLI.
However, I did some more testing, with a gui console password set. Then I located the enginesettings.list file changed the line that said:
cCHNBqBHVh/n6hU+o9jwKEvpVV7SkuJEdw7J8AOnsFEcn0HORaWuXtP1IqtagwdZ
To:
I then saved the file. Then I launched the backup software, it opened right up, I and was no longer challenged for the console password. AES encryption of the master password is beside the point if I can simply turn if off with notepad.
So even if I had a master password set, and even I had dutifully been sure to close the software prior to the breach, it would have been trivial for the attacker to bypass the master password anyway.
For all intents and purposes then if a system being backed up is breached the gui console master password provides no tangible security for the cloud backups.
[reply=“db ots;3144”] GUI can be disabled altogether and you’d actually need to know what to do to achieve these results. Some improvements are on the way regarding that and I’m going to bring this up to our R&D’s attention.
Technically, if the system is breached, nothing is really safe, so separate security investments are the best way to be safe in the modern world.
[reply=“Matt;3148”]
"GUI can be disabled altogether "
How is this gui disable feature actually implemented? Can it also be turned back on by editing an xml file in notepad…? My faith in these ‘security features’ is justifiably pretty low right now.
“you’d actually need to know what to do to achieve these results.”
I found out how to bypass the cloudberry master password on a 3 year article on stackoverflow.
Hackers charging 10s of thousands in ransoms are motivated to learn. Given that this happened to us, and Steve Putnam early in the thread reported a ‘nearly identical thing’ happened to them – this has moved beyond the theoretical.
“Technically, if the system is breached, nothing is really safe, so separate security investments are the best way to be safe in the modern world.”
The cloudberry account wasn’t breached. The storage provider wasn’t breached. I’m willing to make additional separate security investments – I consider using cloudberry backups as one of those investments.
Now if the cloud backups can be completely destroyed beyond recovery from the machine doing the backups by a malicious actor; that’s something that needs to be called out and clearly identified as a risk; and clear reliable strategies to mitigate it need to be identified.
Something more than a master password that can be trivially bypassed.
[reply=“db ots;3160”] As I mentioned, I’ll bring these points to the head of our R&D, the meeting will be next Wednesday, will have an update by the 23rd.
We will make CLI optional on the machine like we have GUI right now, to be configured on Advanced Rebranding page.
Received an update from our devs.
Issue with Master Password security will be fixed in version 6.1.1, ETA is end of June/beginning of July.
After reading this thread, I’m extremely concerned with the CLI commands. Just want to clarify, if image backups are being done, the CLI delete command wouldn’t be able to delete these? I’d love the option to disable CLI entirely as mentioned previously.
We’ve taken the following security measures. Disable delete from console, 2FA for our account login, IP restrictions so only approved IP’s can login to our account. Different Master console password for each client (though this appears to be easily exploitable for the time being).
Ransomware is obviously the largest concern now days in regards to backups, so I want to ensure our online backups are secure as possible since they very well may need to be used in a situation like that.