problems with BATCH DELETEs

MSP360 Backup
1

I’m facing an interesting problem with MSP360, in which some backups completes with warnings, being the warnings similar to:

Warning

Purge error (code: 1047)
An error occurred while attempting to purge data on backup storage: Error on deleting folder ‘s3-bucketname/CBB_SRV-DC01/CBB_Archive/a7345bf3-ae7d-48dc-b9a3-72248d959655/GEN-000000-08392e9b-098a-4f37-bf25-b452ea70d038/20230705030116_000003’

To further investigate, I activated bucket logging and could see the '403 Access Denieds.

  • log from MSP360 access - results in Access Denied
    2023-07-05-20-37-31-EBAB…1ad cityinc-2023-bi [29/Jun/2023:19:45:41 +0000] 187.72.x.x arn:aws:iam::234033xxxxxxx:user/msp360 ATEJ2D9HDYKPHQ6X BATCH.DELETE.OBJECT CBB_SRV-DC01/CBB_Archive/bf1fa97c-81ba-4977-ba96-da61069ccc66/GEN-000000-76fd79e5-5093-4675-80b2-c0090f2d84fc/20230627030546_000003/archive.000001_toc_ok.cbl -403 AccessDenied - - - - - - rvoubh9MS4mWSsQFF9KA9WNSerO_dptJ udVN0sUSqv466cV9vPPo2cTCDW8rf9Fsh+ZUQZBoM3X+onsoWxo0KTQ9dySQs5rFA4YKvsyuPIWz5Bfi2Ssqeg== SigV4 ECDHE-RSA-AES128-SHA AuthHeader bucketname.s3.eu-west-1.amazonaws.com TLSv1.2 - -

the “BATCH.DELETE.OBJECT” seems to translate to the delete-objects (yes, objects in plural) API call. There’s also delete-object in singular.

Well, “Access Denied” is easy. Initially I expected this to be some missing permission, on the policy I created and applied to the user/keys used by the MSP360. But after some researching, I found it wasn’t the case. The user had full DeleteObject permissions, which is the one used by delete-object and also delete-objects. There’s no different permission for them.

Tried issuing the exact delete-objects API call, via AWS CLI, using the very same credentials used by msp360, trying to delete the same file pointed on the error … and to my surprise, got success reply, which could be confirmed by the bucket logging as well

-log from AWS cli, delete-objects, to the same file and using the same access keys - delete OK
2023-07-05-23-18-24-F3B8…1ad cityinc-2023-bi [29/Jun/2023:22:36:54 +0000] 191.22.x.x arn:aws:iam::234033227905:user/msp360 4R96Q8EYJ2H7F544 BATCH.DELETE.OBJECT CBB_SRV-DC01/CBB_Archive/bf1fa97c-81ba-4977-ba96-da61069ccc66/GEN-000000-76fd79e5-5093-4675-80b2-c0090f2d84fc/20230627030546_000003/archive.000001_toc_ok.cbl - 204 - - - - - - - - qWCFvoGCTKJANXa388f2lWc1hJfFxaRZ/CPustSeb4xv5Itkc+5VAgjaZJYxBJsDZuQeEbWHsx8= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader bucketname.eu-west-1.amazonaws.com TLSv1.2 - -

I wasn’t expecting that, I was actually expecting it to also fail to make my testings easier, via CLI call. So it’s not any permission fail after all.

And I don’t have any idea on what to do further, as I keep getting these errors on my backup plans.

Any ideas on why msp360 get errors on the calls, even the user having all the appropriate permissions to do so, as presented? MSP360 is fully upgraded to latest version

2

new information: I granted “S3 Full Access” to the user, and the “error on deleting folder” still occurs … it really doesn’t seem to be a permission problem indeed.

3

[reply=“Leonardo Rodrigues;10417”] Do you have object lock enabled on your s3 bucket?

4

Yes I do, as we discussed on another post some days ago :slight_smile: Default lock is set to 7 days. Do you believe after 7 days the files/folders will be deleted with no problems?

Anyway, funny is that via AWS CLI, I can delete stuff, using the exact same keys MSP is using, as explained. By that, I figured out it shouldn’t be object lock related afterall. If that was the case, I should get an “Access Denied” via CLI as well, as I’m using the very same access credentials MSP360 is using.

5

object locks are set to “Governance” one, which CAN be overriden for those with the correct permissions. I tried giving “S3 Full Access”, which contains the needed permission to override Governance Object Locks, and error remained. That’s another reason for me to believe it’s not object lock related.

6

[reply=“Alexander Negrash;10421”] after some debugging myself, i’m pretty confortable on generating and sending you, if you believe it could help, S3 bucket logging and also CloudTrail logging from the MSP360 request that got Access Denied, and the same request being made via AWS CLI, using the same credentials and same request values. If you believe that would be helpful, please let me know. In that case, I would might prefer opening a ticket to attach data, because it might contain sensitive information there.