Purging files halts all other jobs
I have multiple jobs running on a server (using S3 as repository) and every time one of the jobs goes into "purging files", it halts all other jobs until the purging is done (usually 25-30min).
is this normal procedure?
Here is what the log shows
2022-09-28 10:08:53,901 [PL]  NOTICE - Creating purge queue started. Purge conditions: version count: 10, delete topurge elements
2022-09-28 10:34:14,163 [PL]  NOTICE - SQL query 'SELECT id, local_path FROM (select local_path, cloud_files.id, COUNT(*) as cnt, MAX(cloud_file_versions.date_modified_utc) as last_modified_date from cloud_files inner join cloud_file_versions on
= cloud_file_versions.file_id where destination_id = ? and not exists (select 1 from ransomware_file_versions rfv where rfv.file_id =
) group by
) WHERE cnt>1 '. Parameters: '[param_1]='1'' takes in total: 00:25:20
2022-09-28 10:34:47,782 [PL]  INFO - Files to purge with versions found: 2761281
2022-09-28 10:35:16,906 [PL]  INFO - Folder markers to purge found: 0
2022-09-28 10:35:16,906 [PL]  NOTICE - Purge version search time: 00:26:23. Files checked for this plan: 16883. Files for purge found: 0
What do mean, “halts other jobs”? Do you mean other scheduled backup plans do run run on schedule until the purge is complete, or do you mean other plans that are executing in parallel stop processing while the purge is working?
Also, please post your version.
Let me add, that based on the information I see in the log, that you’re purging / checking 2.7 million files. That is going to take a lot of time, it’s possible that the agent is tied up with all the updating. You may want to consider moving to the new backup format which can do these purges very quickly if this is a common situation where you’re removing / checking that many files in the cloud. But give me an answer to my questions above first and we’ll take it from there. Thanks.
Plans that are executing in parallel halts until purge is working.
This is what I see in their logs while it waits for the other plan's purge to complete
[CL]  WARN - Generating chunks is idle for 15 minutes.....
you’re going to have to open a support case likely to get firm resolution, but it sounds like the PC where you’re running backup is tied up with all the processing. If the plans are pausing, it may be because of all the updating and checking that’s going on in the local repository database for the 2.7 million files. The new backup format would address that particular situation. Having said that it what’s the issue with the temporary pause on running plans while the long purge operation runs? Is it pushing you outside your normal operating times? Are the plans ultimately failing because of the pause? Something else?
See the additional questions I put in my original reply which I was probably replying to while you were replying to me.
pushing outside of normal operating times. I'll open a ticket.
is over 70GB which it has been for some time but this issue just started a couple of days ago.
The issue of the purge operation taking that long? Or the purge operation temporarily pausing other running plans? I am just wondering if the purge is now taking much longer than previous and that's why you are noticing the issue - whereas if it ran in 5 minutes maybe you would not have noticed.
What version of the product are you running?
Were there any changes in plan scheduling prior to noticing this issue?
Were there any major changes in the data being backed up - a lot of files being added or moved between folders?
Did you recently upgrade versions? If so, from what to what?
Any new processes, applications, or services running on that server that may be using CPU / Disk and causing contention with the purge operations?
Anything else you can think of?
Sign in or register to add a comment.
Add a Comment
Welcome to MSP360 Forum!
MSP360 Managed Products
Managed Backup - General
Managed Backup Windows
Managed Backup Mac
Managed Backup Linux
Managed Backup SQL Server
Managed Backup Exchange
Managed Backup Microsoft 365
Managed Backup G Workspace
Backup for Linux
Backup SQL Server
Connect Free/Pro (Remote Desktop)
Cloudberry backup on One Drive - Searching modified files taking long time (3 day) and not completed
Storing files using CloudberryDrive results in 0-byte files
Backup for Linux fails to backup files with diacritics in file names (Spanish, French, German etc.)
Purging deleted files from backup
Terms of Service
Useful Hints and Tips
Created with PlushForums
© 2023 MSP360 Forum