We have been using Cloudberry Drive (v2.4.2.16) on Windows Server 2016 to copy files to Wasabi cloud storage. At present we've managed to copy almost 1TB of data files to Wasabi. 25% of the files are 50MB in size. The other remaining files are 1KB so that's a lot of files we're copying to Wasabi.
We had observed in Task Manager that Cloudberry Drive service seemed to be consuming a lot of memory. At one point it had 8GB allocated. So we stop/quit the service and re-started it and the memory consumption went back down to normal levels. To be safe, we went into the options > Advanced tab and set the "Limit cache size" to 1GB.
Running for about a week, Windows OS informed us that the system was running low on memory. Opening up the Task Manager, we discovered that Cloudberry had almost consumed 80% of the 30GB of total memory available on that VM. Please note we are using Microsoft Hyper-V for our VMs.
This was completely unexpected. Has anyone else observed the same thing with their CloudBerry Drive service? Our first suspicion was that there was a memory leak in the service, but maybe we haven't configured the Cloudberry Drive settings correctly? Most everything is left at the default setting when we installed CloudBerry Drive with the exception of the "Limit cache size" which we set to 1GB.
The engineering team is looking at this issue. I'll report back if I get any additional information. You may want to open a support case for proper tracking.
Can you let us know if over the course of the week, Drive is still syncing files to Wasabi or was the sync complete?
The sync was complete (IE the CloudBerry Drive service completed the mounting) and we are able to access the files on Wasabi through Windows Explorer. The memory usage continues to increase over time and our Engineer has to periodically stop the service and restart it to prevent a system crash.
Thanks David - any news? I have multiple customers relying on this solution for production purposes and they are needing to restart the service at least once a day since the memory consumption continues to grow until it consumes all of it. These customers are running on EC2 instances in AWS and using cloudberry server edition to map to S3 buckets. They are moving many thousands of images through each day since they are processing photo feeds from wire services like AP, gettty, etc....
We would need a dump of the process after it begins to take additional resources. By dump I mean the file created after clicking on "Create dump file" (right-click on CloudBerry Drive process consuming a lot of memory) as it is shown on the attached screenshot. Most likely it will be too large to be attached in email, so you can upload it to a file-sharing service of your choice and send the link to supportatcloudberrylabdotcom
Please put "Drive memory leak. For Matt" in the issue description.
I spoke to support this morning. They said that dev was aware and multiple customers have reported it and it was actively being worked on. So it sound like it has been identified and is a release goal for the next rev.
unfortunately I don't have access to these systems since my customers have stopped using it in production and switched to the AWS storage gateway, so I do not have an environment to assist with.
Hi.
I have Drive v2.4.2 and seem to be suffering from the memory leak problem, as shown in the Windows Event log as "Resource Exhaustion Detection" log entries.
I notice that the latest version is v2.4.3.15 but I cannot find any release notes or similar on your site.
Does this version resolve this particular problem, please?