Maximum upload speed to AWS S3 over AWS Direct Connect?
We're currently trialing MSP360 Ultimate (on Windows Server) attempting to see if it is a potential solution in our use case.
We have to transfer around 70TB from on-prem to S3 weekly (Oracle RMAN backups). They're currently backed up locally using RMAN to an NFS device, and from there we need to get the files up to S3 for longer-term retention. (We've considered just using AWS Storage Gateway for this but we'd like to benefit from the compression of a backup application can offer as every application we've tested gets a solid 3:1 compression on this data).
We have a 5Gbps AWS direct connect, but so far we're only able to push 500 Mbps to 1Gbps from our MSP360 VM.
We're not seeing any obvious bottlenecks on the VM or the on-prem net.
Increasing the chunk size to 256MB and the thread count to 12 and then 24 seems to have marginally improved the speed but it's still far from ideal.
12 CPU VM is around 60% utilization no matter what we have the chunk size or thread counts set to and RAM utilization is acceptable too - around 40-50%.
Can someone share if they know, is there a method to achieve closer to 5Gbps throughput to AWS S3 over an AWS Direct Connect without using the AWS acceleration service? And if so, do you have any suggestions for achieving those speeds with MSP360? Should we separate the backup out into multiple jobs (That's the next thing we're going to try)? I would think that increasing the thread count and chunk size would have the same effect, but perhaps not. Should we run multiple VMs with MSP360 on it, perhaps on different hosts? I wouldn't think this would be necessary since we don't appear to be taxing the VM or the host with just one VM, but again, perhaps someone who knows more could offer some advice here.
I guess these are the variables (or some of them, at least):
1. Speed with which data can be read from the NFS volume by the Windows Server
2. Network speed / adapter settings for the Windows Server VM to AWS Direct Connect
3. Maximum ingest rates for Amazon S3
4. MSP360 Backup processing
Regarding these items:
1. Test your read speed from the NFS volume by the Windows Server. You could try copying a file from a Command Prompt on the Server for a file on the NFS volume to the NUL: device - to eliminate write speeds on the VM. Just select a file that has not been read recently and in cache. Something like this should suffice:
ECHO %time% & COPY /B "AFileOnNFSVolume.BAK" NUL: & ECHO %time%
2. Check your VM to see how the Network Interfaces are configured. Are they capable of speeds greater than 1 Gbps?
3. I've done some digging around the internets in the past and have never been able to find definitive docs from Amazon on maximum ingest rates for S3. I would say that 1 Gbps is extremely fast hitting a single bucket, but i think this is probably a good question for you to ask your technical rep at Amazon. It's possible that even though you have a 5 Gbps link to AWS, it's not possible to get that rated speed when writing to S3
4. On the MSP360 side there are a few things that can affect throughput:
If you are using RMAN to compress the backup (recommended) then re-compressing with MSP360 is probably a waste of CPU cycles. Consider disabling compression, if enabled
If you are encrypting, that will use CPU cycles as well - but I'm not recommending you disable if the backup is not already encrypted using Oracle Secure Backup.
With 12 Virtual CPUs, I'd consider using 10 or 11 Threads at the most to avoid unnecessary thread switching on the CPUs. Set Chunk Size accordingly - maybe 100 MB - but higher could be a little better as it does reduce some latency
Make sure Block-Level backups are disabled (Advanced Scheduling Block-Level). Just have incremental selected. The RMAN backup files are not going to be updated, so no block-level analysis is needed during backup for those files.
Make sure Logging is set at LOW or DISABLED. We can use logs to help diagnose issues and they'll need to be at LOW (at the lowest), but for maximum speed, you could try disabling the logging (Options - Logging)
That's all I can think of right now. Let us know what you find.
Thanks for the response. This is great.
I hadn't thought about creating additional buckets. I'll give that a shot, as wel las contacting AWS support. I couldn't find an answer anywhere either about what the maximum speed of S3 was, let alone specifically over a direct connect.
Running multiple backup plans in parallel did increase the speed somewhat. Seems to get a solid 700 Mbps now, but definitely needs more CPU. The VM is setup with a 10Gbps NIC, and the host is quite capable with 40 Gbps to the network, so that doesn't seem to be an issue.
RMAN is encrypted and uncompressed to the NFS device, so I'm having MSP360 do both. I'll check the block-level backups setting as that's a good point about them not changing.
Logging is set to low.
I'll give all this a shot. Thanks!
Sign in or register to add a comment.
Add a Comment
Welcome to MSP360 (CloudBerry) Forum!
MSP360 Managed Backup
Microsoft SQL Server
MSP360 Backup (stand-alone)
Microsoft SQL Server
MSP360 Remote Desktop
Backup to AWS S3: several customers/users: do I need bucket rules?
Support for AWS S3 Glacier Deep Archive?
If we use AWS-CLI to copy files to the S3 bucket, can ClouldBerry Drive work with them properly ?
Copying directly from AWS S3 to Wasabi?
Terms of Service
Useful Hints and Tips
Created with PlushForums
© 2020 MSP360 (CloudBerry) Forum