Forum tip: Always check when replies were posted. Technology evolves quickly, so some answers may not be up-to-date anymore.

Maximum upload speed to AWS S3 over AWS Direct Connect?
  • Simon Nelson
    0
    Hello,

    We're currently trialing MSP360 Ultimate (on Windows Server) attempting to see if it is a potential solution in our use case.

    We have to transfer around 70TB from on-prem to S3 weekly (Oracle RMAN backups). They're currently backed up locally using RMAN to an NFS device, and from there we need to get the files up to S3 for longer-term retention. (We've considered just using AWS Storage Gateway for this but we'd like to benefit from the compression of a backup application can offer as every application we've tested gets a solid 3:1 compression on this data).

    We have a 5Gbps AWS direct connect, but so far we're only able to push 500 Mbps to 1Gbps from our MSP360 VM.

    We're not seeing any obvious bottlenecks on the VM or the on-prem net.

    Increasing the chunk size to 256MB and the thread count to 12 and then 24 seems to have marginally improved the speed but it's still far from ideal.

    12 CPU VM is around 60% utilization no matter what we have the chunk size or thread counts set to and RAM utilization is acceptable too - around 40-50%.

    Can someone share if they know, is there a method to achieve closer to 5Gbps throughput to AWS S3 over an AWS Direct Connect without using the AWS acceleration service? And if so, do you have any suggestions for achieving those speeds with MSP360? Should we separate the backup out into multiple jobs (That's the next thing we're going to try)? I would think that increasing the thread count and chunk size would have the same effect, but perhaps not. Should we run multiple VMs with MSP360 on it, perhaps on different hosts? I wouldn't think this would be necessary since we don't appear to be taxing the VM or the host with just one VM, but again, perhaps someone who knows more could offer some advice here.

    Thank you!

    Simon
    Simon Nelson Options
  • David Gugick
    118
    ↪Simon Nelson
    I guess these are the variables (or some of them, at least):
    1. Speed with which data can be read from the NFS volume by the Windows Server
    2. Network speed / adapter settings for the Windows Server VM to AWS Direct Connect
    3. Maximum ingest rates for Amazon S3
    4. MSP360 Backup processing

    Regarding these items:
    1. Test your read speed from the NFS volume by the Windows Server. You could try copying a file from a Command Prompt on the Server for a file on the NFS volume to the NUL: device - to eliminate write speeds on the VM. Just select a file that has not been read recently and in cache. Something like this should suffice:
    ECHO %time% & COPY /B "AFileOnNFSVolume.BAK" NUL: & ECHO %time%
    2. Check your VM to see how the Network Interfaces are configured. Are they capable of speeds greater than 1 Gbps?
    3. I've done some digging around the internets in the past and have never been able to find definitive docs from Amazon on maximum ingest rates for S3. I would say that 1 Gbps is extremely fast hitting a single bucket, but i think this is probably a good question for you to ask your technical rep at Amazon. It's possible that even though you have a 5 Gbps link to AWS, it's not possible to get that rated speed when writing to S3
    4. On the MSP360 side there are a few things that can affect throughput:
    • If you are using RMAN to compress the backup (recommended) then re-compressing with MSP360 is probably a waste of CPU cycles. Consider disabling compression, if enabled
    • If you are encrypting, that will use CPU cycles as well - but I'm not recommending you disable if the backup is not already encrypted using Oracle Secure Backup.
    • With 12 Virtual CPUs, I'd consider using 10 or 11 Threads at the most to avoid unnecessary thread switching on the CPUs. Set Chunk Size accordingly - maybe 100 MB - but higher could be a little better as it does reduce some latency
    • Make sure Block-Level backups are disabled (Advanced Scheduling Block-Level). Just have incremental selected. The RMAN backup files are not going to be updated, so no block-level analysis is needed during backup for those files.
    • Make sure Logging is set at LOW or DISABLED. We can use logs to help diagnose issues and they'll need to be at LOW (at the lowest), but for maximum speed, you could try disabling the logging (Options - Logging)

    That's all I can think of right now. Let us know what you find.
    David Gugick Options
  • Simon Nelson
    0
    Thanks for the response. This is great.

    I hadn't thought about creating additional buckets. I'll give that a shot, as wel las contacting AWS support. I couldn't find an answer anywhere either about what the maximum speed of S3 was, let alone specifically over a direct connect.

    Running multiple backup plans in parallel did increase the speed somewhat. Seems to get a solid 700 Mbps now, but definitely needs more CPU. The VM is setup with a 10Gbps NIC, and the host is quite capable with 40 Gbps to the network, so that doesn't seem to be an issue.

    RMAN is encrypted and uncompressed to the NFS device, so I'm having MSP360 do both. I'll check the block-level backups setting as that's a good point about them not changing.

    Logging is set to low.

    I'll give all this a shot. Thanks!

    Simon
    Simon Nelson Options
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment