• Guilherme Pessoa
    0
    I'm sending a large amount of files to the Standard S3. the upload will give a total of 65TB of files with an internet link of 2Gb. all my files are in a QNAP with 10Gb ports, the VM that is installed the CloudBarry backup has 32gb of ram, 10 cores and 10Gb network connection.
    However the upload performance is at a ridiculous rate, at the speed it is going to end in 3 months.

    as an attachment I sent some prints
    Attachments
    config (121K)
    taxa upload (7K)
  • Guilherme Pessoa
    0
    detailing my infra: I'm shipping to aws s3 standard, (Northern Virginia) us-east-1 and I'm in Brazil. The transfer acceleration function is not enabled. No blocking on our firewall.
  • David Gugick
    118
    A few things to consider:
    1. Just because you have a 2 Gbit upstream connection, does not mean you are going to get close to that speed when backing up. You can use this Amazon test page to see what your performance is with and without transfer acceleration: https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html
    2. You are backing up to a region far from the source data which introduces additional latencyand reduces performance. if you can, I'd recommend you consider using either a local region or use S3 Transfer Acceleration if all other recommendations below do not result in the desired performance
    2. I see you have Logging set to High, which is not recommended unless you are instructed to do so by Support for logging purposes. It will generate a lot of local IO as it writes out detailed log information. Please return to the normal level
    3. You are backing up a network share, which introduces additional overhead when compared to backing up locally attached storage.
    4. You did not mention how many files are being backed up and their average size, but you will find performance to be much better when backing up many files if you use the new backup format as opposed to the legacy format. The legacy format backs up each file as single cloud objects. The new backup format combines files into larger archives which reduces IO considerably
  • Guilherme Pessoa
    0
    in this case or I use the tranfer acceleraton or shipping to region of São Paulo. But then I'm going to have to migrate to northern Virginia. Or it would be better to use transfer acceleration


    https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html?result=01249-3894-4219-7448&identityId=unknown
  • David Gugick
    118
    did you test your speed to Virginia from your current location? Make the other changes as well with the log file and consider using the new backup format and maybe you can avoid the additional S3 transfer acceleration charges from AWS altogether
  • David Gugick
    118
    any updates on your testing?
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment