Scheduled Backup (block level) duplicate files when modified instead of just adding blocks
  • dab
    1
    Hello,

    I have an .ods file in my document. In order to test block level backup, I ad image inside then wait for next update and... see on destination that the entire .ods file is uploaded instead of uploading the added parts...

    Do you have any idea?

    PS: Incremental backup is set to afternoon 15:00 and my tries are at 21:00
    PS2: I did not check "run incremental backup... x% of last incremental Backup size
    dab Options
  • Anton Zorin
    30
    Hi
    ↪dab
    ,
    How big is the size of the newly created object?
    How did you identify it's a whole object, not changed blocks?
    Thanks
    Anton Zorin Options
  • dab
    1
    Hello Anton,

    - The newly modified (not created) object is about 200Mo. (Don't forget I have not checked the box about incremental backup when x% of file is modified).

    - How do I know it's not a change block: I go to the backup destination, browse folders until the "my documents", then the folder containing the file versions... There are 2 folders: the first correspond to the file before modification (+/- 150Mo) and the other folder (+/- 200Mo)

    Thanks
    dab Options
  • Matt
    91
    ↪dab
    Can you provide a screenshot of how those files look on storage side? Do you see 2 versions of a file created when you go to the backup storage tab in GUI and click on it?
    Matt Options
  • dab
    1
    Hello,

    3 screens:
    01 source of backup
    02 backup destination (explorer view)
    03 backup destination (cloudberry bakup > backup storage view)

    Are those the ones you asked for?

    Thanks for helping me to understand and if possible adjust.
    Attachments
    01 (92K)
    02 (140K)
    03 (70K)
    dab Options
  • David Gugick
    118
    ↪dab
    The software performed two full file backups according to image #3. There was no block-level backup completed for that file. Did you explicitly schedule block-level backups in the backup wizard? On the Scheduling tab in the wizard, there are two scheduling options: Incremental and Block-Level. Without getting too detailed, the block-level needs to be configured for those types of backups. The Incremental, less frequently, as it backs up complete files.Can you post a screenshot of that page in the wizard?

    If the Schedule tab looks different, you may be running an older version of the software.
    David Gugick Options
  • dab
    1
    Hello,

    A few more informations

    Logs:
    2019-12-02 21:51:55,406 [PL] [1] INFO  - File C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods. Full backup will be created. Parts modified 1411 of 1411
2019-12-02 21:51:55,426 [CL] [1] INFO  - Item generated: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy85\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 21:51:55,431 [PL] [1] INFO  - Command added: Upload operation. Cloud path: E:\destination-bck\CloudBerryBackup\CBB_DESKTOP-SSQ2NJI\C$\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods$\20191202200305\Nouveau Classeur OpenDocument1.ods. IsSimple: False. Modified date: 02/12/2019 20:03:05. Size: 176 MB (184939671)

...

2019-12-02 21:52:09,644 [PL] [11] INFO  - Command completed: Upload operation. Cloud path: E:\destination-bck\CloudBerryBackup\CBB_DESKTOP-SSQ2NJI\C$\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods$\20191202200305\Nouveau Classeur OpenDocument1.ods. IsSimple: False. Modified date: 02/12/2019 20:03:05. Size: 176 MB (184939671)
2019-12-02 21:52:09,960 [PL] [11] INFO  - Update history file called again: C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 21:52:09,964 [PL] [11] INFO  - 6 files copied (total size: 176 MB): C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 21:52:09,967 [CL] [11] INFO  - Running items count: 0, Awaiting: 0, Paused: 0, NA: 0, Failed: 0, SuccessCount: 0, WarningCount:0, 0 In progress count: 0, Idly Running count: 0
2019-12-02 21:52:09,968 [PL] [11] INFO  - Entering sleeping mode event in CommandQueue occurred
2019-12-02 21:52:09,971 [PL] [11] INFO  - Queue completed event fired
2019-12-02 21:52:09,972 [PL] [11] INFO  - Upload queue is empty. Search thread is finished
2019-12-02 21:52:09,977 [PL] [11] INFO  - Delete temporary parts. Time: 00:00:00.0031182
2019-12-02 21:52:10,776 [PL] [11] INFO  - Setting waiting for complete
2019-12-02 21:52:10,777 [PL] [1] INFO  - Backup engine wait completed
2019-12-02 21:52:11,790 [SERV] [1] INFO  - Upload thread finished
2019-12-02 21:52:11,798 [SERV] [1] INFO  - ImportFinalizing Start
2019-12-02 21:52:11,806 [SERV] [1] INFO  - ImportFinalizing End
2019-12-02 21:52:11,899 [PL] [1] NOTICE - Creating purge queue started. Purge conditions: version count: 7, delete topurge elements
2019-12-02 21:52:11,906 [PL] [1] INFO  - Files for purge with versions found: 9
2019-12-02 21:52:11,944 [PL] [1] INFO  - Folder markers for purge found: 0
2019-12-02 21:52:11,948 [PL] [1] NOTICE - Purge version search time: 00:00:00. Files checked for this plan: 4. Files for purge found: 0


......

2019-12-02 22:11:48,041 [PL] [1] INFO  - Item contains explicit acl records: C:\Users\Jean\Documents
2019-12-02 22:11:48,063 [PL] [1] INFO  - Item contains explicit acl records: C:\Users\Jean\Documents\Reflect
2019-12-02 22:11:48,076 [PL] [1] INFO  - Starting MD5 calculation for Nouveau Classeur OpenDocument1.ods. Size 188 MB (197124366)
2019-12-02 22:11:51,917 [PL] [1] INFO  - Inserting 1504 parts. Time: 00:00:00.4403088
2019-12-02 22:11:51,930 [PL] [1] INFO  - File C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods. Full backup will be created. Parts modified 1504 of 1504
2019-12-02 22:11:52,005 [CL] [1] INFO  - Item generated: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy90\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 22:11:52,079 [PL] [1] INFO  - Command added: Upload operation. Cloud path: E:\destination-bck\CloudBerryBackup\CBB_DESKTOP-SSQ2NJI\C$\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods$\20191202210338\Nouveau Classeur OpenDocument1.ods. IsSimple: False. Modified date: 02/12/2019 21:03:38. Size: 188 MB (197124366)
2019-12-02 22:11:52,090 [CL] [12] INFO  - The queue worker thread 12 started.

...

2019-12-02 22:11:52,499 [PL] [1] INFO  - Files to upload count: 1, Data size: 197124366 (188 MB)
2019-12-02 22:11:52,499 [SERV] [1] INFO  - Main thread. Waiting for upload complete event
2019-12-02 22:12:06,536 [PL] [6] INFO  - Command completed: Upload operation. Cloud path: E:\destination-bck\CloudBerryBackup\CBB_DESKTOP-SSQ2NJI\C$\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods$\20191202210338\Nouveau Classeur OpenDocument1.ods. IsSimple: False. Modified date: 02/12/2019 21:03:38. Size: 188 MB (197124366)
2019-12-02 22:12:06,812 [PL] [6] INFO  - Update history file called again: C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 22:12:06,820 [PL] [6] INFO  - 1 files copied (total size: 188 MB): C:\Users\Jean\Documents\Nouveau Classeur OpenDocument1.ods
2019-12-02 22:12:06,843 [CL] [6] INFO  - Running items count: 0, Awaiting: 0, Paused: 0, NA: 0, Failed: 0, SuccessCount: 0, WarningCount:0, 0 In progress count: 0, Idly Running count: 0
2019-12-02 22:12:06,848 [PL] [6] INFO  - Entering sleeping mode event in CommandQueue occurred

...

    I wish it will help

    Thanks for your help and questions
    Attachments
    04 (155K)
    06 (51K)
    0707 (48K)
    dab Options
  • David Gugick
    118
    ↪dab
    From what I can see, the software detected that all parts of the file were changed and proceeded with a full backup. ODS files are ZIP files composed of multiple XML files. Because of the compression used with that file format, it's possible small changes can result in most of the file changing each time it's saved.
    David Gugick Options
  • dab
    1
    Hello David,

    Thanks for this information!

    I will do testing with .xlsx or .docx and see if it can help fo my testing...

    Thanks again for your help, I would never have thought about file compression of ODS and the consequences of compression on file structure even at block level! Thanks for your help
    dab Options
  • David Gugick
    118
    ↪dab
    Those Microsoft office file formats are also ZIP files and get recompressed each time you save. So you may not see block level backups effective with them.

    Just as an aside, you can rename any Office file with a . ZIP extension and open it up to see what's really in it.

    Block-level backups are not going to be as beneficial with files that are recompressed and rewritten each time they are saved. Instead, they are better able to take advantage of files that are changed only in part by the underlying application, like a Microsoft Outlook PST file.
    David Gugick Options
  • dab
    1
    Hello again David,

    Yes, I tried with xlsx and docx and confirm your informations.

    I will try other files including your option with pst file.

    Nice for the zip extension on office files, I wil give a try...

    Have a nice day/evening/Week end David
    dab Options
  • dab
    1
    ↪David Gugick


    David, after my tests I'm wondering what is the purpose of blocklevel backup... It does not seems interesting for pictures, mp3, videos, office documents, zip files...

    In what major case is it interesting?

    Thanks for sharing your view
    dab Options
  • David Gugick
    118
    ↪dab
    As I mentioned previously, any application that is designed to change part of a file instead of an entire rewrite with compression (Office Docs, JPEGs, PDFs, etc.) would be a better candidate. Outlook PST files come to mind as do databases that you might back up when the database is offline. Also, the smaller the file, the more likely that any changes to it would change a high percentage of the overall file size and result in the file being backed up in full.

    There is no harm in leaving the block-level option enabled and scheduled the required recurring backup to make sure any block-level files backed up are backed up in full on some schedule. But, if you don't see your data in need of block-level, then disable it. Backups going forward (after the initial full backup of all files) will always only back up those files that are new / changed.
    David Gugick Options
  • dab
    1
    Thanks David,

    Wish you good time for new year and marry christmas!
    dab Options
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment