Plan process stopped with system error on CBB Ultimate for Linux to S3
  • john-munoz
    0
    I'm running CBB Ultimate on Linux Centos 6.9
    Version: 2.9.0.114

    This is running in an r5.4xlarge EC2 instance (16 vCPU 128 GiB Memory EBS Instance Storage 10 Gbps Networking Performance 3500 Mbps EBS Bandwidth)

    I keep getting failures trying to run a backup to S3 Standard. The error is a generic, "Plan process stopped with system error. Send logs to support." I've sent the logs to support and opened several tickets, but they haven't provided any insight, resolution or troubleshooting steps yet.

    This error occurs at varying times, file points and configurations. There is not particular file that kicks off this error or a particular point at which this error occurs. Sometimes 243 files get copied, sometimes 1866 files get copied. I've been able to get up to 106GB transferred one time before the job failed.

    We are backing up to Amazon S3 Standard bucket. (versioning, server access logging, static hosting, object level logging and default encryption is all disabled)

    We've tried with and without block level backup
    Using encryption, tried different algorithms, server side and client side.
    Standard Storage Class
    Not using S3 Transfer Acceleration
    No pre or post actions

    Set Backup symlinks only
    tried enabling and disabling “ignore SSL certificates”
    Tried limiting cloud storage max speed to 5000 KByte/s
    Max thread count is 5
    Chunk size is 10 Mbyte
    Memory used is 5120
    john-munoz Options
  • Matt
    91
    ↪john-munoz
    I checked the status of the ticket and it seems to be a pretty unusual issue. The problem is being investigated with the help of our R&D and level 2 department. You should expect a reply soon.
    Matt Options
  • john-munoz
    0
    Thanks Matt, it's been three weeks since that ticket was opened and I was wondering if someone in the community may have some troubleshooting ideas of thoughts on other places to look. ciuczdqgrgupu1me.png

    The log_worker files do not seam to indicate any red herrings. The last lines in /opt/local/CloudBerry Backup/logs (even in debug level) just show the last files and that they were successfully backed up. Are there any other log files that might contain information about the program itself or why it may be crashing?

    The localDeamon log shows this...

    2019-06-21 00:00:00,959436 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Fri Jun 21 04:00:00 2019 GMT next run: Sat Jun 22 04:00:00 2019 GMT FULL
2019-06-21 00:00:00,959477 [INFO ]: [ CBB ] [ 7 ] Plan already running
2019-06-21 00:00:00,959503 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-21 04:00:00.959 UTC Qt::TimeSpec(UTC))
2019-06-21 00:00:00,963331 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Fri Jun 21 04:00:00 2019 GMT next run: Sat Jun 22 04:00:00 2019 GMT 
2019-06-21 00:00:00,963352 [INFO ]: [ CBB ] [ 7 ] Plan already running
2019-06-21 00:00:00,963368 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-21 04:00:00.963 UTC Qt::TimeSpec(UTC))
2019-06-21 04:39:39,748559 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
2019-06-21 04:39:39,748593 [INFO ]: [ CBB ] [ 7 ] 
2019-06-21 04:39:39,748605 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
2019-06-21 07:32:55,720801 [INFO ]: [ CBB ] [ 7 ] Request force run plan "{d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}". Request from id: {e492de20-fc76-4cb2-9776-26874e61d3be}
2019-06-21 07:32:55,721019 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
2019-06-21 07:32:55,721991 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test 
2019-06-21 07:51:17,926543 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
2019-06-21 07:51:17,926573 [INFO ]: [ CBB ] [ 7 ] 
2019-06-21 07:51:17,926585 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
2019-06-21 07:59:27,300164 [INFO ]: [ CBB ] [ 7 ] Request force run plan "{d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}". Request from id: {f59b9010-cbf0-477e-8ffe-0a1375b668cf}
2019-06-21 07:59:27,300376 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
2019-06-21 07:59:27,301317 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test 
2019-06-21 15:58:36,688512 [INFO ]: [ CBB ] [ 1 ] Local managment service disconnected
2019-06-21 15:58:36,688852 [INFO ]: [ CBB ] [ 1 ] Local managment service connected
2019-06-21 20:01:33,335633 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
2019-06-21 20:01:33,335679 [INFO ]: [ CBB ] [ 7 ] 
2019-06-21 20:01:33,335691 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
2019-06-22 00:00:00,459825 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sat Jun 22 04:00:00 2019 GMT next run: Sun Jun 23 04:00:00 2019 GMT FULL
2019-06-22 00:00:00,459861 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
2019-06-22 00:00:00,460872 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
2019-06-22 00:00:00,461083 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-22 04:00:00.461 UTC Qt::TimeSpec(UTC))
2019-06-22 00:00:00,464903 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sat Jun 22 04:00:00 2019 GMT next run: Sun Jun 23 04:00:00 2019 GMT 
2019-06-22 00:00:00,464920 [INFO ]: [ CBB ] [ 7 ] Plan already running
2019-06-22 00:00:00,464934 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-22 04:00:00.464 UTC Qt::TimeSpec(UTC))
2019-06-22 00:30:10,415567 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
2019-06-22 00:30:10,415599 [INFO ]: [ CBB ] [ 7 ] 
2019-06-22 00:30:10,415610 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
2019-06-23 00:00:00,459227 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sun Jun 23 04:00:00 2019 GMT next run: Mon Jun 24 04:00:00 2019 GMT FULL
2019-06-23 00:00:00,459270 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
2019-06-23 00:00:00,460445 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
2019-06-23 00:00:00,460667 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-23 04:00:00.460 UTC Qt::TimeSpec(UTC))
2019-06-23 00:00:00,464735 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sun Jun 23 04:00:00 2019 GMT next run: Mon Jun 24 04:00:00 2019 GMT 
2019-06-23 00:00:00,464751 [INFO ]: [ CBB ] [ 7 ] Plan already running
2019-06-23 00:00:00,464766 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-23 04:00:00.464 UTC Qt::TimeSpec(UTC))
2019-06-23 04:03:43,367524 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
2019-06-23 04:03:43,367559 [INFO ]: [ CBB ] [ 7 ] 
2019-06-23 04:03:43,367571 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
2019-06-24 00:00:00,459195 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Mon Jun 24 04:00:00 2019 GMT next run: Tue Jun 25 04:00:00 2019 GMT FULL
2019-06-24 00:00:00,459232 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
2019-06-24 00:00:00,460377 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
2019-06-24 00:00:00,460605 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-24 04:00:00.460 UTC Qt::TimeSpec(UTC))
2019-06-24 00:00:00,464405 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Mon Jun 24 04:00:00 2019 GMT next run: Tue Jun 25 04:00:00 2019 GMT 
2019-06-24 00:00:00,464421 [INFO ]: [ CBB ] [ 7 ] Plan already running
2019-06-24 00:00:00,464436 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-24 04:00:00.464 UTC Qt::TimeSpec(UTC))
    john-munoz Options
  • john-munoz
    0
    I also noticed that these messages were showing up in /var/logs/messages

    Jun 25 10:48:11 ip-10-231-100-183 abrt[30932]: Saved core dump of pid 12462 (/opt/local/CloudBerry Backup/raw_bin/cbbWorker) to /var/spool/abrt/ccpp-2019-06-25-10:48:03-12462 (1048576000 bytes)
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Directory 'ccpp-2019-06-25-10:48:03-12462' creation detected
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Package 'cloudberry-backup' isn't signed with proper key
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-06-25-10:48:03-12462' exited with 1
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-06-25-10:48:03-12462'
    Jun 25 10:54:29 ip-10-231-100-183 abrt[29993]: Saved core dump of pid 6997 (/opt/local/CloudBerry Backup/raw_bin/cbbWebAccess) to /var/spool/abrt/ccpp-2019-06-25-10:54:27-6997 (382730240 bytes)
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Directory 'ccpp-2019-06-25-10:54:27-6997' creation detected
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Package 'cloudberry-backup' isn't signed with proper key
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-06-25-10:54:27-6997' exited with 1
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-06-25-10:54:27-6997'
    john-munoz Options
  • Fred Boniface
    0
    I seem to be having the same issue with backups failing due to system error. I am currently working on trawling through logs to see what is up.
    Fred Boniface Options
  • Fred Boniface
    0
    These are the relevant logs I can see, not too sure whether it is related to your problem or not. The only changed I made recently was to add some additional files to the plan but they don't appear to be the problem here.

    2019-06-26 09:41:36,008940 [WARN ]: [ CBC ] [ 8 ] Cloud operation error (NoContent): No Content
2019-06-26 09:41:36,009211 [WARN ]: [ CBC ] [ 11 ] Finished with error, path: /CBB_domoticz/home/domoticz/domoticz/domoticz.db$/20190621020000/full.cbbitmap message: Operation error. Don't retry it
2019-06-26 09:41:36,009405 [INFO ]: [ CBB ] [ 4 ] Finished download file/part, path: /opt/local/CloudBerry Backup/etc/config/dfmap/{2c5f63cb-b78c-434e-b078-8b9a8b7576c9} position: 0 size: 0
2019-06-26 09:41:36,009499 [ERROR]: [ CBB ] [ 4 ] Error on download file/part path: /opt/local/CloudBerry Backup/etc/config/dfmap/{2c5f63cb-b78c-434e-b078-8b9a8b7576c9} code: 13 message: No Content
2019-06-26 09:41:36,009577 [WARN ]: [ CBB ] [ 4 ] Error on download dirrerential revision map, cloud path: /CBB_domoticz/home/domoticz/domoticz/domoticz.db$/20190621020000/full.cbbitmap error code: 13 mess$2019-06-26 09:41:36,010159 [WARN ]: [ CBB ] [ 4 ] Error on open last revision differential map, id: {2c5f63cb-b78c-434e-b078-8b9a8b7576c9} message: No such file or directory
2019-06-26 09:41:36,010260 [ERROR]: [ CBB ] [ 4 ] Backup error: /home/domoticz/domoticz/domoticz.db
message: Invalid last revision diffirential stream
code: 12
2019-06-26 09:41:36,010305 [ERROR]: [ CBB ] [ 4 ] Backup finished: /home/domoticz/domoticz/domoticz.db with errror: Invalid last revision diffirential stream
2019-06-26 09:41:36,183151 [WARN ]: QObject::startTimer: Timers cannot be started from another thread
    Fred Boniface Options
  • Boris
    0
    ↪john-munoz

    Hi,

    The crash code 11 is a known issue, our developers are currently investigating it. There's no workarounds for it, as far as I know.

    ↪Fred Boniface

    Are you using Openstack, by any chance? We'll need the diagnostic information to troubleshoot this, please send it to us by using the Feedback button in the GUI and mention this thread in the description of the issue.
    Boris Options
  • Jim Q
    0
    Adding that I'm having a similar issue. Just setup CBB Ultimate on Ubuntu 18.04 and have not been able to successfully complete a large backup. I've opened a ticket with Cloudberry and submitted the logs via the Feedback button. I've gotten two small backups to finish but larger ones will die at random intervals.
    Jim Q Options
  • Matt
    91
    ↪Jim Q
    The issue usually happens sporadically without any pattern, that's just a daemon crash. It's one of our top priority issues to fix it for Linux/mac software.
    Matt Options
  • Damian
    0
    ↪Matt
    Is there an update to this issue? I just got started and paid for Cloudberry backup and having the same issue as John. I'm running the latest 2.9.2.14 version and backing up to B2. It seems to be related to thread count, as I was able to get a normal backup exit by setting the number of threads to 1. It's just slower in this case, and overnight backups are going to take a month at this point to complete.
    Damian Options
  • Matt
    91
    ↪Damian
    It's an ongoing problem that is being investigated by our developers. According to the latest info improvements will be made in version 2.9. It seems they were able to localize the problem with RAM and are testing things internally.
    Matt Options
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment