Crashplan killed by systemd (out of mem) though plenty free

[kubaraczkowski]

Hi,

I've installed crashplan with jdk8 following viewtopic.php?f=31&t=5120 and it works great.

Well... It starts ok, begins the backup procedure and then gets killed due to 'out of memory'... (I see that in the journalctl, logs below).
The thing is that 'free' reports >330MB available at the moment of crash (it's the 512MB version). Weird no? Also, I've put crashplan on lower java heap (200MB now) but it does not seem to help...

Any idea what may be killing my crashplan?
BTW. I'm backing up ~100GB. The initial backup went ok in fact (I think), it's the updates now that get killed...

Here is what journalctl reports. The logs of crashplan itself are completely clean...

$this->bbcode_second_pass_code('', '
Jan 08 21:16:32 alarmpi.home-domain kernel: Normal per-cpu:
Jan 08 21:16:32 alarmpi.home-domain kernel: CPU 0: hi: 42, btch: 7 usd: 6
Jan 08 21:16:32 alarmpi.home-domain kernel: active_anon:30331 inactive_anon:49 isolated_anon:0
active_file:21 inactive_file:727 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:82846 slab_reclaimable:765 slab_unreclaimable:1084
mapped:217 shmem:83 pagetables:326 bounce:0
free_cma:74731
Jan 08 21:16:32 alarmpi.home-domain kernel: Normal free:331384kB min:32768kB low:40960kB high:49152kB active_anon:121324kB inactive_anon:196kB
Jan 08 21:16:32 alarmpi.home-domain kernel: lowmem_reserve[]: 0 0
Jan 08 21:16:32 alarmpi.home-domain kernel: Normal: 136*4kB (UEC) 103*8kB (UEMC) 58*16kB (UE) 36*32kB (UEMC) 26*64kB (UE) 17*128kB (UEC) 6*256
Jan 08 21:16:32 alarmpi.home-domain kernel: 833 total pagecache pages
Jan 08 21:16:32 alarmpi.home-domain kernel: 0 pages in swap cache
Jan 08 21:16:32 alarmpi.home-domain kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 08 21:16:32 alarmpi.home-domain kernel: Free swap = 0kB
Jan 08 21:16:32 alarmpi.home-domain kernel: Total swap = 0kB
Jan 08 21:16:32 alarmpi.home-domain kernel: 121856 pages of RAM
Jan 08 21:16:32 alarmpi.home-domain kernel: 83078 free pages
Jan 08 21:16:32 alarmpi.home-domain kernel: 3727 reserved pages
Jan 08 21:16:32 alarmpi.home-domain kernel: 1377 slab pages
Jan 08 21:16:32 alarmpi.home-domain kernel: 262825 pages shared
Jan 08 21:16:32 alarmpi.home-domain kernel: 0 pages swap cached
Jan 08 21:16:32 alarmpi.home-domain kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 56] 0 56 58815 112 120 0 0 systemd-journal
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 88] 0 88 2350 81 7 0 -1000 systemd-udevd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 111] 0 111 694 126 5 0 0 crond
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 117] 0 117 742 48 5 0 0 systemd-logind
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 118] 81 118 676 51 5 0 -900 dbus-daemon
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 257] 0 257 504 39 5 0 0 dhcpcd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 259] 0 259 1584 92 7 0 -1000 sshd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 276] 87 276 1251 87 6 0 0 ntpd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 284] 32 284 565 56 5 0 0 rpcbind
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 294] 0 294 969 62 5 0 0 rpc.gssd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 335] 0 335 627 43 5 0 0 rpc.idmapd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 336] 0 336 692 147 5 0 0 rpc.statd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 337] 0 337 692 147 5 0 0 rpc.statd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 363] 0 363 781 112 5 0 0 rpc.mountd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 374] 0 374 440 19 3 0 0 agetty
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 375] 0 375 440 18 4 0 0 agetty
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1060] 1000 1060 948 66 6 0 0 systemd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1062] 1000 1062 1281 178 5 0 0 (sd-pam)
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1081] 0 1081 2566 143 9 0 0 sshd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1085] 1000 1085 2599 144 8 0 0 sshd
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1086] 1000 1086 827 77 5 0 0 bash
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1249] 1000 1249 777 43 5 0 0 check_mem
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 1372] 0 1372 76374 28585 81 0 0 java
Jan 08 21:16:32 alarmpi.home-domain kernel: [ 4943] 1000 4943 777 154 5 0 0 check_mem
Jan 08 21:16:32 alarmpi.home-domain kernel: Out of memory: Kill process 1372 (java) score 212 or sacrifice child
Jan 08 21:16:32 alarmpi.home-domain kernel: Killed process 1372 (java) total-vm:305496kB, anon-rss:113848kB, file-rss:492kB
Jan 08 21:16:33 alarmpi.home-domain CrashPlanEngine[4950]: Stopping CrashPlan Engine ... /opt/crashplan/bin/CrashPlanEngine: line 144: kill: (
Jan 08 21:16:43 alarmpi.home-domain CrashPlanEngine[4950]: OK
Jan 08 21:16:43 alarmpi.home-domain systemd[1]: Unit crashplan.service entered failed state.
')

[dhead666]

For this size of backup archive you'll need to set the JavaVM at least at 512MB maybe even 700MB for the initial backup/sync so you'll need a swap partition.

After the initial backup/sync is over you could lower down the JavaVM to about 400MB and remove the swap partition and Crashplan will run smoothly.

Keep in mind that is a really bad idea to use flash memory for swap so don't allocate a swap partition from your system sd.
I just used a cheap Class10 Sandisk SD card with a usb adapter and I didn't had any problems to finish the initial sync.

If you believe that your backup archive will continue to grow considerably then get a device with 1GB RAM and your Crashplan setup will stay stable.
I moved my Crashplan setup to a Cubieboard 2 (dual core A20, 1GB RAM) which got enough muscles and memory to serve Crashplan with ease with bunch of other services but sadly it needs an active cooling so I'll be moving the setup to a Cubox-i2.

[kubaraczkowski]

Hm.. isn't this a bit strange?
Am I wrong thinking that if java heap is set to 200MB and crashplan exceeds that limits - it would kill itself and place in the crashplan logs an 'outofmemory' exception message ? (I don't see that at all) I think on the boards of crashplan itself that's what I've seen...

Strange as well that I had it running smoothly on a Netgear Stora - also 512MB of memory. It only was super slow during active backup which made the NAS super slow.

I tried putting swap on the NAS (via NFS) that I am backing up, but it did not help. I also put the temporary files of crashplan on the NAS - no help either.

I'll try again with a swap and a larger javaVM memory limit...

[kubaraczkowski]

Hmm, what I did now is I increased the Xmx value (to 1GB) instead of decreasing it, enabled the over-NFS swap and it works! I am not completely sure which of the two steps helped, because 'free' still reports 350MB available and there is 100MB in the swap...

Thanks though! The answer led me in good direction