Reasons for page allocation failure?

This forum is for topics specific to the Raspberry Pi and Arch Linux ARM

Reasons for page allocation failure?

Postby smaxer » Tue Sep 10, 2013 4:12 pm

Hi,

I was wondering what could be the reasons for the huge amount of page allocation failures I get on both my raspberry pis. (Both Rev. B one with 512MB Ram)
I've also tried to use another and new SD card but it didn't help.

Here is an example of one logged page allocation failure.... but as said...they happen often.
$this->bbcode_second_pass_quote('', '
')[ 123.156248] mount: page allocation failure: order:5, mode:0xd0
[ 123.156338] [<c0012af4>] (unwind_backtrace+0x0/0xf0) from [<c00ba190>] (warn_alloc_failed+0xd0/0x118)
[ 123.156377] [<c00ba190>] (warn_alloc_failed+0xd0/0x118) from [<c00bc854>] (__alloc_pages_nodemask+0x534/0x71c)
[ 123.156420] [<c00bc854>] (__alloc_pages_nodemask+0x534/0x71c) from [<c00e8eac>] (cache_alloc_refill+0x338/0x690)
[ 123.156450] [<c00e8eac>] (cache_alloc_refill+0x338/0x690) from [<c00e9370>] (__kmalloc+0x16c/0x1a4)
[ 123.156499] [<c00e9370>] (__kmalloc+0x16c/0x1a4) from [<c01b46a8>] (ext4_kvzalloc+0x14/0x3c)
[ 123.156531] [<c01b46a8>] (ext4_kvzalloc+0x14/0x3c) from [<c01b9e3c>] (ext4_fill_super+0x2390/0x2748)
[ 123.156573] [<c01b9e3c>] (ext4_fill_super+0x2390/0x2748) from [<c00f6f88>] (mount_bdev+0x19c/0x1d8)
[ 123.156622] [<c00f6f88>] (mount_bdev+0x19c/0x1d8) from [<c01a92a8>] (ext4_mount+0x14/0x20)
[ 123.156655] [<c01a92a8>] (ext4_mount+0x14/0x20) from [<c00f79e4>] (mount_fs+0x44/0x184)
[ 123.156694] [<c00f79e4>] (mount_fs+0x44/0x184) from [<c0110238>] (vfs_kern_mount+0x4c/0xc0)
[ 123.156724] [<c0110238>] (vfs_kern_mount+0x4c/0xc0) from [<c0110304>] (do_kern_mount+0x34/0xd0)
[ 123.156755] [<c0110304>] (do_kern_mount+0x34/0xd0) from [<c0111f58>] (do_mount+0x2d8/0x740)
[ 123.156783] [<c0111f58>] (do_mount+0x2d8/0x740) from [<c0112444>] (sys_mount+0x84/0xb8)
[ 123.156817] [<c0112444>] (sys_mount+0x84/0xb8) from [<c000db20>] (ret_fast_syscall+0x0/0x30)
[ 123.156829] Mem-info:
[ 123.156838] Normal per-cpu:
[ 123.156850] CPU 0: hi: 186, btch: 31 usd: 0
[ 123.156879] active_anon:5840 inactive_anon:28 isolated_anon:0
active_file:3256 inactive_file:35672 isolated_file:32
unevictable:0 dirty:60 writeback:0 unstable:0
free:67124 slab_reclaimable:2206 slab_unreclaimable:1204
mapped:8468 shmem:76 pagetables:393 bounce:0
[ 123.156937] Normal free:268496kB min:8192kB low:10240kB high:12288kB active_anon:23360kB inactive_anon:112kB active_file:13024kB inactive_file:142688kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:483488kB mlocked:0kB dirty:240kB writeback:0kB mapped:33872kB shmem:304kB slab_reclaimable:8824kB slab_unreclaimable:4816kB kernel_stack:976kB pagetables:1572kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 123.156953] lowmem_reserve[]: 0 0
[ 123.156973] Normal: 9644*4kB 9538*8kB 9415*16kB 73*32kB 8*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 268496kB
[ 123.157026] 39023 total pagecache pages
[ 123.157035] 0 pages in swap cache
[ 123.157045] Swap cache stats: add 0, delete 0, find 0/0
[ 123.157052] Free swap = 0kB
[ 123.157059] Total swap = 0kB
[ 123.223892] 121856 pages of RAM
[ 123.223916] 67349 free pages
[ 123.223925] 3663 reserved pages
[ 123.223932] 3409 slab pages
[ 123.223939] 285596 pages shared
[ 123.223947] 0 pages swap cached
[ 123.223963] SLAB: Unable to allocate memory on node 0 (gfp=0xd0)
[ 123.223979] cache: size-131072, object size: 131072, order: 5
[ 123.223994] node 0: slabs: 1/1, objs: 1/1, free: 0
smaxer
 
Posts: 23
Joined: Sat Jun 15, 2013 9:01 am

Re: Reasons for page allocation failure?

Postby sdjf » Wed Sep 11, 2013 3:49 am

One possible explanation is you might have run out of RAM. I was getting the same Page Allocation Failure messages before things crashed on my Pi, see the following answer/discussion and possible solution:

http://www.raspberrypi.org/phpBB3/viewt ... 05#p369705

In my case, I was doing too many things at once, and finally gave up trying to run Midori browser using VNC over ethernet while dialup was running. Are there any memory or byte-intense applications running when this happens for you?
sdjf
 
Posts: 178
Joined: Wed May 08, 2013 1:55 pm

Re: Reasons for page allocation failure?

Postby moonman » Wed Sep 11, 2013 4:45 am

The easy solution for me was to change the memory split to give the VPU 128 MB of RAM, while by default it has 2/3 ram dedicated. On 256 MB model if you are not using XBMC or desktop you can use the minimum (32?).
Pogoplug V4 | GoFlex Home | Raspberry Pi 4 4GB | CuBox-i4 Pro | ClearFog | BeagleBone Black | Odroid U2 | Odroid C1 | Odroid XU4
-----------------------------------------------------------------------------------------------------------------------
[armv5] Updated U-Boot | [armv5] NAND Rescue System
moonman
Developer
 
Posts: 3388
Joined: Sat Jan 15, 2011 3:36 am

Re: Reasons for page allocation failure?

Postby smaxer » Fri Sep 20, 2013 2:15 pm

Hi Guys,

thanks for reply, but I don't think it has something to do with low memory because I get those page allocation failures very quick right after startup. And it also says:

$this->bbcode_second_pass_quote('', '
')[ 123.223892] 121856 pages of RAM
[ 123.223916] 67349 free pages


So half of the memory is unused, this also matches up with what is gkrellm and top are reporting about memory consumption...

For some reason I suspect it has something to do experimenting with overclocking a few month ago. I already turned it off because it didn't help that much. What I'm going to try now is to start with a clean installation and try to figure out when the page allocation failures occur.

If you have another idea, fell free to tell :)

Best,
smax
smaxer
 
Posts: 23
Joined: Sat Jun 15, 2013 9:01 am

Re: Reasons for page allocation failure?

Postby zeludo » Mon Nov 18, 2013 10:47 am

Hi,

Got the same problem and I found why and how to get rid of these allocation failures.
I also had many page allocation failure while still having plenty of free RAM.
The cause was a heavy memory fragmentation that prevent allocation of a contiguous memory zone.

Your log show you have free RAM (268496kB)
$this->bbcode_second_pass_quote('', '[') 123.156937] Normal free:268496kB min:8192kB low:10240kB high:12288kB active_anon:23360kB inactive_anon:112kB active_file:13024kB inactive_file:142688kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:483488kB mlocked:0kB dirty:240kB writeback:0kB mapped:33872kB shmem:304kB slab_reclaimable:8824kB slab_unreclaimable:4816kB kernel_stack:976kB pagetables:1572kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

and a high memory fragmentation, the largest contiguous block is 128kB
$this->bbcode_second_pass_quote('', '[') 123.156973] Normal: 9644*4kB 9538*8kB 9415*16kB 73*32kB 8*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 268496kB


You can watch fragmentation by running
$this->bbcode_second_pass_code('', '# cat /proc/buddyinfo')

This will give you how much block of contiguous free ram you have for different size : 4kB 8kB 16kB 32KB...4096kB

You should see a lot of small blocks and maybe a few or no free block at all of more than 64kB => This means that every allocation greater than this size will fail !!

The kernel memory allocator try to limit fragmentation but i noticed that the default arch linux kernel for the rpi was using SLAB allocator and not using memory compaction.
You can check how your runing kernel was configured with these commands :
$this->bbcode_second_pass_code('', '
# zcat /proc/config.gz | grep CONFIG_COMPACTION
# zcat /proc/config.gz | grep SLAB
# zcat /proc/config.gz | grep SLUB
')

The easiest way I found to get rid of memory fragmentation was by using the official rpi kernel and rpi-update to update it.
https://github.com/Hexxeh/rpi-update

Don't forget to run rpi-update after each update of the arch linux kernel packages.
Also disable CMA in your config.txt if you used it because this doesn't seem to work anymore with latest official kernel. Use a fixed memory split instead.

Hope this helps.

Ludo.

PS : I don't have a pi for testing the commands I wrote so it may not work as is, sorry. I wrote everything from memory.
zeludo
 
Posts: 3
Joined: Mon Nov 18, 2013 10:15 am
Top

Re: Reasons for page allocation failure?

Postby WarheadsSE » Mon Nov 18, 2013 1:30 pm

Ludo, you could always have informed the devs via an issue @ github, or a pull request with a better option set.

Note to any readers: using a kernel other than the packaged, or self-compiled from the package will likely end with "not our problem"
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Re: Reasons for page allocation failure?

Postby zeludo » Mon Nov 18, 2013 2:07 pm

I plan to inform devs about that, just wanted to investigate a bit more on kernel config to confirm my thoughts about allocator and memory compaction by compiling a kernel with only these modifications (I even think that only activating compaction would do the trick).
Then i'll make a pull request.
zeludo
 
Posts: 3
Joined: Mon Nov 18, 2013 10:15 am

Re: Reasons for page allocation failure?

Postby pepedog » Mon Nov 18, 2013 6:00 pm

Lido,
Thank you
pepedog
Developer
 
Posts: 2431
Joined: Mon Jun 07, 2010 3:30 pm
Location: London UK

Re: Reasons for page allocation failure?

Postby zeludo » Thu Nov 21, 2013 12:12 am

zeludo
 
Posts: 3
Joined: Mon Nov 18, 2013 10:15 am


Return to Raspberry Pi

Who is online

Users browsing this forum: No registered users and 4 guests