Pi Zero kernel compilation issues

Ask questions about Arch Linux ARM. Please search before making a new topic.

Re: Pi Zero kernel compilation issues

Postby quark » Thu Dec 09, 2021 1:14 pm

I tried distcc again with Pi4 and it indeed is much faster. I used the info from here: https://archlinuxarm.org/wiki/Distcc_Cross-Compiling
Used my PC and my laptop at the same time, the Pi4 was constantly 100% loaded and seems it needs at least 200mbps network upload speed.
The Zero had a wifi adapter on USB and it saturated the link.
So for anyone else reading this, it's totally worth it to setup your machines for distcc compiling.
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby quark » Thu Dec 09, 2021 11:25 pm

A mention on the wiki entry:
$this->bbcode_second_pass_quote('', '
')Change the MAKEFLAGS -j flag to reflect the total number of processors available on the master system. The common wisdom is to set this to the number of physical cores + 1. Keep in mind that only compiles are distributed; preprocessing and linking still takes place on the master system. Therefore, this number should reflect the capabilities of the master system and not the total number of distributed cores available.


I've tried it both ways, with -j5 and -j32 (12+4 cores doubled) and there's a big difference. With -j5 it would barely send info out, low network activity, CPU cores not maxed out but some activity.
With -j32 the CPU is constantly maxed out on all 4 cores, and higher network activity, I'd say around twice as high if not more.
As I see it going for the info (regarding this particular detail) in the main Arch wiki will result in faster compile speeds.
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby graysky » Fri Dec 10, 2021 7:15 am

More CPU usage does not necessarily mean faster compilation. You have to test it. Run the compilation 3-5 times with each condition, and compare the results. Average and standard deviation are two key metrics for each. Here is a script I used in the past you can adapt.
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Pi Zero kernel compilation issues

Postby quark » Fri Dec 10, 2021 12:03 pm

Yes I didn't actually run them timed, I just figured more network activity/higher CPU usage means more jobs sent out. But this might be overhead in reality.
I did time the one with -j32, now gotta do it once more for -j5 and compare. same config file.
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby graysky » Fri Dec 10, 2021 12:34 pm

And you will have to start the clock on the make step and end at after the make step. Make flags have no effect on untarring and preparing source. The dead time will dilute any differences. That's why I wrote that script.
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Pi Zero kernel compilation issues

Postby quark » Sat Dec 11, 2021 11:43 pm

I timed it since it started sending out jobs on the network and finished when it stopped sending jobs, when started to package files etc I stopped the timer.
Will try it again the exact same setup and config file, I just didn't yet have the time.
But will also look at your script.
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby quark » Sun Dec 12, 2021 11:26 pm

I did a run today with both -j5 and -j64 on the Pi4. With -j5 it takes about 1 hour and 13 minutes from when it starts sending jobs over the network until it finishes sending them. htop said around 3.8 load average (third column) almost the whole time. CPU was not 100% all the time, it would jump a lot. Didn't wait for package creation for any of the two runs.
With -j64 ( had 6 cores on one volunteer PC and 4 cores on the laptop) it finished it in 1 hour and one minute. CPU was almost constantly 100% on all cores, network peak transfer speed was about twice as high as with -j5, average just a bit higher. htop showed a load average of 12.7 this time.
Still in both cases both laptop and PC were doing very light work.
So if anyone wants the faster option to build the whole thing then clearly exaggerating the total number of available cores, according to the main Arch Wiki, is faster than using -j5, for a RPi4.
I also made a cross compilation of Raspbian on my PC and it completely finished it in 8 minutes and 8 seconds.
To me distcc is still better than chrooting from PC, which is still better than compiling on the Pi itself.
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby quark » Thu Dec 16, 2021 12:40 am

I reinstalled aarch64 and used linux-rpi, and now it compiled the whole linux-rpi kernel in 20 minutes. I didn't do anything different, network transfer average was way higher, solid 200mbps or so. I really don't know what the difference is, but it's pretty fast this time with this config. Seems pretty nice to be honest.
I did change the sdcard, could that be a reason for it compiling slower?
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

Re: Pi Zero kernel compilation issues

Postby armuseru » Sat Dec 18, 2021 6:44 am

just for comparision: cross compling armv7 kernel on my old 4-cores core2-quad type x86_64 system takes up 20 minutes (-j5); compiling same kernel at qemu virtualized arm system at same 4-cores old pc takes up 8 or more hours; compiling same kernel on natural armv7 2-cores exynoss chromebook takes up 2 hour (-j3). In all cases fs was btrfs with zstd compression
armuseru
 
Posts: 34
Joined: Sat Jan 09, 2021 9:36 am

Re: Pi Zero kernel compilation issues

Postby quark » Sat Dec 18, 2021 9:31 pm

What is interesting for me is that there's a huge difference in network activity between compiling linux-rpi and linux-aarch64.
When compiling linux-rpi there's around 20GB of data sent from the Pi to the volunteers, but when compiling linux-aarch64 there's around 40GB of data sent. This explains why linux-rpi compiles so much faster.
@graysky Curious about why, I suppose it's the default config? In each case I installed the default one, exported the config and made a few modifications to it then compiled it. Do you know off the top of your head why there's this huge difference in network activity between the two configs?
quark
 
Posts: 59
Joined: Thu Nov 25, 2021 8:26 am

PreviousNext

Return to User Questions

Who is online

Users browsing this forum: No registered users and 5 guests