I've been using linux for roughly 7 years now, but I've never bothered to compile a kernel and never tweaked one much beyond boot flags or installing kernel headers. That being said, while I'm an experienced linux user, I'm still a noob when it comes to kernels.
Anyway, I own an UDOO (i.MX6 based platform) and I've noted that all images I tried for it (arch, debian, and ubuntu) have very few drivers - basically just enough to make the UDOO's built in hardware work, plus some other simple things like flash drives and webcams. This is somewhat of a problem, as I intend to use a lot of quirky USB devices down the road (kinect, touchscreens, fingerprint readers, etc).
Since there is no installed package for the kernel image, I was wondering if it was possible to take a currently existing armhf linux-image (maybe from debian or ubuntu since I know they sometimes keep record of some of their older kernels) package and copy its drivers to the booting kernel. If this is possible, would I be able to take drivers from the 3.12 kernel, or am I limited to strictly the 3.0 kernel?
I'm open to learning how to compile the kernel, but I'm not really sure where to begin and I can't imagine ARM is the best place to start due to how picky they seem to be.