What about the CrOS setup is so different from a full traditional Linux install? The kernel/X11/mali driver should all work on Arch Linux installed natively. If Ubuntu can perform so well even if it is "mooching" off of Chrome OS, I see no reason why Arch cannot. They're both still Linux, utilizing the same underlying technologies. Just differing versions (with the one in ALARM repos being much less than optimal).
I feel like the problem in Arch would probably be X11. CrOS either uses an older version, or a patched version which performs better with the mali driver.
I also find it difficult to believe that X11 performance when scrolling and dragging is "fine" for you. Maybe you have lower expectations than I, but.. have you tried crouton? It makes the native ALARM install look unbearably slow.