OK, just typing these ideas - so you man want to check /proc/device-tree
So in /boot/boot.txt after $this->bbcode_second_pass_code('', ' if load ${devtype} ${devnum}:${bootpart} ${fdt_addr_r} /boot/dtbs/${fdtfile}; then') add:
$this->bbcode_second_pass_code('', 'fdt addr ${fdt_addr_r}
ftd set /usb@ff500000 no-relinquish-port')
Which works with modern u-boot. This is what miniarm had - which I think is the only change from what we created for you.