I dunno about wifi ... need to compare the two kernel configs and see if ours is missing something. It's odd that unloading it then reloading it works so perhaps my previous statement is off the mark.
Recommend you (andisci) post this to the
raspberry pi forum as they might have a suggestion, addition of some module parameter or whatever.
About kodi-rbp3 ... yes, that is expected. The RPi4 does not use windowing with dispmanx for GL window surfaces like the 1-3 do so these packages won't work on Pi4. It will try to use the firmware GL libs (in /opt/vc/lib) and headers (in /opt/vc/include) which won't work on the Pi4 (GL hardware is only supported by MESA libs). It will need to be gbm but currently there is no hardware decoding for this.
Reference:
https://github.com/xbmc/xbmc/pull/16324