I'm not aware of any "home" routers that would be rackmountable. You could get a shelf to set the router in, or you would be looking at a a 1U server of some kind to run a routing OS.
I would highly recommend watching a few configuration videos on RouterOS before purchasing as it is much more involved than other routers. Even SOHO routers like the EdgeRouter (EdgeOS) are much less complicated to configure than RouterOS.
It may just "click" for you but it's worth taking a little bit of time to be sure you want to mess with it in my opinion.