modem freezes

Every so often my modem goes into some sort of hardlock and we lose all connectivity in the house.
No phone, no internet, not even our local network.

You can't "ask it" for a new ip.
You can't even ping the damned thing.

Calling my ISP confirms this suspicion as the only reaction i tend to get from them is "huh, weird, it indeed does not respond to my commands".

So, there is one solution that works but it's an annoying one.
1. unplug all ethernet cables
2. did that work? go to 4
3. unplug power for half a minute or so
4. insert ethernet cables one by one untill they're either all plugged in or the modem has gone back into this locked up state. In that case goto 1.

This process can take anywhere from 1 to 30 minutes and is one of the main reasons it's so difficult to debug. I can't seem to pinpoint the issue on one single cable.
The other reason is the unpredictability of the issue. Some days it happens multiple times and then we can go weeks without it occuring even once.

It doesn't seem to have anything to do with how many devices are online at the time nor with how much bandwith we are using.
I can download happily at full capacity for several hours at end without issue and it might crash while there's only some minor browsing going on (like now).

We have tried replacing the modem, which didn't work. So I'm guessing it might be either with how our network is set up or (in the worst case) our wiring.

Any suggestions?

Can we get a little more detail? Model of modem / router. Network setup, ISP?

my ISP is telenet (i live in belgium).
The modem is a CBN CH6541E, do note that we cannot choose what kind of modem we use. It's what the ISP gives us or nothing...

I'll try to keep it simple:

modem is a modem/router combo (has 4 gbit ports + wireless capablities with a wireless network at both 2 and 5 GHz):
1. living room connection 1, to parent's pc
2. living room connection 2, to a switch:
----1. digibox
----2. playstation 3
3. a switch in my brother's room:
----1. his pc
----2. our server
----3. wap
----4. possibly a laptop (not always)
4. a switch right next to the modem:
----1. my room (1 pc)
----2. my youngest brother's room (1 pc)
----3. alarm system

Then there's (at most) 5 smartphones, 2 laptops and a tablet connected to the wifi, depending on who's home ofcourse.