Not exactly. When you run N network in 2.4 band, you may get slower speeds if G clients connect or this band is too crowded (there's only space for about 3 busy N networks). That's why it's recommended to use 5ghz band to get full N speeds, only you need equipment that supports this (eg. dual-band). But if there's not many other networks around, 2.4ghz is ok for N too.
Basically it depends on how many other people are using the air at the same time.