Home Network overhaul 10gig SFP+/multigig/wifi/router

Z

zandor

Supreme [H]ardness
Joined
Dec 14, 2002
Messages
4,180
I'm looking to finally do the network overhaul I've been thinking about for a while, so I need to figure out what equipment to buy. I've been looking around but don't have much of an idea what's good or what all is out there. I need 10gig for a programming project I want to do. I'm also eyeing an internet upgrade to 1.2Gb and that will require a new modem or router and >1gig lan. Thanks to Comcast I only have 2 options, a Motorola WiFi router or a Netgear modem. While I'm at it I figure I might as well look at multi-gig, but might not bother with that until a bit more stuff that uses it comes out since I'm going 10gig on the desktops/servers unless I need to bridge from the router to the 10gig switch.

Switches
Looking for a 10-gig SFP+ switch with VLANs, probably 8 ports. My have these things gotten cheap compared to what they used to cost, but are they any good? I'd like to avoid the massively loud used datacenter switches on eBay. Some examples: https://www.amazon.com/TP-Link-TL-SX3008F-Enterprise-Integrated-Protection/dp/B0973T7BGL https://store.ui.com/collections/unifi-network-switching/products/unifi-switch-aggregation https://www.qnap.com/en-us/product/qsw-m804-4c

I might also want a multi-gig switch depending on what I end up with for a router & WiFi. Again nothing crazy, probably like an 8-port with a 10gig uplink, preferably SFP+. 5gb and 10gb support over twisted pair would be nice, but isn't a must. 2.5gig is enough. If I need a 10gig run over copper I can get a transceiver. VLANs would also be a nice bonus. Haven't really looked into these yet since I've been looking for 10gbe switches and routers.

Router & WiFi
So go for the combo Motorola, get the modem and a WiFi router, or go separate with a modem, wired router and access points? Both my options for connecting to Comcast have a 2.5Gbe port.

Seems like there aren't many wired routers with 2.5gig or faster WAN and LAN ports unless you start looking at "real" routers. This one seems like a possibility for a WiFi router, plus it does incoming VPN: https://www.qnap.com/en-us/product/qhora-301w Another perk is it eliminates the need to go out and get a multi-gig switch right away since it has 1/2.5/5/10Gbe LAN & WAN ports on it plus 1Gbe LAN ports. I'd just need an RJ45 transceiver for the 10gig switch and could use my existing 1Gbe switches for the low speed stuff.

I'm kind of thinking I might want an additional WiFi access point since my 5GHz signal is weak at the front of the house away from the router. OTOH if I could get decent speed off of 2.4GHz that might be all I need. It seems to reach farther, but my 2.5GGHz radio is in legacy mode. I should see if I still need that. My WiFi usage is pretty light. I only use it for phones, things that don't have an RJ45 on them, and a laptop I barely use.

NICs
Definitely getting at least one Solarflare, hopefully an SFN8522-PLUS if I can find one used. I need a Solarflare with an Onload license (-PLUS) for my programming project. Yeah, I know, I'll probably need to hack some cooling to use it in a desktop.
Other than that, I guess I'm looking for some desktop friendly 10gig SFP+ NICs for the rest of the machines. Not sure what, just something cheaper than Solarflare. Maybe Intel? They'll need to work with Linux and Windows 10 & 11. Most of my boxes are dual boot.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Switches - if you can deal with / mitigate noise (they're not "datacenter" loud by any means, but they're not passive/silent) and don't mind the power draw, Brocade ICX-6610's. 2x40GbE, 16x10gbE, 24x or 48x 1GbE ports (models with PoE available, too). These run ~$225 on eBay (cheaper if you skip PoE and less ports). The consumer ones you linked are kind of junk (and more expensive) by comparison.
Edit: Just realized you might've been targeting those switches since they have dummy UIs available. The ICXes do as well, but it's clunky and most things are much better done through cli. If that's a need for you, then you'll most likely want to stick to the consumer level gear.

Router - Mikrotik. I'm a big fan of the RB4011

WiFi - APs with hardline back to somewhere sane. I've had good success with the TPLink EAPs.

NICs - Intel or Mellanox ConnectX. Again, not sure what is driving your "need" here.

What do you mean by
"I need 10gig for a programming project I want to do"? What project and what is making you "need" 10 gig for it?

I'm a little unsure you know what to do with this stuff right now -- it should be pretty easy to read spec sheets and figure out what you need and what is available. Tons and tons and tons of information readily available, once you can specify criteria. However, if you're learning that's great. Don't skimp and do it right the first time since you've apparently been wanting to do this awhile. Those Brocades will set you up for 10Gb for a long time - til you run into needing more devices than that, and have a nice 40GbE backhaul you can use (I use that for spine through floors in my home, an ICX hanging off each end).
 
Last edited:
Once getting up into this level of performance, I would definitely split the modem/router/wireless and get each component individually. I'm partial to using used/open-box enterprise routers for their tremendous value and then the rest can be consumer stuff. You can even repurpose your existing router(s) as additional access points as this is very easy.
 
I'm a software engineer working for a vendor in the financial trading industry. I write network applications, but my experience with enterprise networking equipment is more or less plugging computers into switches and telling the network engineers how I want things set up. They "own" all the switches, routers, etc. and won't let engineering touch them. Thus I have a decent idea of the concepts and I'm not afraid of a CLI, but have no hands on experience. So I know just enough to be dangerous. Probably why IT won't let engineering touch any of the switches.

The project I'm looking to do is build a high performance exchange simulator/test harness. I don't actually need 10Gb speed until I basically finish it and start using it, but I can't get anywhere without a Solarflare NIC since I'm planning to use their proprietary TCPDirect API for kernel bypass. Solarflare NICs start at 10Gb and go up from there. As for the rest of it, well, I just want something faster than the 1Gb stuff I've been using since 2004 or so for general use. If I didn't need a card that only comes in 10+Gb 2.5 or 5Gbe would be fine.

Sounds like I'm going with Mellanox for the rest of the NICs. I should have included them in my original post. We also use those at work too (though Solarflare is more popular in the industry in general), and maybe I'll play around with OFED programming and RoCE after I'm done with my project. I have a couple of old ConnectX VPI cards laying around. Those are ancient though. Dual mode Infiniband/10Gbe with CX4 connectors.

I'll have a look at the Mikrotik routers. That's the sort of suggestion I'm looking for. I haven't paid much attention to network gear in years other than what we use at work.

I'm thinking I might actually want two switches. One as a home network switch and one to play with and use for application testing. Realistically I'm just going to set up whatever I use for my network then it'll sit there until I need to change it. Then I'll end up in a situation like I did back when I used a Linux box as a router years ago. Set it up, leave it for 2 years, then when I need to change something I don't remember how. Getting two also frees me up to get a loud AF one since I'd just shut it down when I'm done with it. That Brocade could work as a home switch. It's 39.6-48.7dB. That should be fine since my network stuff is in the basement.

I'd want a low latency switch as a toy/test switch. Something like https://www.arista.com/assets/data/pdf/Datasheets/7150S_Datasheet.pdf or https://www.mellanox.com/related-docs/prod_eth_switches/PB_SX1024.pdf. I'd get more realistic test results with one and I figure I'm better off learning on something that's popular in my industry. Just in case you're not aware financial trading is not normal IT. For this one I can just ask around at work.
 
A lot of times you can find these 'big iron' switches in the CDW outlet at a pretty good discount from new, sometimes even less than used. I'll see these things but never knew anyone that could use them before.
 
I'm liking the MikroTek RB5009UG+S+IN for the router. $219 list price and it's one of two home sized routers I've found that have an SFP+ and a multi-gig port to plug into a Comcast approved modem. The other one is a $450 Asus RT-AX89X WiFi router which has an SFP+ and a 1/2.5/5/10 port, and of course consumer grade router features.

They have a couple other interesting goodies too. Especially this S+RJ10 1/2.5/5/10 SFP+ transceiver. A little Googling says they have compatibility issues with non-MikroTek stuff and version 1 was crap and version 2 is better. I wonder if they'd work in 2.5 or 5GB/s mode in a Brocade switch. I can't see any reason to use one for a 1 or 10 Gb link. Then there's the teaser switch. CRS312-4C+8XG-RM has 8 1/2.5/5/10 and 4 combo multi-speed/SFP+ ports so not enough SFP+.
 
SamirD said:
Decisions Decisions. :D
Click to expand...
Indeed. I ordered the MikroTek RB5009UG+S+IN, so that's out of the way. Hardly any in stock but a little distributor called roc-noc.com a bit over an hour from here in Rockford, IL had them. If I have any trouble they're within driving distance.

I think I have my wireless APs picked out. 2 TP-Link EAP610 for $100 ea. They're a cheap version of the EAP620 and don't support cloud management. :D I'll just use their Windows program to set them up. Nothing too fancy, just "AX1800". My laptop has an Intel wireless AC chip that can do up to~1700Mbps, but that requires 2 160MHz wide channels in the 5GHz range. Thing is there are only two 160MHz wide channels available, so that doesn't work with 2 access points. I don't expect to upgrade my personal laptop for a while. It's 2 years old and I hardly use it. I just keep a laptop around so I have something portable. Browses web without pissing me off? Good enough. I also have a work laptop, but I don't think I've ever used it on my home WiFi. When it's at home it's either on my desk and plugged in or powered off in a bag. Also the only WiFi gear I could find that would do the 160MHz wide channels was all consumer stuff, so no multiple SSIDs or mapping SSIDs to VLANs. I'm also thinking most of the current wireless stuff is going to be obsolete soon due to the addition of the 6GHz range. Not much WiFi 6E stuff available and what is there is expensive, but it's a new chunk of spectrum and it'll get occupied quick.

For the switch I'm thinking MikroTek CRS326-24S+2Q+RM, partly because of those 1/2.5/5/10 RJ45 copper S+RJ10 transceivers, a little bit because of the 2 40GbE QSFP uplink ports, and because of port count and price. 24 SFP+, 2 QSFP... $500 (new). Google search says the S+RJ10 v2 works, but compatibility is really spotty in non-MikroTek switches. I can stuff 10GbE cards in my desktops, but all the consumer stuff seems to be slowly moving towards 2.5 GbE with a random occasional 5GbE offering. I'd like to be able to hook that stuff up to my core switch. Short term I'm only going to want a couple 2.5 or 5GbE connections, so a multi-gig switch isn't economical vs. those transceivers. MikroTik also has an 8-port and a 16-port SFP+ switch, but the 8-port isn't enough ports and the 16 is $400. I'll cough up the extra $100 for those 2 QSFPs even if they're just for future proofing.

Other than that I'll need to pick up a small, managed PoE switch for the APs with the ability to power cycle PoE devices remotely (~$80) and maybe another 1GbE desktop switch.

I thought I had the NICs figured out but those QSFP plugs and ConnectX-4 pricing have me thinking... was going to just be 2x Mellanox ConnectX-4 and a Solarflare Flareon Ultra snf8522 all 2xSFP+, but CX4s that'll do 25GbE are +$20 and doing something with those 40GbE ports is tempting. Used 40GbE NICs on eBay aren't priced much differently from 10GbE stuff. OTOH I'm not sure what I'd actually use 40GbE for other than "being more [H]" and it would make a mess of the wall plate in my office.
 
All I can say is go big or go home. :D

It would be awesome to have 40Gb even if it is just some dac's between the switch and some equipment. I'm sure what you're working on will eventually need to scale to 40Gb so there you go. :D
 
Funny thing is the most useful way for me to use 40GbE without one of those loud as fuck low latency switches is a direct connection between 2 machines without a switch... after I get my project working in a state where it's ready for performance testing. Other than that I guess I could get a 1-2TB NVMe SSD for my little Linux file server (i3-10100 and some hard drives) to use as a scratch drive or try out Steam game streaming over 40GbE. 1GbE could do 3fps@4k uncompressed. 10GbE could do 30fps. 40GbE should be capable of 120fps. Probably need jumbo frames enabled. It would be kind of fun to file a bug report against Steam for using compression. I should think about that scratch drive. Even over just 10GbE it would be rather nice.
 
You're not going to saturate 40GbE with a single workstation. Even when I'm copying from a PCI nVME to my storage box, I barely get over 10GbE speeds. Perhaps in your special software and NIC you could, I'm unsure about that, but, not with normal workloads. I decided to just use mine as a Spine between floors in my house, and 40GbE to my server. The desktops upstairs are them 10GbE to the switch. We're only a family of two, no kids (F that!), so, it works well for us. I can't imagine needing or wanting more for normal use for quite some time.
 
I could easily saturate 40GbE using Mellanox cards on a file copy it if I could get NFS-RDMA working and had a pair of PCI-e 3.0 NVMe drives in Raid-0 or a PCI-e 4.0 drive. NFS-RDMA got broken around kernel 3.10 or so, but that was in like 2014 and I haven't checked to see if it's fixed. But that's also kind of useless. I just don't move that kind of data around on a regular basis. The project software would get better results with 40, 100, 200, etc. than it would with 10, but it's not because of throughput. Financial trading is all about latency. How many GB/s you can do doesn't matter. It's how fast you can send a 100 byte packet. 100 is 10x as fast as 10. 1 gig sends a 100 byte packet in about 1000 nanoseconds. 10gig takes ~100 nanos. 100gig only needs ~10ns. In trading you're a loser if you have 1gig. 10 is ok. Serious trading firms will upgrade to 25, 40 or 100 to save a few ns... if the exchange supports it. Many don't, and are still on 10gig for order entry. So really 10GbE is what's right for the job... but I'm still willing to pay a mere extra $100 for a couple 40GbE + 8 more 10GbE ports so I have them when someday comes.

edit: Oops. Screwed up the math. (fixed)
 
Last edited:
Eulogy said:
You're not going to saturate 40GbE with a single workstation. Even when I'm copying from a PCI nVME to my storage box, I barely get over 10GbE speeds. Perhaps in your special software and NIC you could, I'm unsure about that, but, not with normal workloads. I decided to just use mine as a Spine between floors in my house, and 40GbE to my server. The desktops upstairs are them 10GbE to the switch. We're only a family of two, no kids (F that!), so, it works well for us. I can't imagine needing or wanting more for normal use for quite some time.
Click to expand...
Forgetting the 40/10/etc. GbE stuff, ever played with a MikroTik switch? I'll be running it with SwitchOS rather than Router OS and compared to an unmanaged switch probably just using VLANs. If anything needs processing between VLANs I think I'll make the router do it. The RB5009 router has a quad core processor running at ~1.4GHz and will be connected to the switch over 10GbE fiber. The switch has a single core running at 650Mhz. Fiber to protect the rest of my stuff from lightning strikes, etc. hitting Comcast's cable.
 
zandor said:
I could easily saturate 40GbE using Mellanox cards on a file copy it if I could get NFS-RDMA working and had a pair of PCI-e 3.0 NVMe drives in Raid-0 or a PCI-e 4.0 drive. NFS-RDMA got broken around kernel 3.10 or so, but that was in like 2014 and I haven't checked to see if it's fixed. But that's also kind of useless. I just don't move that kind of data around on a regular basis. The project software would get better results with 40, 100, 200, etc. than it would with 10, but it's not because of throughput. Financial trading is all about latency. How many GB/s you can do doesn't matter. It's how fast you can send a 100 byte packet. 100 is 10x as fast as 10. 1 gig sends a 100 byte packet in 100 nanoseconds. 10gig takes 10 nanos. 100gig only needs 1ns. In trading you're a loser if you have 1gig. 10 is ok. Serious trading firms will upgrade to 25, 40 or 100 to save a few ns... if the exchange supports it. Many don't, and are still on 10gig for order entry. So really 10GbE is what's right for the job... but I'm still willing to pay a mere extra $100 for a couple 40GbE + 8 more 10GbE ports so I have them when someday comes.
Click to expand...
Cool! I don't know much about trading, I've never worked in financial at any level. I've read a little about it and know that latency is king, and that getting whatever gear you have as close to the exchange is a priority as well. Gotta beat those other HFT platforms and it seems any tiny fraction of a percent is worth a ton of money!
 
zandor said:
Forgetting the 40/10/etc. GbE stuff, ever played with a MikroTik switch? I'll be running it with SwitchOS rather than Router OS and compared to an unmanaged switch probably just using VLANs. If anything needs processing between VLANs I think I'll make the router do it. The RB5009 router has a quad core processor running at ~1.4GHz and will be connected to the switch over 10GbE fiber. The switch has a single core running at 650Mhz. Fiber to protect the rest of my stuff from lightning strikes, etc. hitting Comcast's cable.
Click to expand...
I did have a Mikrotik switch, but, ended up settling on the ICX platform and gave the Mikrotik to a friend. They're good, for sure, just at the time didn't have enough 10GbE SFP slots for my needs, and I'm not a big fan of switchOS (or routerOS for that matter, but I can get my way around routerOS now that I've had to for a few months).
 
Eulogy said:
I did have a Mikrotik switch, but, ended up settling on the ICX platform and gave the Mikrotik to a friend. They're good, for sure, just at the time didn't have enough 10GbE SFP slots for my needs, and I'm not a big fan of switchOS (or routerOS for that matter, but I can get my way around routerOS now that I've had to for a few months).
Click to expand...
Good but not enough SFP+ slots at the time. Works for me. 24 is more than enough. I can deal with clunky. I'm a software engineer. We make clunky.
 
zandor said:
Good but not enough SFP+ slots at the time. Works for me. 24 is more than enough. I can deal with clunky. I'm a software engineer. We make clunky.
Click to expand...
haha yeah, same here. I could've learned and made do with it, but, learning routerOS was plenty at the time, and I have so much other home and work projects that I didn't want to spend the effort :). The really nice thing about the ICX was the OpenFlow support, so doing SDN was an absolute breeze.
 
Eulogy said:
haha yeah, same here. I could've learned and made do with it, but, learning routerOS was plenty at the time, and I have so much other home and work projects that I didn't want to spend the effort :). The really nice thing about the ICX was the OpenFlow support, so doing SDN was an absolute breeze.
Click to expand...
Hopefully this won't be too messy. I'm pretty much planning a layer 2 setup just separating things with VLANs. 1. Computers 2. MTU 9000 storage access for desktops/servers. Probably 10Gb only. 3. IoT stuff 4. Guest network - internet access only. The WiFi access points I ordered support multiple SSIDs and can map them to VLANs. I'll probably put my phone on #3 with the IoT stuff. A lot of that stuff has phone apps and might need to connect over WiFi. All but the storage network would have internet access. All the machines on the storage network would also have access to #1, so the only device that would only access #1 would be my laptop. I'll probably make a few more VLANs for testing software so I can fake multiple NICs.
 
I ordered the rest of the parts last night and today. MikroTik 24SFP+ 2QSFP switch, bunch of transceivers and cables and a couple Mellanox ConnectX4s from fs.com, WiFi access points and a 2TB SSD from Amazon (fast scratch drive for my little file server), and of course a Solarflare SFN8522-PLUS used off of e-bay. $300 for the Solarflare, not bad. Could have gotten a new one for $500, but I'm thinking I wouldn't get full support even if the card was new so... used pull with a guarantee from the seller. I'm a big fan of used pulls from recyclers. Never had a problem with one. The Mellanox cards are new because buying used wouldn't save much.
 
zandor said:
Hopefully this won't be too messy. I'm pretty much planning a layer 2 setup just separating things with VLANs. 1. Computers 2. MTU 9000 storage access for desktops/servers. Probably 10Gb only. 3. IoT stuff 4. Guest network - internet access only. The WiFi access points I ordered support multiple SSIDs and can map them to VLANs. I'll probably put my phone on #3 with the IoT stuff. A lot of that stuff has phone apps and might need to connect over WiFi. All but the storage network would have internet access. All the machines on the storage network would also have access to #1, so the only device that would only access #1 would be my laptop. I'll probably make a few more VLANs for testing software so I can fake multiple NICs.
Click to expand...
I'd suggest blackholing the IoT VLAN as well. Make it one that things can't talk to WAN on at all. I have my amcrest cameras, TV, and a couple other "smart" devices on it, and just seeing how much they try to reach external services (even with all options disabled) has made me feel good about that decision. Otherwise looks like a fine plan.
 
Actually that's another VLAN... the IoT VLAN I'm thinking about is for stuff like my Roku and Nest. Maybe those aren't really IoT devices. When someone says "IoT" the first thing that pops into my head is an internet enabled fridge. "Time to change your water filter!" Not sure what the point is because they send me emails anyway and all knowing I changed it would do is let them know I already bought one... but that wouldn't stop them from trying to sell me another one anyway. At any rate the Roku and Nest are all I've got for internet gizmos. They have no need to talk to anything other than the Internet. Roku without internet access is useless. I'm thinking about getting rid of the Nest. Just have to get around to finding a workable alternative. The only thing I want it for is the ability to learn how my HVAC system responds relative to outdoor temps. I like to let it get a little cold in the house overnight (I let it drop to 63F), but want it to warm up by the time I get up. The Nest will figure out when it has to turn the furnace on to hit 70F by wake-up time and will vary the start of the heating call based on outdoor temps. I'd like to swap it out for something that just has an outdoor temp sensor and uses that to decide when to turn the heat on.

But yeah, I can certainly see having another VLAN for stuff that demands a network connection that you don't want talking to anything, so you just give it a network and set the router policy to "deny" for anything coming from that subnet. I can't think of anything I'd do that to at the moment, but I'll pencil in making that VLAN anyway since I'm sure I'll want it eventually.
 
I myself am waiting for a switch with PoE, 2.5gbps, and 10gbps that isn't $500+. I've been running PoE Tplink managed switches in my house for quite some time and they are awesome. They feed all the vlans and power to my unify APs. I use pfsense for my router.
 
Got the switch and router. Now I need some NICs, cables, transceivers and access points before I can start using them and none of them have shipped yet aside from the used Solarflare sfn8522-plus NIC from a recycler on eBay. So far SwitchOS seems pretty straightforward but RouterOS is going to take some getting used to. Figuring out what does what isn't too bad but finding settings is a pain in the ass.

OFaceSIG said:
I myself am waiting for a switch with PoE, 2.5gbps, and 10gbps that isn't $500+. I've been running PoE Tplink managed switches in my house for quite some time and they are awesome. They feed all the vlans and power to my unify APs. I use pfsense for my router.
Click to expand...
I looked for a switch like that, then decided to punt on it for now after looking at access points. I figured I'd be better off just getting relatively inexpensive access points until I have a WiFi 6E client. By the time I replace my laptop 6GHz capable APs will be all over the place. My laptop is a couple years old and has 2x2 AC wireless so I just went with a 1GbE managed PoE Netgear switch for $80 and a couple of $100 TP-Link access points. In theory the laptop can do ~1700Mb, but that requires 2 160Mhz wide channels in the 5GHz range and almost no access points support 160MHz channels. That would cost a whole lot more and given how little I use my laptop just wouldn't be worth it.

I did run across one switch with 2.5GbE, 10GbE and PoE for <$500. Netgear MS510TXPP is apparently $450 (so under $500...) but OOS everywhere and doesn't have enough 10GbE ports for my setup.
 
zandor said:
Got the switch and router. Now I need some NICs, cables, transceivers and access points before I can start using them and none of them have shipped yet aside from the used Solarflare sfn8522-plus NIC from a recycler on eBay. So far SwitchOS seems pretty straightforward but RouterOS is going to take some getting used to. Figuring out what does what isn't too bad but finding settings is a pain in the ass.


I looked for a switch like that, then decided to punt on it for now after looking at access points. I figured I'd be better off just getting relatively inexpensive access points until I have a WiFi 6E client. By the time I replace my laptop 6GHz capable APs will be all over the place. My laptop is a couple years old and has 2x2 AC wireless so I just went with a 1GbE managed PoE Netgear switch for $80 and a couple of $100 TP-Link access points. In theory the laptop can do ~1700Mb, but that requires 2 160Mhz wide channels in the 5GHz range and almost no access points support 160MHz channels. That would cost a whole lot more and given how little I use my laptop just wouldn't be worth it.

I did run across one switch with 2.5GbE, 10GbE and PoE for <$500. Netgear MS510TXPP is apparently $450 (so under $500...) but OOS everywhere and doesn't have enough 10GbE ports for my setup.
Click to expand...
If all you want is cheap APs that work well, get a TP Link Mesh system. Honestly I've arleady installed 3 or so in various peoples' homes. Compared to the google system and the Netgear high end (which are great but $$$$$) the tp links are fantastic and they aren't too expensive.
 
I got a rather entertaining email today from FS.com. They wanted me to fill out a form saying I was not going to resell the Mellanox NICs. Apparently NVidia doesn't want them selling to resellers because of the chip shortage. Funny how they do this for NICs but not vid cards. Maybe I should pop over to the video cards forum and try to get some gamers wound up.

I actually looked at the TP-Link mesh stuff and checked the user manuals for a couple of them. Like most consumer WiFi gear they don't appear to support multiple SSIDs and mapping them to VLANs. They do have some sort of VLAN features, but it looks like that's just for IPTV and internet connections that want packets tagged. Apparently sticking a VLAN on the internet connection is common among fiber providers.
 
zandor said:
Got the switch and router. Now I need some NICs, cables, transceivers and access points before I can start using them and none of them have shipped yet aside from the used Solarflare sfn8522-plus NIC from a recycler on eBay. So far SwitchOS seems pretty straightforward but RouterOS is going to take some getting used to. Figuring out what does what isn't too bad but finding settings is a pain in the ass.


I looked for a switch like that, then decided to punt on it for now after looking at access points. I figured I'd be better off just getting relatively inexpensive access points until I have a WiFi 6E client. By the time I replace my laptop 6GHz capable APs will be all over the place. My laptop is a couple years old and has 2x2 AC wireless so I just went with a 1GbE managed PoE Netgear switch for $80 and a couple of $100 TP-Link access points. In theory the laptop can do ~1700Mb, but that requires 2 160Mhz wide channels in the 5GHz range and almost no access points support 160MHz channels. That would cost a whole lot more and given how little I use my laptop just wouldn't be worth it.

I did run across one switch with 2.5GbE, 10GbE and PoE for <$500. Netgear MS510TXPP is apparently $450 (so under $500...) but OOS everywhere and doesn't have enough 10GbE ports for my setup.
Click to expand...
As a reference point, in case you do go with any of the MS510 series, which sounds like you aren't, I run my 510tx with a noctua instead of the included fan. Thing was horrible. Uses a standard 4 pin connector and doesn't run too warm with the mod. I'm not hammering it all day with 10 gig traffic, but I"m sure even then it'd be fine. I paid $270 for mine. Seems like a steal these days.
 
Last edited:
Randall Stephens said:
As a reference point, in case you do go with any of the MS510 series, which sounds like you aren't, I run my 510tx with a noctua instead of the included fan. Thing was horrible. Uses a standard 4 pin connector and doesn't run too warm with the mod. I'm not hammering it all day with 10 gig traffic, but I"m sure even then it'd be fine. I paid $270 for mine. Seems like a steam these days.
Click to expand...
No, I didn't. I went with a MikroTik CRS326-24S+2Q+RM. 24 SFP+ 10gig ports and 2 40gig QSFPs. $500. It's overkill on the SFP+ ports, but I just might use those QSFPs eventually and it was only $100 more than the 16 SFP+ model. So far it's not loud, but I'm not really using it yet. It alternates between fans off and fans on low speed. I'm still waiting for my DAC and fiber patch cables and fiber transceivers. They should be here on Friday.
 
I finally got a couple machines plugged in. My workstation will have to wait since I have to disassemble my desk, clean everything, pull fiber to a wall plate, then put everything back together, but at least 10gig is working. Ok, maybe I don't technically have to clean, but I'd be nuts not to if I'm going to tear my whole setup apart. At least it seems to be working like it should:
copy.png
 
I got everything set up aside from the router. The MikroTik RB5009 has some compatibility issues with the 2.5G port that make it slower than 1G so I've been playing with it and doing a little digging. The problem tends to go away if you run it at 1G, but that defeats the purpose of having a 2.5G port. Fortunately I haven't bought the new cable modem to plug it into yet. After seeing this Reddit post I think I'm going to go with a NetGear CM2050V for the modem. Most of the 2.5G port problems seem to involve Motorola modems and sometimes Comcast provided ones. Also some issues with a buggy Intel 2.5G NIC. The 2.5G port works just fine doing local testing between computers. One of my boxes has a 2.5G port on the mainboard, so I plugged the 2.5G on the RB5009 into that and the SFP+ port into my switch, then ran some tests.
 
zandor said:
The only thing I want it for is the ability to learn how my HVAC system responds relative to outdoor temps. I like to let it get a little cold in the house overnight (I let it drop to 63F), but want it to warm up by the time I get up. The Nest will figure out when it has to turn the furnace on to hit 70F by wake-up time and will vary the start of the heating call based on outdoor temps. I'd like to swap it out for something that just has an outdoor temp sensor and uses that to decide when to turn the heat on.
Click to expand...
I haven't fully researched this yet myself, but I know industrial thermostat controls can do this type of stuff. Plus, because they're made for real business work and not just consumer farting around, they're more robust in terms of security and whatnot, a lot of times have a web-interface and whatnot. If I find anything when I do my search, I'll try to remember to post here. (y)
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
The catch with building a custom furnace controller is I have a network programming project queued up first that was half the reason for doing this whole 10gig upgrade. The other reason of course is just to have a 10gig lan.

I finally got the RB5009 deployed last weekend. I still have some setup to do on the VLANs, but it's working. I think it might be a bit faster than the Arris cable WiFi router I was using. Speed tests are coming back at around 720Mbps vs. usually just under 700 with the Arris, but that might just be luck. Getting IPv6 working was a little interesting. Comcast seems to expect you to use DHCP for your IP address range and auto configuration for routing, so the "add default route" option being on breaks your connection. At least they gave me the /62 I asked for so I have 4 subnets to play with. It was a lot easier than the Linux firewall/router setups I had back in the late '90s and early 2000s, but definitely not something I'd recommend to people who don't know at least a little about networking. That Linux setup was kind of fun. I had two static IPs on a DSL line. I had both run through one interface on my Linux box. One forwarded whatever I let through the firewall to my main PC, the other one ran ip masquerading and later NAT for everything else.
 
zandor said:
The catch with building a custom furnace controller is I have a network programming project queued up first that was half the reason for doing this whole 10gig upgrade. The other reason of course is just to have a 10gig lan.

I finally got the RB5009 deployed last weekend. I still have some setup to do on the VLANs, but it's working. I think it might be a bit faster than the Arris cable WiFi router I was using. Speed tests are coming back at around 720Mbps vs. usually just under 700 with the Arris, but that might just be luck. Getting IPv6 working was a little interesting. Comcast seems to expect you to use DHCP for your IP address range and auto configuration for routing, so the "add default route" option being on breaks your connection. At least they gave me the /62 I asked for so I have 4 subnets to play with. It was a lot easier than the Linux firewall/router setups I had back in the late '90s and early 2000s, but definitely not something I'd recommend to people who don't know at least a little about networking. That Linux setup was kind of fun. I had two static IPs on a DSL line. I had both run through one interface on my Linux box. One forwarded whatever I let through the firewall to my main PC, the other one ran ip masquerading and later NAT for everything else.
Click to expand...
I played with IPv6 last year as well, but, I wasn't confident enough in my firewall policies to feel certain I was being safe about it. Since everything is effectively plumbed straight to the internet in that case, I decided to back off until I had time to learn and set myself up better haha.
 
The Mikrotik default ipv6 firewall isn't bad. It'd basically deny everything except established connections and outgoing connections. It does let through some ports used by known VPN/tunneling protocols, but that's about it. Easy enough to turn that off. Other than that it doesn't even let ssh in. So really it's pretty standard. I'm basically just using the default for now aside from turning off unused VPN stuff. I'm sure I'll probably find a reason to punch a hole or two in it eventually, but at least for now it's working for me. It's much easier than setting up a firewall back in the late 90s/early 2000s. Back then a lot of internet stuff expected a real IP. These days it's just assumed you're behind a firewall running NAT.
 
zandor said:
The Mikrotik default ipv6 firewall isn't bad. It'd basically deny everything except established connections and outgoing connections. It does let through some ports used by known VPN/tunneling protocols, but that's about it. Easy enough to turn that off. Other than that it doesn't even let ssh in. So really it's pretty standard. I'm basically just using the default for now aside from turning off unused VPN stuff. I'm sure I'll probably find a reason to punch a hole or two in it eventually, but at least for now it's working for me. It's much easier than setting up a firewall back in the late 90s/early 2000s. Back then a lot of internet stuff expected a real IP. These days it's just assumed you're behind a firewall running NAT.
Click to expand...
Great to know! Maybe this weekend I'll fire it all back up again, if I get time. Thanks :)
 
Eulogy said:
Great to know! Maybe this weekend I'll fire it all back up again, if I get time. Thanks :)
Click to expand...
The default firewall was probably the thing I was most comfortable with setting up the RB5009. What, you recommended MikroTik and you're not even running one? :p That's ok, I like it anyway. I can read a firewall rule chain, and "made in Latvia". WTFBBQ? Latvia? I like it. I feel like I'm getting competitive capabilities at Soviet prices. It's not Cisco but it was only $200. Cheaper than the mass market consumer stuff than can handle a >1Gb WAN connection and it can handle full blown enterprise routing. Not with enterprise performance in this cheap little thing, but it offers those options if you need them.
 
zandor said:
The default firewall was probably the thing I was most comfortable with setting up the RB5009. What, you recommended MikroTik and you're not even running one? :p That's ok, I like it anyway. I can read a firewall rule chain, and "made in Latvia". WTFBBQ? Latvia? I like it. I feel like I'm getting competitive capabilities at Soviet prices. It's not Cisco but it was only $200. Cheaper than the mass market consumer stuff than can handle a >1Gb WAN connection and it can handle full blown enterprise routing. Not with enterprise performance in this cheap little thing, but it offers those options if you need them.
Click to expand...
I honestly just glazed over when I saw the 62 rules that were there for IPv6. I was like "Yeah, these look OK, but, what if I miss something because I just don't know?". Haven't felt a hard need to re-evaluate it since :)
 
Maybe I should have gone for a /60. Maybe Comcast would have given it to me. 16 subnets instead of 4. Not sure what I'd do with that many, but the IPv6 address space is huge. Aside from some reserved bits for link local and private networks and stuff it's nearly 2^64 subnets and the smallest subnet is 2^64 addresses. So I have 73,786,976,294,838,206,464 public IPv6 addresses available in my home network and that's not a lot. It's only 1/4,611,686,018,427,387,904th (1/2^62) of the total IPv6 address space. Kinda hard to comprehend aside from the "really big number" part. I'm generally ok with everything having a real IP address as long as I have a proper firewall that has a policy of "deny" for incoming connections unless I explicitly create an exception.
 
If you haven’t settled on your access points. Mikrotik CAPSMan let’s you easily manage everything related to the wifi network. You can even tune the client / AP relationship down to the decibel.
 
You must log in or register to reply here.
Back
Top