used 10gb switches

[applegrcoug]

I have a tiny 10gbe switch that I connect to my NAS and then a few systems. However, it doesn't have enough ports.

So I'm on ebay and i see all these older enterprise 10gbe switches for sale.

For example:
lenovo
cisco

So what's the catch? What am I missing here....do they need a $3000 license to work or something?

 

[uOpt]

They are loud and use a lot of power.

 

[ochadd]

Can't say anything about those specific models but... They are often loud and power hungry. Generally speaking about enterprise switches. Sometimes configured exclusively from command line interface. Can also require a license and/or registration to get access to firmware. Often have to replace 40mm fans as they've been running 24/7 for many years. Sometimes GBIC/SFP or DAC cable compatibility can be an issue.

If I were going to commit to making enterprise gear work it's worth considering SFP28 and QSFP28 stuff. Likely face similar challenges but the returns are bigger. The way I figure it is if a workload benefits from more than 2.5gbps and small desktop switches don't cut it, might as well jump into the deep end of 25/50/100gbps. I've priced out 10 and 40 gbps stuff for home many times but the final cost is too close to better gear to pull the trigger.

Couple of example
MikroTik CRS504-4XQ-IN
MikroTik CRS510-8XS-2XQ-IN

 

[MrGuvernment]

Stick to Brocade ICX like the 6450 or the 7250. DAC's from 10gTek work fine, FS.com has any other things you need. Brocade tends to be far less picky.

I run a 6450 for 2 years now and am working to get my 6610 online but additional fans added to keep the temps down so built in fans don't rev up.

Some models can be fan modded as well, but some can't or take a lot of work (you do not just wan tot try and swap lower RPM fans as the models like the 6610 run HOT)

https://forums.servethehome.com/ind...s-cheap-powerful-10gbe-40gbe-switching.21107/

 

[toast0]

I had a friend convince me not to get that exact cisco ebay listing last month.

It's probably very loud and has high power use, even at idle. It probably doesn't work at 10M, and maybe not at 100M, but I think it does. If you buy it, you're going to end up buying a lot more 10g NICs and some qsfp+ gear. Also, mayhaps no 2.5g/5g support, because it's geared for datacenters, not for consumers or wifi distribution.

That cisco series has 'honor system' licenses. You can do whatever, but might feel bad because you didn't pay for it.

Watch out for the 3000? series ciscos, I was seeing a lot of them with tons of 10G rj45 ports, but they're fabric extenders, and they don't do any in-chassis switching... It's a pumpkin unless you have a master switch to uplink it to.

The other downside is that qsfp+ became disfavored in enterprise, so upgrades are going to be ugh; you'll need to use a whole 100gb port on your new switch to feed this one with 40gb, and who wants that? Not very future looking for a switch that probably has more capacity than you really need for at least the next decade.

 

[dawsonkm]

Yeah you're right toast0. 40gb is a dead end. I'm glad i didnt jump in on it

 

[SamirD]

I have a tiny 10gbe switch that I connect to my NAS and then a few systems. However, it doesn't have enough ports.

So I'm on ebay and i see all these older enterprise 10gbe switches for sale.

For example:
lenovo
cisco

So what's the catch? What am I missing here....do they need a $3000 license to work or something?

This is the real 10Gb enterprise gear so made for a datacenter. If you have one in your home it will be no different than your 2u rack server or anything else. But if you don't, then you will notice the power usage and potentially the sound (just like for a rack server).

On a side note, crazy to see 10Gb rj45 now down to <$4/port when new unmanaged 2.5Gb switches are still far more expensive.

 

[MrGuvernment]

Yeah you're right toast0. 40gb is a dead end. I'm glad i didnt jump in on it

Dead in enterprise sure, but you can get Chelsio 40Gb dual port nics for dirt cheap (I have 2, one going into my TrueNAS and one going into my main desktop), but, a BrocadeICX 6450 - the fan mods to quiet it down need some electrical skills, so unless you can stick the switch in your basement... vs say a 7250 where you get 8 x 10Gb ports and could bond them....(unless you need more than 8 x 10Gb ports)


And power draw is not awful either

https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=BrocadeICX+7250&_sacat=0

 

[dawsonkm]

Dead in enterprise sure, but you can get Chelsio 40Gb dual port nics for dirt cheap (I have 2, one going into my TrueNAS and one going into my main desktop), but, a BrocadeICX 6450 - the fan mods to quiet it down need some electrical skills, so unless you can stick the switch in your basement... vs say a 7250 where you get 8 x 10Gb ports and could bond them....(unless you need more than 8 x 10Gb ports)


And power draw is not awful either

https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=BrocadeICX+7250&_sacat=0

View attachment 679243

I really didn't think about it like that. Is a great point. I'm almost at the point of needing something more than 10Gb between servers. Thanks for the comment.

 

[MrGuvernment]

I really didn't think about it like that. Is a great point. I'm almost at the point of needing something more than 10Gb between servers. Thanks for the comment.

Yup, assuming you have multiple active connections utilising the system, they should each balance across each link evenly (different end users sort of thing) so dual bonded 10Gb would give each user 10Gb each themselves, assuming your backend storage can keep up.

 

[applegrcoug]

So here is what I'm up to, maybe someone that knows this stuff better can give an assist.

I have cat5 all throughout my house that all comes back to a central storeroom in my basement. In there, I also have a couple home servers. One does truenas with iscsi shares where we house our steam libraries, then it also does all the security cam stuff and houses all the emby libraries. The other is a minecraft server and does ollama. They take up a lot of space and one is pretty loud, so I want them out of the main storeroom. The server that gets the most traffic, already has a nic with dual 10gbe ports, so I thought about aggregating them. But yeah, some of that old 40gb stuff is cheap. My truenas server can saturate a 10gbe connection, so it keeps loading games nice and snappy.

I am just wrapping up construction of a new pool pump house, storage and changing room. So, I want to take my two servers and put them in the pump house. There is plenty of space away from the pumps and I'm getting power put in for the servers. Then I get the added benefit of keeping the thing warmish in the winter.

Sooo, I want to chase a cable or two from the network panel and then run it overhead strapped to some of those sting patio lights to keep it discreet and then into to the pump house.

 

[toast0]

So here is what I'm up to, maybe someone that knows this stuff better can give an assist.

I think you can move your servers to the pump house, but you're gonna need to keep a louder enterprisey switch in the store room.

As you're contemplating going beyond 10g, you should run fiber to your pumphouse for your servers; I don't think there's hope for twisted pair based ethernet beyond 10g and it won't be cheap if it happens. I'm not a fiber junky, but I'd expect there's some overlap in fiber termination standards so you can get 10g optics now, and upgrade later with the same cabling, someone else should know.

I assume you've got something in the store room already with 4? spf+ ports that can do link aggregation. If so, I'd say pick up a ebay sfp+ switch for the store room; these are super cheap cause enterprise is beyond 10g, and home users want at least some rj45 for their regular computers. Go ahead and run 4x fibers and run 4x aggregation between your current switch and your pumphouse switch. Then you can run 2x da s for your media server and 1 or 2x for your nas for now, and upgrade later.

 

[SamirD]

So here is what I'm up to, maybe someone that knows this stuff better can give an assist.

I have cat5 all throughout my house that all comes back to a central storeroom in my basement. In there, I also have a couple home servers. One does truenas with iscsi shares where we house our steam libraries, then it also does all the security cam stuff and houses all the emby libraries. The other is a minecraft server and does ollama. They take up a lot of space and one is pretty loud, so I want them out of the main storeroom. The server that gets the most traffic, already has a nic with dual 10gbe ports, so I thought about aggregating them. But yeah, some of that old 40gb stuff is cheap. My truenas server can saturate a 10gbe connection, so it keeps loading games nice and snappy.

I am just wrapping up construction of a new pool pump house, storage and changing room. So, I want to take my two servers and put them in the pump house. There is plenty of space away from the pumps and I'm getting power put in for the servers. Then I get the added benefit of keeping the thing warmish in the winter.

Sooo, I want to chase a cable or two from the network panel and then run it overhead strapped to some of those sting patio lights to keep it discreet and then into to the pump house.

Two comments.

One is that every single electrical panel in every single pool pump room has ended up rusted due to the number of various stuff in the environment. I would not want to put a server there.

Two is that over head cables will eventually snap or stop working (or both) as we had one of those professional installed on a guy wire and after less than a decade it stopped working.

My suggestion would be to either run some fibre along with whatever electrical is being run or to use powerline adapters.

 

[applegrcoug]

Two comments.

One is that every single electrical panel in every single pool pump room has ended up rusted due to the number of various stuff in the environment. I would not want to put a server there.

Two is that over head cables will eventually snap or stop working (or both) as we had one of those professional installed on a guy wire and after less than a decade it stopped working.

My suggestion would be to either run some fibre along with whatever electrical is being run or to use powerline adapters.

I live in a desert and things don't really rust here, so I'm not too concerned about that. My stuff will be like 12ft from the pumps anyway. To be honest, my bigger concern is summer heat. I just hope the building exhaust fan I installed will turn the air over quick enough.

You can't run fiber and copper in the same conduit...against code here. My conduit was installed years ago when I redid the yard and I'm lucky they signed off on that.. Not only that, my main panel is in a completely different place than is my patch panel. I'd be time ahead to replace the cat6 every three years than go from the basement up a wall someplace and then into the attic and then down to the electrical panel and into the conduit.

I have another place where I did some overhead and it is still working......except for a giant maple tree limb that fell on it, but that's a different issue. The original plan I had was to run two UV stabilized and shielded cat6 wires from the house to the pump house. It is a pretty straightforward run of cable to get from my patch panel outside. Cat6 still may be the simplest option. I'd just need a couple switches to aggregate things the way I figure. I suppose if it all worked great, I could rerun fiber.

Originally I was thinking about a couple of these switches until I saw the used enterprise gear for less.

 

[magnetik]

MikroTik CRS309-1G-8S+in is what I went with. Didn't have room for rack gear or needed the extra noise. SMB multichannel works great fwiw. Good way to utilize unused ports.

 

[SamirD]

I live in a desert and things don't really rust here, so I'm not too concerned about that. My stuff will be like 12ft from the pumps anyway. To be honest, my bigger concern is summer heat. I just hope the building exhaust fan I installed will turn the air over quick enough.

You can't run fiber and copper in the same conduit...against code here. My conduit was installed years ago when I redid the yard and I'm lucky they signed off on that.. Not only that, my main panel is in a completely different place than is my patch panel. I'd be time ahead to replace the cat6 every three years than go from the basement up a wall someplace and then into the attic and then down to the electrical panel and into the conduit.

I have another place where I did some overhead and it is still working......except for a giant maple tree limb that fell on it, but that's a different issue. The original plan I had was to run two UV stabilized and shielded cat6 wires from the house to the pump house. It is a pretty straightforward run of cable to get from my patch panel outside. Cat6 still may be the simplest option. I'd just need a couple switches to aggregate things the way I figure. I suppose if it all worked great, I could rerun fiber.

Originally I was thinking about a couple of these switches until I saw the used enterprise gear for less.

Ah, that's pretty awesome then. And unless you're hitting above 110F in the pump room, I wouldn't worry about the servers as their screamer fans will kick into high gear. Having an exhaust fan will help out a lot so worse case you can upgrade that.

If you've had good success with overhead, I'd say definitely go for it and run some fibre up there while you're at it as there's some good preterminated stuff available--the labor is what gets you on doing an overhead so I'd run everything you could in one shot.

 

[SamirD]

SMB multichannel works great fwiw.

View attachment 679581

This is probably the only thing about the newer versions of smb that I really like--you can literally just pick up a few 1Gb nics for nearly nothing and connect them together if you need more bandwidth somewhere and it 'just works'.

 

[magnetik]

This is probably the only thing about the newer versions of smb that I really like--you can literally just pick up a few 1Gb nics for nearly nothing and connect them together if you need more bandwidth somewhere and it 'just works'.

Dropped a quad port 10g card into my TrueNas server (16 x 12tb configured w/ mirrored vdevs) and was able to transfer faster than configuring the switch to aggregate multiple connections using LACP. All by just plugging in some extra cables. Need to try the same transfer with RDMA enabled nics.

 

[SamirD]

Dropped a quad port 10g card into my TrueNas server (16 x 12tb configured w/ mirrored vdevs) and was able to transfer faster than configuring the switch to aggregate multiple connections using LACP. All by just plugging in some extra cables. Need to try the same transfer with RDMA enabled nics.

Yep, this is always the way (especially with that many spindles since I would guess you're even maxing out 10Gb). I actually wasn't talking about LACP but just pure smb3 multi-path where you can literally just give it additional paths between the two points and windows will just use them--no LACP, nic trunking, or anything special needed.

 

[Deadjasper]

I have a Arista DCS-7050TX-64-F I picked up from eBay. It's an Ethernet 10g switch. Works great but as others have pointed out, power usage and noise are real issues. This beast has 2 1000w PSU and consumes about 200w when running. I only fire it up when transferring large amounts of data. It's not as noisy as a lot of switches but it's not quiet either.

 

[ochadd]

I picked up a used Mokerlink 10G080GTM for $199 last week. An 8x10gb ethernet switch that does 1/2.5/5/10gbps. Just got it an hour ago and it is completely silent at idle and 1 gbps transfers. Fans are not running. Web interface is nicely laid out. It's a fully managed switch. Have not tried 10gbps connections yet.
Enabled SSH but get bad username even after creating a new admin user. Not a big issue to me but suspect. On Amazon they go for around $280 new.

My plan with it is a 2x10gb trunk from a Server 2022 Hyper-V host for VM traffic. Not storage traffic, ISCSI. Then a 2x10gb link from primary client. Both using Intel X550-T2 NICs so two ports on each side. Hoping SMB multichannel works for the client. No idea how I'd make SMB multichannel work for the VMs on the server side and I know how to setup trunks/teaming so starting there. In the end I hope to effectively have 20gbps throughput. Full backup times for client are 26 minutes on 2.5gb connection now. Hoping to get it under 10 minutes.

 

[DogsofJune]

I always wondered how well the Mokerlink stuff fared.

 

[ochadd]

I always wondered how well the Mokerlink stuff fared.

Anything you want to know, it's on my desk right now with a laptop connected. My first device from them as well. I paid $200 for the switch but spent $320 on the X550-T2 NICs. Seems dumb but I had X540, Marvel Aquantia, and one other in my cart several times but nothing had straight forward drivers for Server 2022. I'm not that cheap, I'll pay extra for something to just work. There are some Intel X540-T2 NICs on Ebay right now that are so cheap, two for $37. Brand new Chinese units.

 

[SamirD]

monkeylink and 'new chinese units'--no thanks.

But no doubt used x540 cards have really fallen in price. I can't remember which ones they were, but I've found dual 10Gb rj45 cards for $15 shipped before. And with SMB multipath, you realistically can just put in a 4-port 1Gb card and you're already halfway to 10Gb pretty easily.

Oh, found the dual 10Gb cards--they're $10!
https://forums.servethehome.com/index.php?threads/x520-da2-for-10.45520/

 

[ochadd]

monkeylink and 'new chinese units'--no thanks.

But no doubt used x540 cards have really fallen in price. I can't remember which ones they were, but I've found dual 10Gb rj45 cards for $15 shipped before. And with SMB multipath, you realistically can just put in a 4-port 1Gb card and you're already halfway to 10Gb pretty easily.

Oh, found the dual 10Gb cards--they're $10!
https://forums.servethehome.com/index.php?threads/x520-da2-for-10.45520/

The Mokerlink is not silent once you start plugging stuff into it. I have 7 of the 8 ports in use now and even with very little load it's the loudest thing in my network closet. It's not hot but the fans are spun up. This is not a switch you will want to run outside of a closet. I might need to change the fans out as I sit 3 foot away and can hear it through the door.
Edit: I tried swapping the fans to some Noctua but this Mokerlink 10G080GTM has two pin fans. Two on mine, the non-POE version. Decided to order a couple 2 pin to 4 pin adapters instead of cutting and splicing. Noctua NF-A4x20 PWM physically fit fine. Plenty of space between the main board and the fan.