School Me on how to start this fiber setup please.

IceDigger

IceDigger

[H]F Junkie
Joined
Feb 22, 2001
Messages
12,088
Goal is to setup a working 10gb network in house and external shed 300' away from house.

My biggest hangup is the cabling as I know almost nothing about fiber cables.

Internets - 1gb/1gb Fios
Cables Needed (do not know anything about fiber but have used cat6/7 for years now) ?
Switch Needed (would like all ports being 10gb) ?
Router ? Currently have an older lenovo i3 8gb ram desktop w/ intel dual 1gb rj45 ports and pfsense.
Router OS (pfsense, microtic, etc...) ?
Anything else ?

Budget of around $500-$1k
Help a brother out.

BTW, I need some more cocaine!
 
At 300 feet, you could run 10GBaseT. If you go optical, you'd want to run at least two strands of multi-mode fiber (you may want to run spares as well); then you'd need switches at both ends with sfp+ and multimode transceivers. You can do direct burial in a shallow trench, but if you can lay in a conduit, it might be easier to replace or augment later. I don't think fiber termination is as easy as copper, so if you can, you might see if it makes sense to run preterminated cables; it's been ages since I handled fiber, but the ends weren't that big, so it could be possible depending on the building openings.

If your internet speed isn't changing, your existing router should be sufficient, your LAN traffic doesn't need go through the router, so it's fine if your lan is 10g and your router is 1g.

How many 10G ports you need for your switches depends on what you're trying to do... Maybe you have a NAS in the house, and a couple clients in the shed, so you want 10G for the NAS, and 10G for the clients (bottleneck would be the 10G uplink), maybe you have a lot of clients in the shed that only need 1G each. Maybe you have a lot of different servers, so nothing needs 10G, but you still need a 10G uplink between switches because the aggregate traffic is more than 1G. You might be able to do some link aggregation to 2x or 4x 1G for servers.

There's a lot of switches with one or two sfp+ and then 24-48 1G ports, which may work well for you.
 
For 300', definitely some pre-terminated fibre that's buried. You don't want to get into grounding variances which will induce current across copper and you are pretty much at the length limit for ethernet and shouldn't push it. 10Gb sfp+ are easy to find these days and switches are too so should be easy enough to get equipment on both ends.

I'd definitely look at used equipment and going with sfp+ and dac where you can as that will save you tons of money. I think the bulk of your spend will be in the switches.
 
SamirD said:
For 300', definitely some pre-terminated fibre that's buried. You don't want to get into grounding variances which will induce current across copper and you are pretty much at the length limit for ethernet and shouldn't push it. 10Gb sfp+ are easy to find these days and switches are too so should be easy enough to get equipment on both ends.

I'd definitely look at used equipment and going with sfp+ and dac where you can as that will save you tons of money. I think the bulk of your spend will be in the switches.
Click to expand...
Can't do burried, too much asphalt in between.

Right now I just have cat 7 hanging in the air from the house to the shed.

It's almost 200' from where the router is in the house to the opening in the house where I can hang the wire. Then about 30' to the shed, and another 50' inside the shed to where I want it.

So not too much is outside. Just can't bury it.
 
SamirD said:
For 300', definitely some pre-terminated fibre that's buried. You don't want to get into grounding variances which will induce current across copper and you are pretty much at the length limit for ethernet and shouldn't push it. 10Gb sfp+ are easy to find these days and switches are too so should be easy enough to get equipment on both ends.

I'd definitely look at used equipment and going with sfp+ and dac where you can as that will save you tons of money. I think the bulk of your spend will be in the switches.
Click to expand...
One thing I know nothing about is sfp, sfp+, qsfp, dac
 
I think SamirD's advice about length is absolutely right... On the other hand, you've already got a twisted pair run, if you're willing to spend some money on something that might not work, I'd totally try 10GbaseT on the run you have; but I also ran 1G over cat3 at my last house (maybe 15ish feet underground between the house and the detached garage, with a 30 foot run from the panel to where the cat3 came in), so I'm not like a pinacle of following advice or standards ;)

Re: grounding, are the two buildings on the same electric service, or sepeate meters? If they're on the same service, in theory the grounds should be bonded? In practice, there's a non-negligible chance of frying your switches, I suppose; but new switches every so often should be compared against the cost to fish optical cable through and any general equipment replacement schedule.

*fp is more or less an external transciever, so switches and network cards can be built without regard to the physical medium of connectivity.

Sfp is 1G, sfp+ is 10G (sometimes exactly 10g, sometimes could do other standard ethernet speeds) qsfp is quad sfp or something, usually 40g or more, sometimes with 4x physical connections. Dac is direct attached copper, a very low cost thing with sfp at both ends and cable attached, useful for short distance connections from server to server or server to switch, or switch to switch. If you need 10G to servers, it makes sense to put your server near your switch and use dac, rather than a short fiber or 10GbaseT run, because it will cost less and often generate less heat too.
 
toast0 said:
I think SamirD's advice about length is absolutely right... On the other hand, you've already got a twisted pair run, if you're willing to spend some money on something that might not work, I'd totally try 10GbaseT on the run you have; but I also ran 1G over cat3 at my last house (maybe 15ish feet underground between the house and the detached garage, with a 30 foot run from the panel to where the cat3 came in), so I'm not like a pinacle of following advice or standards ;)

Re: grounding, are the two buildings on the same electric service, or sepeate meters? If they're on the same service, in theory the grounds should be bonded? In practice, there's a non-negligible chance of frying your switches, I suppose; but new switches every so often should be compared against the cost to fish optical cable through and any general equipment replacement schedule.

*fp is more or less an external transciever, so switches and network cards can be built without regard to the physical medium of connectivity.

Sfp is 1G, sfp+ is 10G (sometimes exactly 10g, sometimes could do other standard ethernet speeds) qsfp is quad sfp or something, usually 40g or more, sometimes with 4x physical connections. Dac is direct attached copper, a very low cost thing with sfp at both ends and cable attached, useful for short distance connections from server to server or server to switch, or switch to switch. If you need 10G to servers, it makes sense to put your server near your switch and use dac, rather than a short fiber or 10GbaseT run, because it will cost less and often generate less heat too.
Click to expand...
Thanks for the explanation!

Right now the current 2.5gb switch is right next to the server and router. So I basically need one long line going from a new 10gb switch to the other building.

FIOS box -> Pfsense Computer -> 6' cat7 cable - > 2.5gb 5 port switch -> 200' cat 7 -> attic over attached garage 2.5gb 5 port switch (gets hot/cold) -> 200' cat7 cable going to detached garage/workspace -> Threadripper Desktop

What I would like
FIOS box -> Pfsense Computer -> 10gb switch -> 300' - 400' cable (get rid of switch in attached garage attic and just use one long cable) -> 10gb switch in detached garage

Our property is weird, we have a bunch of outside buildings. Our own little compound on 5.5 acres.
 
So I got in 2 FREE QLogic QLE2564 cards w/those little adapters already in them!

They will only go up to 8gb instead of the 10gb I was looking for but they were FREE!

Will I have any trouble with them?
 
IceDigger said:
So I got in 2 FREE QLogic QLE2564 cards w/those little adapters already in them!

They will only go up to 8gb instead of the 10gb I was looking for but they were FREE!

Will I have any trouble with them?
Click to expand...
Pretty sure those cards are FC HBA and not NICS. Also, I'm afraid to ask but .... All that external cable you describe in #7 is plugged into grounding blocks on both ends of the external pieces before being attached to your gear yes? If you were to switch to 1 run at 400 feet assuming 10G SR you would need OM3 or better fiber.
 
Last edited:
Not sure the diff between FC HBA and a regular NIC. My forst foray into enterprise other than SAS raid cards.

I did buy OM3 outdoor cable. Should be in Monday maybe?

Another free thing I just got was a Dell Poweredge T130 w/96GB RAM.

It already has 2x 1gb regular rj45 nics on it and I can add 2 of the QLE2564 to them and install ClearOS rather than pfsense and make a super router/switch?

Don't ask where I get the "free" stuff ;)
 
FC HBA are great if you're running fibre channel not so much (read worthless) for ethernet.
 
Damn, recommend me some cheap 10gb nice please. Need around 6 of them.
 
Cards are almost certainly going to be SFP+. Cards with built-in optical transceivers are weird and I'd be scared of buying them.

OM3 is what the transceivers need to support. Most of them support multiple fiber types, at least for multi-mode like OM3. Like OM3 and OM4 IIRC. OM3 is kinda what you want unless you need OM4 for a bit more cable length, just because OM4 costs a lot more. OM3 is good to 300m, so it's overkill for 300'. I'm running all OM3 for fiber links here. You'll want to pick the cards out before buying transceivers so you can ask around for what works with card X. I'm running FS.com generic stuff for my optical transceivers. Works with Mellanox ConnectX-4 and Solarflare sfn8522-plus and my Mikrotik switch, but YMMV. FS.com also carries cables. Arguably they're more useful for cables, since used from eBay is a much better way to get optics than cables.

Also don't forget about DAC cables for short runs. DAC means "direct attach copper". It's a cable that plugs directly into the cage you'd stick an optical transceiver in, so there are SFP, SFP+, QSFP, etc. versions of them. DAC cables only support short lengths. I forget how much exactly but it's like "same rack or a couple racks over". DAC is cheaper than buying 2 transceivers and a fiber patch cable. If you're looking for 6 cards I'm guessing there's at least one server in the mix that'll be sitting right next to the switch. For 10Gb you'll want SFP+. QSFP is basically 4 SFP+ in one plug. It's used mostly for 40Gb but can also be split out into 4 SFP+ connections if the switch, NIC, etc. supports it... at the cost of buying an adapter.

Intel and Mellanox are good bets for cards, especially if you need Windows desktop support. The cheap, low hassle way to get them is from a recycler on eBay. One of those companies that disposes of off-lease and retired corporate equipment. I'm running mostly Mellanox, mostly because I wanted to mess with OFED programming. I bought a Solarflare so I could play with their proprietary APIs and I don't recommend them unless you have an actual use for Onload or their proprietary APIs or it's a cheap, used older model from eBay that supports the OS you're using. Those can be ok - Linux includes a basic driver for them.

You might need to rig up some sort of cooling for server NICs if you're not using a server chassis. Gaming cases aren't necessarily good at keeping server NICs cool. These cards tend to have small heatsinks and no fan, and expect some airflow across the card. Gaming cases assume the CPU and vid card have a fan or water cooling and all they have to do is keep the air temp in the case down. That's not good enough for some NICs. So you might need a 40x10mm fan & a zip tie or something. Probably run it on 7V. So far I haven't had any trouble with the Mellanox ConnectX-4s, but I used to have an old Mellanox Connect-X card that needed a fan on it. Don't get too paranoid. Just check thermals and rig something up if a NIC is roasting. That old Connect-X card from like 2010 or something was fine unless I was gaming. It was in an SLI rig between 2 GTX680s.

For 10Gb switches used gear off eBay is the cheap option unless you want a small one, but those are just about all going to be 1U rackmount switches. Some of them are really fucking loud, so check specs before buying.

I'm running a MikroTik CRS326-24S+2Q+RM switch. $500+tax+S&H new for 24 sfp+ and 2 QSFP (40Gb). It's loud enough I wouldn't want it in my office, but it's not too bad and doesn't bother me sitting down in the utility room with my furnace, washer and dryer. I don't hear it unless I'm in the utility room. The main reason I mention MikroTik though is they have some small, cheap 10GB sfp+ switches, like a CRS305-1G-4S+IN with 4 sfp+ and one 1Gb port for $150 and an 8 sfp+ CRS309-1G-8S+IN for $270. Another cool thing they have is a 1/2.5/5/10Gb RJ45 transceiver that fits in an sfp+ cage. That's part of why I got the MikroTik switch instead of going with a used Enterprise switch off eBay. $65 or so a pop but worth it if you want to keep that 2.5Gb switch around and plug it into a 10Gb switch.
 
Its alive!

Thanks to whoever bought my peplink wwan devices!

Got some cheapo sunlink 10gb cards! All hooked up and wow what a diff in speed.

Linux isos transfer so much faster now!
 
Good to hear! Glad you got your 10Gb setup up an running. Now you just need to figure out how to plug a RWD Cadillac into the 10Gb network. Preferably a full size model. 10g Fleetwood? :D :p
 
You must log in or register to reply here.
Back
Top