As far a setting up an optical metropolitian network goes what do I need to know, I'm going to use it for connecting WiMAX base stations to my network infrastructure (when it gets installed). I've been doing some reading but it's not making much sense to me, I know there are different topologies, circuit or Point to Point. Which would work for me? Does a ring circuit offer reduncany in that if one link between two nodes is disrupted the signal can be put round the other way? Another question, how would I connect the optical to my GigE network that all my servers will run on, wouldn't that be a huge bottleneck?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Optical Metro Network
- Thread starter Keith130
- Start date
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Is this a serious question? This is a very complicated setup, if you don't know how this is certainly not something to experiment with.
Most metro systems use sonet rings.
Most metro systems use sonet rings.
BollWeevil
Limp Gawd
- Joined
- Feb 9, 2005
- Messages
- 373
No kidding, and how much budget do you have for this? Metro Optical rings are more than just a conceptual idea from a textbook, they usually employ kilometers of fiber and require equipment ranging from several hundred thousand to millions of dollars in cost.
Yes, im completely serious. At the moment this is still in the conceptual stages of a buisness plan but we want to know everything we can before we have to outsource for experts that'll cost money. Anyway in my case it will be costing pounds as the proposed network is going to be installed in the UK.
So perhaps now you know that im not some random kid joking around someone could answer my questions?
What is a sonet ring? And about this DWDM technology, I dont understant how some sources state diffent max speeds for optical fibre, is that do do with DWDM creating 'Virtual' pipes?
Does a ring circuit offer reduncany in that if one link between two nodes is disrupted the signal can be put round the other way?
Another question, how would I connect the optical to my GigE network that all my servers will run on, wouldn't that be a huge bottleneck? Also how would I connect the WiMAX basestations into it as that is the primary function of the network, would this not also create bottlenecks?
So perhaps now you know that im not some random kid joking around someone could answer my questions?
What is a sonet ring? And about this DWDM technology, I dont understant how some sources state diffent max speeds for optical fibre, is that do do with DWDM creating 'Virtual' pipes?
Does a ring circuit offer reduncany in that if one link between two nodes is disrupted the signal can be put round the other way?
Another question, how would I connect the optical to my GigE network that all my servers will run on, wouldn't that be a huge bottleneck? Also how would I connect the WiMAX basestations into it as that is the primary function of the network, would this not also create bottlenecks?
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Keith130 said:
A sonet ring consists of two sets of fiber. If a link in the ring fails, it can automatically fail over to the other ring. Sometimes it can do this fast enough that a customer monitoring their network won't even notice.
The optical network can be terminated on Cisco ATM cards running on devices such as a 6509 (and likely devices from Juniper and maybe foundry), or on more specilized ATM switches, like Nortel's 7k or 15k passports. This can then be converted for gig or 10 gig ethernet use in the LAN.
You are unlikely to bottleneck WAN links with LAN links. For example, a OC-48 runs at 2.4Gbps. You can easily achieve this speed by aggregating 2 gig-e links, and you can bundle up to 8, easily pushing more than the WAN can handle.
Now, the thing is, most people don't build their own optical network. This is what telcos are for. Building your own sonet ring metro area network will cost millions, actually more like 10s of millions. Why not rent from a reliable telco who has already invested in the infrastructure? Sure, you'll end up shelling out probably 50-100k per month for circuits, but thats nothing compared to trying to lay fiber in a city.
Ok thanks for your answer.
Would it be advantageous to use an ATM network since the primary function of the optics would be for carrying VoIP, however it would be used for other internet functions such as browsing, email etc? Would it not be better to use TCP/IP?
Does link aggregation require special switches or can it be done on any switch, presumably [the higher end] Cisco swiches will have this as that is whe we are planning on using.
I take your point about using an existing fibre network and I have researched it, for example I found this comany (http://www.fibernet.co.uk) which would seem to meet our needs. Our thoughts however are that if we can finance our own network in the first instance there will be no need to:
1. Outsource and be dependant on another company
2. Pay monthly fees.
and 3. we can run more cabling than we require giving us the ability to upgrade the infrastructure if/when nessicery, and/or to wholesale out any dark fibres ( <-- is that the correct use of the term?)
Would it be advantageous to use an ATM network since the primary function of the optics would be for carrying VoIP, however it would be used for other internet functions such as browsing, email etc? Would it not be better to use TCP/IP?
Does link aggregation require special switches or can it be done on any switch, presumably [the higher end] Cisco swiches will have this as that is whe we are planning on using.
I take your point about using an existing fibre network and I have researched it, for example I found this comany (http://www.fibernet.co.uk) which would seem to meet our needs. Our thoughts however are that if we can finance our own network in the first instance there will be no need to:
1. Outsource and be dependant on another company
2. Pay monthly fees.
and 3. we can run more cabling than we require giving us the ability to upgrade the infrastructure if/when nessicery, and/or to wholesale out any dark fibres ( <-- is that the correct use of the term?)
It's probably best to lease wavelengths. Unless you are planning to spend 10's of millions of dollars laying the fiber.
On top of that, you are talking a very large chunk of cash for the optical switches, and repeaters needed.
Not to mention, you are likely to pay monthly costs for the easement of the fiber, and it's associated maintance.
On top of that, you are talking a very large chunk of cash for the optical switches, and repeaters needed.
Not to mention, you are likely to pay monthly costs for the easement of the fiber, and it's associated maintance.
Keith130 said:
Please don't take offense to this, but if you don't know the difference between ATM (layer 2) and IP (layer 3) then you are in way over your head and really don't have any business pursuing something of this magnitude. I would suggest you hire a consultant and/or employ an established provider/carrier to help with this.
P.S. - I hope you have an enormous budget if you are going to try and lay your own fiber across a city because it is going to be very very expensive...
No I wont take offence, Im not much of a networking guy I admit that, I find it difficult enough to keep my iBook connected to my various WiFi networks. Thats why I came back the forum; to learn as much as I can before we are forced to seek expert help. I do at least know you are talking about the OSI model with layer 2/3. But could you explain what im missing, I never really got a hold of that in Higher computing.
Luckily we do have a large budget that is flexible enough to cover potentially huge costs, we understood from the beginning that this could end up costing millions but we are creating a completely new service combining areas of all our buisness' into one package deal. I'd love to tell you but we arent going public yet.
About the repeaters, what is the max distance optics can travel without having to be repeated, I thought that was one of the main advantages of optics that it didnt have to be repeated so often?
I have someone looking into this for me but if anyone knows it would be helpful, are there ducts or pipes avaliable in cities for laying cabling? If not we have a work around but it could be difficult to pull off.
Luckily we do have a large budget that is flexible enough to cover potentially huge costs, we understood from the beginning that this could end up costing millions but we are creating a completely new service combining areas of all our buisness' into one package deal. I'd love to tell you but we arent going public yet.
About the repeaters, what is the max distance optics can travel without having to be repeated, I thought that was one of the main advantages of optics that it didnt have to be repeated so often?
I have someone looking into this for me but if anyone knows it would be helpful, are there ducts or pipes avaliable in cities for laying cabling? If not we have a work around but it could be difficult to pull off.
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Keith130 said:
ATM is Layer 2. IP is L3 and TCP is L4 on the OSI model. Your TCP/IP traffic rides the ATM network in a very similar fashion to how physical switch links operate at L2, just with different media (sorta, this is a crude way to compare it). ATM is a long distance transport mechanism that can be used with L3 IP networks.
Keith130 said:
It is common on any decent switch anymore, from 200 dollar dell switches to 200k cisco GSRs. If you are using any Cisco switch from the 2900 series and up, you will have link aggregation. I am assuming your core will probably be 6500 series Cisco switches from what you are saying, and you can go up to 8 10Gb links in an aggregated link. Now thats some serious bandwidth.
Keith130 said:
I would look seriously at ROI and breakeven on a project like this. If it costs you 10 million (a very conservative estimate) to set this up, it may take 8 years to reach breakeven, not even considering the opportunity costs of capital used in the project, nor inflation.
Plus, you have to have someone monitor and maintain a fiber network, which is expensive and requires expert techs.
In addition, you will probably need to tear up streets in order to run your cable, unless you plan to use the sewer system. You may run into some very opposed city governments.
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Keith130 said:
The distance depends on the type of media being used. I don't have my book here at home that has the standards in it, but I know some media can run 65km before requiring a repeater. There are other types of fiber, generally used in longer distance LAN connections that can only go 200 or so meters. Still much further than copper, however.
Sewer systems are often used for adding additional fiber capacity. They just have the unfortunate characteristic of flooding in some areas during heavy rain fall.
What kind of maintenance does a fibre network require? I have a 10/100 network at home with 802.11b & g WiFi and it requires no maintenance besides the rebooting of my smoothwall or modem when im torrenting healivy. Would it be nessicery to employ someone to sit around and watch the equiptment in head office? We are looking for a five nines uptime network and off the top of my head thats 5.2 minutes of downtime a year.
The only maintenance i can see is having qualified staff watch the network and upgrade the IOS when necessary.
However, since you stated you want five nines uptime, your costs at least doubled, if not more.
Just imagine running a fiber from 1 building to another. One would think simple, right? Sure, if you just want a cisco 6509 at each site with 1 fiber running across it. Oh you want redundancy?, now you need at least 2 6509 at each site with at least 4 fiber connections running between them. Oh you want 5 nines? Now we are talking about multiple connections from different carriers going into each of the sites.
All this adds up. And this is only for 2 sites. Multiple above x the number of sites.
Now before i join the bandwagon in seriously recommending some serious help in this, i just want to review what the OP really wanted.
Explain what you mean by Wimax connecting to your network. Are you using Wimax to connect different buildings to your corporate network? Are you using Wimax as a different path out the building?
However, since you stated you want five nines uptime, your costs at least doubled, if not more.
Just imagine running a fiber from 1 building to another. One would think simple, right? Sure, if you just want a cisco 6509 at each site with 1 fiber running across it. Oh you want redundancy?, now you need at least 2 6509 at each site with at least 4 fiber connections running between them. Oh you want 5 nines? Now we are talking about multiple connections from different carriers going into each of the sites.
All this adds up. And this is only for 2 sites. Multiple above x the number of sites.
Now before i join the bandwagon in seriously recommending some serious help in this, i just want to review what the OP really wanted.
Explain what you mean by Wimax connecting to your network. Are you using Wimax to connect different buildings to your corporate network? Are you using Wimax as a different path out the building?
No, the metro core is going to be used to move WiMAX around the city providing city wide coverage, more than one base station is going to be needed 'cos can you imagine sharing 280Mbps theoretical bandwidth with an entire city? The actual bandwidth will be less like in other WiFi senarios? It's not happening if we want to provide baseline 2Mbps per connectee plus I wouldn't have thought that the point-to-multipoint architechure of WiMAX that they are aiming for won't scale to several hundred thousand people per base station.
Why would I need two 6509's per PoP? I thought I would only need two of these in the main office (where all the app. servers will reside) and then have two 15600's in the core with two 15454's in each PoP then run muliple fibres between them. Then have GigE cards or whatever media in the 15454's to connect with the WiMAX hardware. Am I way off?
Why would I need two 6509's per PoP? I thought I would only need two of these in the main office (where all the app. servers will reside) and then have two 15600's in the core with two 15454's in each PoP then run muliple fibres between them. Then have GigE cards or whatever media in the 15454's to connect with the WiMAX hardware. Am I way off?
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Keith130 said:
Unless there are a huge # of connections at each POP, there is no need for 2 6509's unless you cannot get ATM cards with a significant enough capacity (looks like you'll need an OC12 card or a couple of MUX'd OC3s).
So is the plan to terminate the fiber at the 15454's and then you need some LAN connectivity for the base stations? How many ports will you need at a POP? Now I am not at all familiar with the ONS platforms, I don't work with them. Do they have copper or fiber ethernet modules, or will you have to connect them via coax to a cisco ATM card in a switch like a Passport 7/15k?
You may be limited by the type of cards you need. Dont skimp by running 1 6509 if you do have to get them. They're pretty redundant in and of themselves, but chassis failures do happen, and an ISP network needs to be nearly invincible.
I just took a look at Cisco's site, and it looks like 5500s have an OC12 module available. The 4000's seem to have only OC3 modules, so it looks like youll need 5500 series and up if you have to run ATM to the switches.
EDIT: I found info about fiber gig-e modules for the ONS platforms. Looks like 2 of those at each POP would work fine. Just make sure you can link each wimax station to both of the 15454s on site, you do not want to lose a line card and have it take out your wireless network in the area.
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
JTY said:
Well, if hes running 280Mbps on one Wimax station, and he is going to run several, he needs at least an OC12, but its looking more like an OC48 sized ring at this point, unless he is going to ditch the ring idea and go point to point with a central data center.
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Oh no kidding. That is why I keep suggesting they lease the circuits from an already established telco. Still expensive, but not the massive capital outlay this project will require. Still, if they are determined to do it themselves, it will be interesting to see what happens.
Personally, here's what I would do.
Lease bandwidth on somebody elses ring. At the data center end, get an OC-48 port, and then at each POP get OC-12 port. Then just have it all brought back over the data center's OC-48.
Could possibly save even more cash, if a telco can do Metro-Ethernet ports. However, that isn't very popular outside of "well lit" facilities.
Lease bandwidth on somebody elses ring. At the data center end, get an OC-48 port, and then at each POP get OC-12 port. Then just have it all brought back over the data center's OC-48.
Could possibly save even more cash, if a telco can do Metro-Ethernet ports. However, that isn't very popular outside of "well lit" facilities.
So what are the pros. cons. of using either a ring or point to point?
Also 280Mbps isnt much per PoP so is there an eqivalent to link aggregation for WiFi? I.E deploying more than one AP/Radio per PoP?
I am well aware of how extereme we will have to go to get 5 nines, that's why I've chosen apple hardware/software for its reliability and redundancy. We're doubling up on all servers, with IP failover etc. For storage I'll be implimenting RAID arrays, along with two (one initially) asyncronous backup facilities.
Another question, how will address allocation work in a WISP environment? I know I need to apply for a static block for my servers but what IP's do I allocate my connectees? Is it a simple as running a DHCP server on the network and once the client is authenticated with the RADIUS they DHCP gives them an address?
Also 280Mbps isnt much per PoP so is there an eqivalent to link aggregation for WiFi? I.E deploying more than one AP/Radio per PoP?
I am well aware of how extereme we will have to go to get 5 nines, that's why I've chosen apple hardware/software for its reliability and redundancy. We're doubling up on all servers, with IP failover etc. For storage I'll be implimenting RAID arrays, along with two (one initially) asyncronous backup facilities.
Another question, how will address allocation work in a WISP environment? I know I need to apply for a static block for my servers but what IP's do I allocate my connectees? Is it a simple as running a DHCP server on the network and once the client is authenticated with the RADIUS they DHCP gives them an address?
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
Keith130 said:
Sonet Ring
Pros:
They are built in a redundant fashion
Great for Campus or Metro area connections from multiple businesses into a telco
Cons:
Ring bandwidth must be wide enough to aggregate all incoming links
P2P
Pros:
Bandwidth is only as much as the site connecting to the Datacenter needs
One hop from the DC to every POP
Cons:
Not redundant by nature, need 2 circuits on 2 paths (usually 2 telcos) with 2 last miles for full redundancy.
Keith130 said:
You need to test how much you can oversubscribe those wifi units. You should be able to get far more than 140 people @ 2Mbps on them due to oversubscription. If you had to have dedicated bandwidth, telcos and ISPs would of killed themselves.
Keith130 said:
Hrm, I have never seen Apple chosen as the vendor for a company's core server environment, I didn't know they even had the product for that. Usually you see stuff like HP + unix or sun + solaris in that sort of application. But I am a network guy, not a server guy, and my experience there is limited.
For server redundancy in the core, you may want to look into cisco's CSM load balancers. I am a fan. There are other people who make that product of course, I am just more familiar with the CSMs.
So how to I setup a ring topology? Is it as simply as running a pair of fibers to the terminator, then from that terminator, run a pair of fibres to the next one etc?
So if I want complete redundancy I will need two 15454's per PoP, running two seperate rings with a backup for each so thats four rings total, then run from the 15454's into a gigabit switch where the wireless basestations can also connect into. However if the switch goes then that PoP is useless, what can I do about that?
Something else I have been wondering about even though it might not apply to me, on the cisco site I read of a piece of equiptment (can't remember what it was or the model) that supports an OC-768 card. Is it technically possible to have that in the data centre and then only smaller cards throughout the PoP or do fibre rings have to run on the same speed?
So if I want complete redundancy I will need two 15454's per PoP, running two seperate rings with a backup for each so thats four rings total, then run from the 15454's into a gigabit switch where the wireless basestations can also connect into. However if the switch goes then that PoP is useless, what can I do about that?
Something else I have been wondering about even though it might not apply to me, on the cisco site I read of a piece of equiptment (can't remember what it was or the model) that supports an OC-768 card. Is it technically possible to have that in the data centre and then only smaller cards throughout the PoP or do fibre rings have to run on the same speed?
Keith130 said:
If you want to break off smaller circuits from a larger circuit (i.e., T1's from a DS3) then you will need a MUX (Stardax is what I am most familiar with) in order to make this happen.
Basically, for redundancy you need at least two of everything, everywhere in your network, with different access path's, on different power grids, in different buildings, etc.... the list can go on forever it just depends on how diverse you want to get...
Keith130 said:
I really don't feel like typing all of this up and getting into working and protect paths, etc. so here is a link for you to read up on SONET (in the UK where you are located it's SDH).
INTRO TO SONET
In addition to this I suggest heading over to your local or Internet book seller and picking up some books on data (specifically SDH if that is what you are looking at using) technologies. The information you get here is just bits and pieces of what you need and I doubt anyone on this forum is going to sit down and design this architecture for you. not for free anyway...
I am not trying to make you go away, just trying to get you on the right path. This type of thing is not something you can piece together from comments on a forum...
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
At this point you are headed beyond my rather modest amount of telco knowledge. I know there are 2 physical fiber runs in the ring, but since I do not work for a telco I don't know much about the physical setup beyond that. Maybe someone here does work for a large cap telco and can help, but you are really at the point that you need paid assistance.
Darkstar850
[H]ard|Gawd
- Joined
- Feb 18, 2004
- Messages
- 1,307
PHUNBALL said:
Yup, they are one of the other companies I was thinking of, as well as the BigIP product (can't remember the company right now) and radware.
The CSMs will do layer 4-7 as well. Do you like the foundry's (if you work with them)? The CSMs are very nice and powerful, but working with them takes some practice, as they are even more complicated than the normal cisco CLI.
Darkstar850 said:
The foundry's are rather nice. The CLI emulates Cisco in a lot of aspects so it was not hard to get used to (considering that is what I do every day). They also have a web based interface, but you can not configure everything through this interface so I stick with the CLI. We set up a few of the ServerIron GT-E's (4 blade modular switch) with two 16-port copper blades, one 4-port fiber blade, and one supervisor module. So far they have worked great, we are using them to load balance server's providing streaming media (video)...
I beg to differ, whenever I undertake a project (admittadly never one of this magnitude) I dive in the deep end and find out what I need to learn and get on with it. Thats how I operate. I'll take the given cisco certifacations as suggestions.
Thanks for the link, I think i've pretty much got all my current questions answered. I'll post again if I have anymore I think someone here could help me with.
Thanks for the link, I think i've pretty much got all my current questions answered. I'll post again if I have anymore I think someone here could help me with.
Keith130 said:
Same with myself, but even spending all of your time for the next 6 months studying to understand the subtlities of these types of projects still wouldn't completely prepare you for what you're about to try and implement.
Trust me, spending the money on a good engineer (whether vendor or contract) is the best possible solution in this situation.
Treating a 10 million dollar project like the science fair isn't the greatest idea and I'm saying that for your job's sake.
To quote an old techie proverb : "Go for quality and only cry once, when you get the bill."