Building a NOC

K

killerasp

Gawd
Joined
Jul 17, 2001
Messages
963
The company i work for is at a point where money is flowing in faster than they can count it. I was joking around with my boss about getting plasmas and building a mini NOC. He took me seriously and we are looking into building a NOC enviroment.

What should a typical NOC consist of? What type of data should be displaying on these plasmas/LCD/TV/projector screen? Traffic throughput? Security stats? What is the most important thing ot have in a NOC? Least important? The list goes on and on.....

I am interested in hearing your opinions and comments. Thanks!
 
Right up my ally :D I work in a NOC right now. Granted I work for a multi-billion dollar company but I give you some advice.

Plasmas - usually we have CNN and weather channel up on a couple of ours, to keep an eye out for any natural disasters that may impact service. We also have something similiar to MRTG (multi router traffic grapher) that monitors the load on some of our key systems.

Netcool - for a alarm monitoring solution Netcool (dont know who makes it) is pretty good. Interfaces with clarify (trouble ticket system) and allow you to assign a trouble ticket to certain alarms. Netcool can do so much more than this, but we do not use that functionality.

Workstations - this is a biggy. You are going to have a hard time watching the network if your workstations are running 24x7x365. We use Sun workstations running solaris, and have ZERO problems. I have been here for over a year, and have yet to have my workstation crash. Workstation uptime is critical. use a Unix/Linux OS avoid windows. If you have apps that require Windows to use, setup a Citrix server and run a Citrix session in linux. It helps with security and removes windows from the equation of problems.

There really is no "most important, least important" designation for a NOC. It all depends on what your company considers mission critical. The software i listed above is not cheap, so if you are looking for something a little cheaper, checkout Nagios which is a free enterprise level Network monitoring software which runs on Linux.

Other than that, I'm not %100 sure. It sounds like the company you are working for doesnt need a full fledged NOC, but maybe a few workstations in/around the area where your techs/engineers are that displays the information mentioned earlier. I don't have specific numbers but NOC's tend to be one of the more expensive departments within a company. Having to staff a group of people 24x7x365, with software/hardware that is almost more important than the customer interfacing equipment means you need som pretty inteligent people who know how to set a NOC up to make sure it all works. which means $$$$$$$$
 
Flagg said:
Right up my ally :D I work in a NOC right now. Granted I work for a multi-billion dollar company but I give you some advice.

Plasmas - usually we have CNN and weather channel up on a couple of ours, to keep an eye out for any natural disasters that may impact service. We also have something similiar to MRTG (multi router traffic grapher) that monitors the load on some of our key systems.

Netcool - for a alarm monitoring solution Netcool (dont know who makes it) is pretty good. Interfaces with clarify (trouble ticket system) and allow you to assign a trouble ticket to certain alarms. Netcool can do so much more than this, but we do not use that functionality.

Workstations - this is a biggy. You are going to have a hard time watching the network if your workstations are running 24x7x365. We use Sun workstations running solaris, and have ZERO problems. I have been here for over a year, and have yet to have my workstation crash. Workstation uptime is critical. use a Unix/Linux OS avoid windows. If you have apps that require Windows to use, setup a Citrix server and run a Citrix session in linux. It helps with security and removes windows from the equation of problems.

There really is no "most important, least important" designation for a NOC. It all depends on what your company considers mission critical. The software i listed above is not cheap, so if you are looking for something a little cheaper, checkout Nagios which is a free enterprise level Network monitoring software which runs on Linux.

Other than that, I'm not %100 sure. It sounds like the company you are working for doesnt need a full fledged NOC, but maybe a few workstations in/around the area where your techs/engineers are that displays the information mentioned earlier. I don't have specific numbers but NOC's tend to be one of the more expensive departments within a company. Having to staff a group of people 24x7x365, with software/hardware that is almost more important than the customer interfacing equipment means you need som pretty inteligent people who know how to set a NOC up to make sure it all works. which means $$$$$$$$
Click to expand...

Thanks for the huge input. I appreciate it. For the last 2 months ive been setting up OpenNMS to monitor out network. So far it has saved our butts more times than i can count. The implmentation is progressing slower than i like..but we are moving foward everyday with that. Sooner or later i hope to have RRD graphs of all our networking equipment programmed into OpenNMS so we can monitor our switches, routers, firewalls, load balancers etc, etc.

But for starters we will be getting
1. a dedicated line to our hosting facility so we dont use the same connection as the rest of the office
2. plasma that will show networking stats and such. possibly some security stats once we get that going
3. maybe some aeron chairs...who knows ;)
 
killerasp said:
Thanks for the huge input. I appreciate it. For the last 2 months ive been setting up OpenNMS to monitor out network. So far it has saved our butts more times than i can count. The implmentation is progressing slower than i like..but we are moving foward everyday with that. Sooner or later i hope to have RRD graphs of all our networking equipment programmed into OpenNMS so we can monitor our switches, routers, firewalls, load balancers etc, etc.

But for starters we will be getting
1. a dedicated line to our hosting facility so we dont use the same connection as the rest of the office
2. plasma that will show networking stats and such. possibly some security stats once we get that going
3. maybe some aeron chairs...who knows ;)
Click to expand...

Plasma's are great (I have one at home), but for displaying static content from applications such as Netcool it's really not the best solution. These displays will most likely be run 24x7x365 and at that rate they will not last long. The main problem you will have is burn-in. In a NOC environment the best solution are rear projection screens. You can have them custom built to any size and all the gear is housed behind the wall. Your maintenance will include replacing bulbs, but the quality of the displayed image should never change...

Have you put any thought into redundancy of power (UPS or diesel/propane generators)??
 
PHUNBALL said:
Plasma's are great (I have one at home), but for displaying static content from applications such as Netcool it's really not the best solution. These displays will most likely be run 24x7x365 and at that rate they will not last long. The main problem you will have is burn-in. In a NOC environment the best solution are rear projection screens. You can have them custom built to any size and all the gear is housed behind the wall. Your maintenance will include replacing bulbs, but the quality of the displayed image should never change...

Have you put any thought into redundancy of power (UPS or diesel/propane generators)??
Click to expand...

Now for those that display stats on a display, how doyou go about configuring what apperas on the screens? Did you make custom web pages that display the data and rotate them every X secs? I am using openNMS right now and plan on implment some sort of open source security IPS. Eventually, i would like to put those stats on some rotating screen.
 
killerasp said:
Now for those that display stats on a display, how doyou go about configuring what apperas on the screens? Did you make custom web pages that display the data and rotate them every X secs? I am using openNMS right now and plan on implment some sort of open source security IPS. Eventually, i would like to put those stats on some rotating screen.
Click to expand...

Well all you really need is a PC with a good graphics card. Also plasmas may not be the best option for the NOC. We actually use projectors for our TV and stats. The plasmas we used to use had horrid burn in from running all the time.

As far as displaying stats, all you really need is MRTG and apache. You can setup MRTG to refresh every x number of seconds. There are also other solution out there that use Java that offer real time graphing without having to refresh the browser.
 
With plasmas, burn in is a huge issue on a 24/7 basis. We had two plasmas, when you changed to a blue screen you could tell which one was running FOX and which one ran CNN.

Also, STAY AWAY from Cube projectors, Sure they're cool in that you can make any size display you want, but they are a royal PITA(I would have to say impossible) to color balance and even if you get them balanced, next week they'll be off again because each bulb ages differently. If the guys at CNN can't even balance their cubes, what hope is there for the rest of us.
 
what about communication gear? Do you guys use regular cell phones? Nextels? Anything special?
 
we have solarwinds installed here at our office. so far everyone seems to like it very much. It is a very pretty interface I can say.

B
 
Dear Forum members,

As the feedback on this topic is from years ago, I would like to understand what would be the best kitlist for setting up a Next Generation NOC.

Different partners mentioned using PRTG and placing 2 x 55"screens on the wall and you are up and running.

Connect into a DC, to keep the number of licenses to a minimum and start monitoring.

Purpose: monitoring multi-vendor: Wifi, LAN and security remotely, troubleshooting, intervention reports and SLA reports.

What is your idea based upon today's knowledge? Kitlist, architecture pictures more than welcome... :(
 
We use SolarWinds in our NOC for device/interface monitoring, flow collection, and config management.
 
For our NOC, we use LabTech RMM (we're an MSP). We have 3x 42" LCD TVs mounted on a wall, 1x runs the SNMP monitor out of LabTech, 1x runs Outlook to show Alert Emails (the 911 stuff goes to Engineer's cell phones), and 1x shows the WAP controller or ticketing metrics out of Connectwise.

Our NOC is pretty small, but runs well. We have 3x racks; 2x server racks and a single 19U communication rack. The room is locked with an RFID key that only 2 people have. In an emergency, there is a physical key. We have 3 internet connections across 2 providers using 3 mediums (fiber, cable, DSL). We do use DNS Failover (DNS Made Easy) to guarantee our customers email and our RMM are always online.

We are looking into a project next year to build a new building with a dedicated 24/7 NOC, with generators, solar panels, etc. The whole sha-bang.
 
I work at a NOC for a telco, here is the stuff we have, it may give you ideas:

Projector screen: Fibre optic transport system
TV: News station in case anything major is going on

Our own PCs:
Our office's DMS100 switch (we monitor many more through other apps)
MTX switch
environmental monitoring app for power outages, battery voltages etc
app for misc monitoring, pretty much everything goes in there from equipment failures to environmental stuff.
CDMA network
app for DMS100/DMS10/DMS1U/RCUs etc...
Microwave transport
App for customer servers/switches as well as our own stuff, DSLAMs etc...

I'm probably missing stuff, not at work now. We have 4 monitors each and they are filled with apps that we monitor. We also have lot of stuff that comes in through email like emergency cable locates.

Oddly we don't have any kind of graphs, but our stuff is mostly analog, not that much IP stuff. There is a Cactus weather map of our IP backbone that one of the network techs setup but it's not something we officially monitor, but we can go check it out if we see other alarms so it's useful for that.
 
wow holy bump from the dead

anywho, remember the #1 priority of a NOC is to look really cool for reporters and investors

things like

logstalgia
https://code.google.com/p/logstalgia/

NORSE
http://map.ipviking.com/

Cyber Threat Real Time Map
http://cybermap.kaspersky.com/

CACTI (preferably weather maps)
http://www.cacti.net/

or The Dude if you prefer windows
http://www.mikrotik.com/thedude

of very little use in reality but look super cool on a 50+" screen and will make moguls go "ooohhh wow, you must be smart!"

no seriously, NOCs are for show... there is nothing you can't just send an email alert for in reality

but NOCs kick ass :)
 
I am glad I started the conversation. firedrow special thanks. Anyone experience with PRTG vs. Solarwinds (price per sensor, implemenation costs, difficulties, Quick wins)? Or the new versions of Nagios? I also would like to keep track of the cyr0n_k0r activities. As soon as we start building our NOC (planning with 5 weeks ;-) ) I will post also some pictures.
 
I sat next to the guy that had to support our Solarwinds systems. Solarwinds support is gawd-awful. I would rank their agents on par with Comcast. At a trade show I approached the Solarwinds booth and talked to them about other avenues for support and they acknowledged this was an area they need to do better. Buyer beware.
 
vr. said:
I sat next to the guy that had to support our Solarwinds systems. Solarwinds support is gawd-awful. I would rank their agents on par with Comcast. At a trade show I approached the Solarwinds booth and talked to them about other avenues for support and they acknowledged this was an area they need to do better. Buyer beware.
Click to expand...
I've had the opposite experience. I've actually had 6 different tickets that I've opened with Solarwinds in the past 2 weeks and all but 1 has been resolved. The 6th one is a display bug that we're trying to determine where it's coming from since an SNMP walk shows us the correct characters, but solarwinds displays a ">" as a ")".

The reason for so many tickets is we're getting through our initial implementation and have lots of questions.
 
You must log in or register to reply here.
Back
Top