Network pics thread

Yes. It is. You obviously don't understand.

I do understand that some people keep 30 days of daily backups onto tape using either Incremental or differential backups and then go to monthly and so can snap shot back each day for 30 days. and then hit the monthly if they need it

we snap shot 7 days then a weekly, a monthly and a yearly so I can see that there is a difference there but not enough to justify a tape library at the size we would need. (we are talking hundreds of TB here)

Also have you ever done a big DR from tape? Fucking nightmare.

Saying that DFS and snapshots to redundant SANs in multiple datacenters is not backup is just ridiculous!

If I am wrong here please correct me.
 
not only that but in a large SAN you know when a drive fails, a tape sitting in a storage silo, who knows if it works or not until you actually need it!

sorry but tape really does not make any sense to me... at one point, maybe, but with todays disk sizes and the costs/gig of tape vs spinning disk...

tape is only fast when you stripe tape drives, that adds an entire extra level of complexity and points of failure...

no thanks
 
I posted these before, but today I just made a breakthrough on this subject:


Battery backup


Environmental control/monitoring terminal blocks


Full rack


What I've been working on is an environmental monitoring system. I used an arduino board which is on the shelf above and is connected to the environmental monitoring server (hal9000). I tried Nagios and Pandora among other monitoring solutions, but they all had certain quirks. Mostly, they were difficult to deploy.

I coded my own environmental monitoring app that uses a server-agent approach with centralized configuration. The agents are deployed, the only thing that is configured is the server they connect to, and that's it. At the server all config for each agent is setup such as each monitor. A monitor is simply a Linux command and the output is the value. I can then set minor, major and critical thresholds and can do min/max or just one or the other. Can monitor stuff like memory free, disk space and of course the arduino connected stuff. So my breakthrough today is I finally managed to write an app that can talk to the arduino to get the values. For some reason I had trouble doing this with C++ so I did it with python.

With this custom app I will be able to monitor not only the server related stuff like memory and disk space, but I will be able to monitor physical states such as presence of hydro service and DC battery voltage. Even mouse traps in remote locations... :D



I've also been trying to figure out how to get this to work with smoke detectors. I'd like to have smoke detectors in every room and pretty much every location that has any electronics and if it goes off, it will trigger an alert. For now alerts are emails, but eventually I want to get it to text me as well.

I'll also be adding a feature so when a condition occurs on a monitor, it can run a command. That way certain things like "critical low voltage" can trigger all the servers to shut down. The possibilities are endless.

I'll most likely be releasing this app to public once I deem it satisfactory for release.


Oh and speaking of backup SANs, that SAN you see I use for occasionally backing up my entire environment.
 
And it works! I just got home from my night shift (I coded that serial thing from work lol, vpn ftw) and flicked the power bar off and it totally works! Got the email alerts.



Flicked it back on and got the restore. The high voltage alarms are due to the charger kicking in after. I still have to tweak some of the values and calibrate the voltage with my volt meter as it may be slightly off. I need decent precision as once I set it up properly when it gets to 11.1v or so it will initialize a server shut down. Though I'll have to ensure it only does a shut down if the power is actually out. would suck to have everything shut down because I temporarily disconnected the sensor. :D

I can't wait to setup the mouse traps properly now that I got this working. :D I've been checking em with my ohm meter.
 
not only that but in a large SAN you know when a drive fails, a tape sitting in a storage silo, who knows if it works or not until you actually need it!

sorry but tape really does not make any sense to me... at one point, maybe, but with todays disk sizes and the costs/gig of tape vs spinning disk...

tape is only fast when you stripe tape drives, that adds an entire extra level of complexity and points of failure...

no thanks

<soapbox>

I have never had a tape fail. You must be using a crappy tape drive or something. I have pulled out a 5 year old tape and read it just fine.

As for speed... I have a SAN only used for backing up my server farm. The data from that SAN is then written off to tape Weekly, Monthly, Quarterly, Yearly, etc.

Thinking that snapshots are backups is like thinking that raid is a backup. I hope for the sake of the companies you guys work for there is never a catastrophic failure. Good luck getting to your daily snapshots when the building burns down (happened to me!)

As far as size is concerned, I have a 100TB SAN but a majority of it is not backed up to tape. Only really important stuff like finacials, email DB's, and the like are written to tape.

You can NEVER have too many backups. I have 15min snapshots, replication to a DR SAN, A SAN for only backup software use, and tapes.

</soapbox>

=========================================================\

Now back on topic...

Cleaned up a nasty wiring closet today:

5S5vQl.jpg


UMo6gl.jpg


K9B1ol.jpg


QaSKvl.jpg


And a picture of 2 Cisco UCS blades when we were having the motherboard replaced

Mud3Zl.jpg
 
We have replicated SANs in multiple DC around the country. It would have to be a pretty big fire!
 
We have replicated SANs in multiple DC around the country. It would have to be a pretty big fire!

I've worked for a company that had multiple SANS wiped by what was assumed to be, disgruntled employee. They had to decide between doing forensic investigation on the SAN to see if there was any info that could pin it on a suspect, or attempting recovery of the client data. No offsite tape. They had to recover the data off the array, pretty much eliminating hope on using it for evidence, don't think anyone ever got charged. They could have just restored from ofsite backups.

There's a reason offsite(and offline) backups exist.
 
I've worked for a company that had multiple SANS wiped by what was assumed to be, disgruntled employee. They had to decide between doing forensic investigation on the SAN to see if there was any info that could pin it on a suspect, or attempting recovery of the client data. No offsite tape. They had to recover the data off the array, pretty much eliminating hope on using it for evidence, don't think anyone ever got charged. They could have just restored from ofsite backups.

There's a reason offsite(and offline) backups exist.

That really sucks. That's a good enough reason as any to incorporate cold/offline backups into your disaster recovery plan.
 
I've worked for a company that had multiple SANS wiped by what was assumed to be, disgruntled employee. They had to decide between doing forensic investigation on the SAN to see if there was any info that could pin it on a suspect, or attempting recovery of the client data. No offsite tape. They had to recover the data off the array, pretty much eliminating hope on using it for evidence, don't think anyone ever got charged. They could have just restored from ofsite backups.

There's a reason offsite(and offline) backups exist.

SANs don't have to be live sync'd. Our backup ones do nightly syncs, and do difference snapshots at various schedules. Our different kinds of SANs are also handled by different groups and users. Data Storage Infrastructure handles live stuff, and Data Archiving does backups.

None of the systems are so linked that one user or one event could wipe them all. They only connect to each other to backup data on schedule.

I'm not hating on tape, in fact we also backup to tape and take it away from our DCs for archiving. But I'm unaware of ever needing to go to tape to get something.
 
SANs don't have to be live sync'd. Our backup ones do nightly syncs, and do difference snapshots at various schedules. Our different kinds of SANs are also handled by different groups and users. Data Storage Infrastructure handles live stuff, and Data Archiving does backups.

None of the systems are so linked that one user or one event could wipe them all. They only connect to each other to backup data on schedule.

I'm not hating on tape, in fact we also backup to tape and take it away from our DCs for archiving. But I'm unaware of ever needing to go to tape to get something.

We do basically the same. A couple of production SANs and backup SAN at each site. Production does a full backup to the backup SAN at night, then from the backup SANs to tape. Then the backup SANs replicate to one another. Incremental/snapshots are done intermittently during the day.
 
I've also been trying to figure out how to get this to work with smoke detectors. I'd like to have smoke detectors in every room and pretty much every location that has any electronics and if it goes off, it will trigger an alert. For now alerts are emails, but eventually I want to get it to text me as well.

I'm not sure what options you have looked into but I did some work with cheap 433 Mhz smoke / fire alarms / detectors: http://tickett.wordpress.com/2012/06/27/more-433mhz-rf-hacking/

Essentially you'd need a small, cheap 433 Mhz receiver which easily hooks up to an arduino.
 
I'm not sure what options you have looked into but I did some work with cheap 433 Mhz smoke / fire alarms / detectors: http://tickett.wordpress.com/2012/06/27/more-433mhz-rf-hacking/

Essentially you'd need a small, cheap 433 Mhz receiver which easily hooks up to an arduino.

I'm trying to keep it as simple as possible. Basically any electronic components should fit inside the hole I will make in the wall. I don't want anything at the arduino itself either as it will get messy fast on the rack if I start adding external stuff to the terminal alarm points. I also have limited components to work with, couple transistors, capacitors and resistors and some diodes. I need to go to the dump and look for an old TV or something, probably can get more stuff.

There's a new safety place that opened here recently I might go check to see what they have. There's got to be a smoke detector out there that has alarm point terminals that simply complete the circuit, that would make life easier. I was also looking at piezo detectors but I don't like the idea of using a sensor for a sensor, it adds a layer where something can go wrong.

Today I also calibrated the battery voltage.



Good enough. :p

I get the occasional bump where it's like 1 volt more than it should be though, but maybe the arduino is just faster than the multi meter and the multi meter is not catching it. I set it so alarms will only set off if it's beyond the threshold at least 2 times in a row, so I should be ok for the odd bad reading. It could also be legit and it's the charger sending some pulses. Just recovered from a 5 hour power outage. Almost lost my server, but managed to ride through the outage ok.
 
I've worked for a company that had multiple SANS wiped by what was assumed to be, disgruntled employee. They had to decide between doing forensic investigation on the SAN to see if there was any info that could pin it on a suspect, or attempting recovery of the client data. No offsite tape. They had to recover the data off the array, pretty much eliminating hope on using it for evidence, don't think anyone ever got charged. They could have just restored from ofsite backups.

There's a reason offsite(and offline) backups exist.

That really sucks. That's a good enough reason as any to incorporate cold/offline backups into your disaster recovery plan.

Or a good kick in the ass to adjust their policy and whatever manager let the guy go(if he was fired) w/o making sure to disable his access and inform his co-workers/dept heads that he was not to have access any longer.
 
I'm trying to keep it as simple as possible. Basically any electronic components should fit inside the hole I will make in the wall. I don't want anything at the arduino itself either as it will get messy fast on the rack if I start adding external stuff to the terminal alarm points. I also have limited components to work with, couple transistors, capacitors and resistors and some diodes. I need to go to the dump and look for an old TV or something, probably can get more stuff.

There's a new safety place that opened here recently I might go check to see what they have. There's got to be a smoke detector out there that has alarm point terminals that simply complete the circuit, that would make life easier. I was also looking at piezo detectors but I don't like the idea of using a sensor for a sensor, it adds a layer where something can go wrong.

Today I also calibrated the battery voltage.



Good enough. :p

I get the occasional bump where it's like 1 volt more than it should be though, but maybe the arduino is just faster than the multi meter and the multi meter is not catching it. I set it so alarms will only set off if it's beyond the threshold at least 2 times in a row, so I should be ok for the odd bad reading. It could also be legit and it's the charger sending some pulses. Just recovered from a 5 hour power outage. Almost lost my server, but managed to ride through the outage ok.
You could add another value to show how many in a row have been above/below the limit
Also could have yet another for how long since the last value was above/below the limit, even if it didn't trip the alarm, and how many in the last X-seconds (say 30) are above/below

that way you could see if it was hovering right at the limit and and bouncing off it or w/o tripping the alarm due to only being one, instead of multiple in a row

or even average the last X-seconds so you can see if it was right at it but just below/above so still problematic but not tripping the critical alarm


that way if you consider "+" being a value at or above the limit, "-" being a value below the limit, " * " being a value that tripped some alarm
each value is a second, the count is 30 seconds

eg1. ----+--+----+-+-+------+-+--**
eg2. -+-++-+-+--+-++--++-+-++-++---

In eg1. if you had the trigger set to 2 in a row, you could tell if it was a false positive or if it had been close but not tripping the alarm for a while there indicating a possible problem that requires a tweak of the alarm settings or installation

in eg2 if you had the trigger set to 3 or more in a row, and averaged you could see that it was at or over the limit based on the time (30 seconds) but not triggering the alarm based on count in a row setting for the alarm.

this way, your short messages via email/txt can give you a better idea w/o having to look at the raw data

also consider seconds could be changed with captures, if for example you have 5 seconds between captures, then you have 150 seconds for 30 captures


P.S. forgive me if I sound a slight bit incoherent, I kinda'sorta just woke up :-/

EDIT: afaik they do make AC line voltage smoke detectors with alarm outputs that are suitable for home use , you can also contact one of your local home security installer companies for the details
 
<soapbox>

I have never had a tape fail. You must be using a crappy tape drive or something. I have pulled out a 5 year old tape and read it just fine.

As for speed... I have a SAN only used for backing up my server farm. The data from that SAN is then written off to tape Weekly, Monthly, Quarterly, Yearly, etc.

Thinking that snapshots are backups is like thinking that raid is a backup. I hope for the sake of the companies you guys work for there is never a catastrophic failure. Good luck getting to your daily snapshots when the building burns down (happened to me!)

As far as size is concerned, I have a 100TB SAN but a majority of it is not backed up to tape. Only really important stuff like finacials, email DB's, and the like are written to tape.

You can NEVER have too many backups. I have 15min snapshots, replication to a DR SAN, A SAN for only backup software use, and tapes.

</soapbox>

nice wiring work!

what you can do backup wise is a function of $$$, time, and space.

When I can, I opt for backup on disk, and tape.

if I had to choose, it depends on circumstances. At one job I chose a backup storage box, but we had outside of the building location with fiber to it, climate control, UPS, generator.

and I was doing 3 jobs....keeping track of tape changes was low on the list, and the drive needed to easily do tapes was outside the budget

Tape is easy when you are small or pretty big, the middle market gets messy when you hit budget limits
 
I wonder if a smoke detector that is used on a commercial fire system would work...either 2 wire or 4 wire smokes. Take a look at System Sensor, for one.

Here is a picture I took yesterday when visiting my high school. They are doing a huge renovation and this one data closet had a window in it...so I took a picture. Kind of sad that all the stuff I worked on when I was there is in a landfill somewhere...and that was not 'that' long ago! The MDF remains, but the old IDFs throughout the school, all gone. Computer labs I helped wire, all gone. TV studio...you get the picture.

You will notice that none of the patch panel ports are labeled, and none of the wallplates were labeled either...WTF
C128F592-DFFA-43FE-87E7-B8AB0A28F49E-23981-0000071D0BB461FC_zps8b92a584.jpg
 
They always had Cisco everywhere...but of course that perfectly good stuff is no good in a remodeled building :p
 
They always had Cisco everywhere...but of course that perfectly good stuff is no good in a remodeled building :p

Maybe the switch gear was leased? I get new switches every 3 years; one of the many pluses to leasing.

Also lots of times budgets are use it or loose it so with education discounts being so nuts they may have used their surplus budget on new switches.
 
I'm trying to keep it as simple as possible. Basically any electronic components should fit inside the hole I will make in the wall. I don't want anything at the arduino itself either as it will get messy fast on the rack if I start adding external stuff to the terminal alarm points. I also have limited components to work with, couple transistors, capacitors and resistors and some diodes. I need to go to the dump and look for an old TV or something, probably can get more stuff.

If wired is an option- awesome :)

I'm sure you've detailed somewhere, but with regard to temperature sensors and other bits you have dotted around- do you not have any issues with the length of wire between the sensor(s) and arduino? (I was lead to believe that they rely on using resistance to determine readings- of course with a long wire this will be different to a short wire?)

Additionally... how are you "multiplexing" tons of different signals with a single arduino GPIO set?

Nice setup by the way. I have so much on my to-do list but work is pretty much taking up 100% of my time so I can't see any of it happening for a good few years!
 
If wired is an option- awesome :)

I'm sure you've detailed somewhere, but with regard to temperature sensors and other bits you have dotted around- do you not have any issues with the length of wire between the sensor(s) and arduino? (I was lead to believe that they rely on using resistance to determine readings- of course with a long wire this will be different to a short wire?)

Additionally... how are you "multiplexing" tons of different signals with a single arduino GPIO set?

Nice setup by the way. I have so much on my to-do list but work is pretty much taking up 100% of my time so I can't see any of it happening for a good few years!

The temp sensors I use are actually digital. At the time I did not know anything about arduino and ended up buying a board that is made for them, but turns out they do work with arduino and use up a digital pin (which is awesome, since there are more digital pins than analog). Any additional sensors I add I'll probably use the arduino.

As for multiplexing, no need as the arduino has enough pins for each sensor (most of the stuff I have in mind only needs 1 digital pin). Though with all the ideas that have been popping in my head such as multiple water sensors, mouse trap sensors etc I regret not getting the arduino mega. :p. I can easily add another in the future though. At this point I'm more worried I'll run out of DIN rail space and have to rearrange stuff on my rack again lol.
 
I wonder if a smoke detector that is used on a commercial fire system would work...either 2 wire or 4 wire smokes. Take a look at System Sensor, for one.

Here is a picture I took yesterday when visiting my high school. They are doing a huge renovation and this one data closet had a window in it...so I took a picture. Kind of sad that all the stuff I worked on when I was there is in a landfill somewhere...and that was not 'that' long ago! The MDF remains, but the old IDFs throughout the school, all gone. Computer labs I helped wire, all gone. TV studio...you get the picture.

You will notice that none of the patch panel ports are labeled, and none of the wallplates were labeled either...WTF
*snip*

Not exactly a long term solution but those pieces of paper look like labels?
 
Thought I might as well post my home/uni setup while its (reasonably) tidy... looks better with the door on and lit up green.

IMG_0654.jpg


IMG_0356.jpg


One old(ish) PowerEdge 840 running ESXi and a pair of HP Micro Servers, one of which belongs to my flatmate. WAN comes in on the red ethernet to the layer 2 Netgear, and gets piped to a pfSense VM on the Dell over a VLAN. I really like this way of doing it, no need for extra kit sitting in the cabinet sucking down power :) Wifi taken care of by a WG102. Only G but multiple SSIDs / VLANs are more important. If I want to copy something large to a laptop, I just plug it in...

'Danger of death' sign found out the back of the flat, seemed like a good place for it :D
 
I'm in Bournemouth :) Ty! Just need some more hot to cold cables to tidy the power up a bit, and some more coloured ethernet cables to replace the few grey/white ones to match the colour/function.

I didn't know the rack was APC? Hasn't got any branding on it at all, even on the door. I made the base though, cost me more in materials + wheels than I paid for the cabinet :D But it will take the weight and a lot more.
 
Personal toys at work. Top two 1Us are dual node systems, so 5 servers total and the bottom one has another 12 hot-swap bays on the back that aren't visible. Wiring to be worked on when I have time.

12100027s.jpg
 
It's all running? Small lights don't get photographed very well by cell phones when overhead lighting underexposes everything else.
 
Doesn't having drives front and back cause issues with cooling? I don't think I would like drives on the hot isle.
 
With the amount of air the fans in that thing push, it's nothing to worry about. Quite something considering that entire chassis is cooled by effectively 4 x 80mm fans (two sets of fans for redundancy).
 
Doesn't having drives front and back cause issues with cooling? I don't think I would like drives on the hot isle.

Supermicro cases are awesome. They have awesome airflow and are readily cooled like Blue Fox said with a mere four 80mm fans.
 
With the amount of air the fans in that thing push, it's nothing to worry about. Quite something considering that entire chassis is cooled by effectively 4 x 80mm fans (two sets of fans for redundancy).

I always wondered the same about those cases, how does the air flow work exactly, is the back drives basically part of the exhaust? With enough air flow I can see that the air would still be fairly cool even after going through the 24 drives then some of it goes through the other set.
 
Yes, that's essentially how it works. When such a large volume of air is moved, the temperature difference is negligible. This is why a lot of dual and quad socket motherboards are laid out so that the exhaust from the front socket goes into the rear ones. Should also mention that on top of the 36 drive bays, there are 4 internal ones as well. Supermicro also makes a version of the chassis with 45 hot-swap bays that forgoes a motherboard (there would be 48, but the power supplies and SAS connectors have to go somewhere). I'll post a picture of the inside later today.
 
Back
Top