53rv3r pr0n

FrgMstr

Just Plain Mean
Staff member
2FA
Joined
May 18, 1997
Messages
58,694
A few shots from the cabinet last night. I know how many of you guys get into these. This thread contains a look at the new hardware configuration (although some specs have changed). The boxes specified to be HardOCP Web and HardForum Web will be virtualized into an 8 box farm which will give us approximately 84GHz of CPU and 60GB of memory to divvy up between the virtualized servers. We are looking now at virtualizing all of our database servers as well…which I think will happen. Hopefully this build will take care of us well into the future as well as supply a level of redundancy we have never enjoyed before.

Cliff, our longtime sysadmin, is the guy in the t-shirt. He has been with us pretty much since day one. He is installing an evaluation copy of VMware server for live test this week. We have been testing in a simulated virtual environment for over a year now. Cliff will be rolling out a new version of our Content Management System as well, although this will not be apparent to you guys.

The next shots are the front of our cabinet, which happens to be pretty full at the moment. We are retiring around 10 boxes this coming week. In four years we have migrated from an all-AMD configuration to an all-Intel setup, so we have certainly seen both sides of that argument. And yes, you might also notice that we are moving to using all Dell servers as well; R300s, 1950s, 2950s as well as one older 850 and one 1900. Most of the current server purchases have been made through DellOutlet.com. These are “refurbished” boxes mostly, but they come with the same warranty as a new box, and from what I can tell, most of them are brand new. While it is somewhat tedious to search for exactly what you need, we estimate that we have saved between $15,000 to $20,000 buying the outlet servers. I have used Dell Outlet for years now for family laptops and computers that I wanted support on with certain family members and it has always worked out the same as if we had purchased the items new. This is our first foray into buying servers from Dell Outlet, and while the customer support is sometimes not up to snuff, it is hard to complain with the amount of savings that is passed along.

Lastly is a shot of the back of the cabinet; what a mess. Hopefully in the next couple weeks we will be able to show off some spiffy cabling and organization. We are currently located inside the ViaWest Data Center inside the InfoMart in Dallas, Texas.
 
I'm very glad for you guys at [H] it appears that you guys are enjoying this like it's an early Christmas present (I don't blame you at all).

Hope the setup continues to go smoothly for you guys. :)
 
And here I figured we rolled from an old TRS-80 located in Kyle's basement!

Nice stuff fellas.
 
I remember back in the days [H] had a different hosting service....was it rack space?
 
We are cutting loose tons of cabling ties to move things around and out of the way, and are NOT tidying any of it up since over 10 boxes will be moved out in 10 days and we have to move almost every box in the cab, not to mention replacing two switches and including a load balancer.

If you care to discuss the temp cabling job, go do it in your own thread, not mine. I put these here so that our readers could communicate about the hardware.
 
I'm very glad for you guys at [H] it appears that you guys are enjoying this like it's an early Christmas present (I don't blame you at all).

Hope the setup continues to go smoothly for you guys. :)

It looks like things are going to be fairly smooth so far. Kinda like getting ready for Christmas. Once we get this done hopefully we will never have to drive to the colo again in the next year.

Gonna fold?

Not even close.

I remember back in the days [H] had a different hosting service....was it rack space?

Been with Texas.net, then a company in New Jersey, then a private colo back in Texas, then with Verio, then with The Planet, and now with ViaWest (formerly DataSide).
 
ANy chance you guys will be doing a full writeup of the setup? The most ambitious I have done is home level stuff, with at most, 10-12 connections. This is vastly unknown to me, and I'll read about anything new (yes, I'm a bastard to play Trivial Pursuit against).
 
Thanks for the info Kyle, nice to see what goes on behind the scenes. What type of load balancer are you running / going to be running?


Alteon

Nortel Networks Alteon Web Switches provide front-end processing for server farms, data centers, content delivery networks. The Alteon Web Switch platform simultaneously supports Layer 2/3 and Layer 4-7 switching, and handles hundreds of thousands of sessions per second.
 
ANy chance you guys will be doing a full writeup of the setup? The most ambitious I have done is home level stuff, with at most, 10-12 connections. This is vastly unknown to me, and I'll read about anything new (yes, I'm a bastard to play Trivial Pursuit against).

I think we will likely give an overall/general writeup about it. Looks like it will be something worth showing off when we get done.
 
Hrmm, this might be a little off topic, if so move/mod it where ever. ;)

With the increasing popularity of 'cloud computing' did you consider not laying down the cash for actual equipment and move the entire infrastructure into 'the cloud', either like Amazon's offering or the multitude of other cloud services out in the wild?

I have no idea if it makes true business sense to do so, but the fact that at least some of the hardwork would be done for you would be very nice.
 
I am a system architect/Geospatial engineer right in the middle of putting together our 3rd entperprise as we speak. We use IBM Blades (yay your tax dollars at work baby!) but the idea is the same. Also, if you need stroage, cant say enough good things about Hitachi Data Systems, they have taken care of us very well, both when flush and when we need to go on the cheap.

I am curious which virtualization software you are going to use?

We are tossing around the idea of VMWARE right now, but thats mostly cause of other guys in our organization have used.

Lastly, the thing that made me laugh the most was the Server names, love them, great idea. Our current two are both sports team names of all of favorite teams and our 2nd is an ode to geeks everywhere, Warcraft Lore Names.

Yes we are dorks, but its so much fun building this kind of thing you cannot imagine, good luck on the new hardware and setup.
 
I think it's a risk/trust thing with cloud computing. Some things make sense, others don't, and sometimes even if it does, you just don't trust whoever's running your stuff without major legal protection.
 
I work as a DBA and support extremely large enterprise environments on oracle and mysql. My advice is to be really careful about virtualizating the databases. VMware and such all have whitepapers showing great performance but that just doesn't materialize in the real world. Oracle in particular is horrible on virtual machines and is only certified on oracle's homebuilt platform. Not that you're running oracle, of course.

Mysql is stable enough on VMs (as stable as mysql ever gets, anyway) but its performance is also dramatically impacted, particularly on concurrent I/O.

MySQL is a challenge to scale horizontally and flat-out doesn't scale vertically past a certain point (which is much higher in v5.1). Generally speaking if you're looking to scale your database infrastructure on mysql, the first move is multiple read slaves, which given hardocp's size you've almost certainly already done. Given the relatively small number of writes in a bog-standard website that might scale perfectly well for you guys for the indefinite future. If not, the next step is to segment data into lots of small cheap servers, like blades.

Beyond performance, one major advantage of virtual machines you might be interested in is high availability. I'm sure VMware's guys took you through the demo where they had a VM running on one physical box, pulled the plug, and migrated it to another machine without losing any connections. That's some seriously cool technology, and it's hard to match with standard HA mechanisms. For oracle you can use active/active RAC, but mysql's clustering sucks rocks-- it's a main-memory only solution. Depending on budget, it can be pretty easy to setup active/passive HA clusters for mysql using a SAN and something like redhat heartbeat or veritas clusters. If your budget doesn't cover a SAN (which seems likely) mysql's DRBD technology is pretty neat too. It's basically a software replicated SAN stored on local disks on two separate machines.

Looking forward to seeing more details of your configuration. It's always neat to see what sites I read daily are running on.
 
i dont get what you mean by "virtualizing" the servers, they seem pretty real and tangible from the pictures.

also, i presume the servers are at the Planet ISP server place? I always for some reason imagined the server cabinet in your house or something, but then i figured any operation bigger than tiny probably would have to put their hardware in the place their host is stationed for the bandwith hookup, right? (and in most cases use/rent their hardware, but in this case you just use the space, but provide your own hardware and then just hook it up to the ISP's bandwith plug in their server room).

is that about right?
 
Virtualizing refers to running a bunch of virtual machines on one big server. It allows you to use your hardware more efficiently. For example, say you have a four-way quadcore machine dedicated to forum searches. When nobody's searching, that box is idle. If you also run an appserver on the same hardware you can properly loadbalance the services and use the hardware you bought all the time. It also makes for incredibly easy backups and new hardware provisioning. You just install the vserver software on the host and load up a file and boom, you've got 4 more appservers or whatever.

Almost everybody colocates their servers. It's simply not cost-effective to do your own hosting until you get very large. It's not just the plug in the wall, you also need raised floors, multiple levels of redundancy on everything, physical security, 24/7 management, and so on.

It always pisses me off when clients name their servers with "cool" names. I had one client a few years back that named their servers after fish; flounder, tilapia, salmon, etc. "Crappie" actually crashed a lot. I name my servers proddb01, stgdb01, qadb01, drdb01, prodapp01, prodweb01, and so on. Boring, but not confusing.
 
I work as a DBA and support extremely large enterprise environments on oracle and mysql. My advice is to be really careful about virtualizating the databases. VMware and such all have whitepapers showing great performance but that just doesn't materialize in the real world. Oracle in particular is horrible on virtual machines and is only certified on oracle's homebuilt platform. Not that you're running oracle, of course.

Not to hijack, but we run Oracle 10G currently on our operational enterprise, in the middle of a new install and we wanna go ESX VMWARE, most likely, then run our Oracle box on a 2008 server 64bit OS. Now we are only doing this on the Oracle box for DR purpases, as we wanna move the VM onto the SAN for our Coop site and have that as a bare metal backup. Are you saying that even with all of the resources that Oracle performance dropped or just if we add a few more VMs, which we are not intending to do. I have to play DBA as we cant afford one, which is why I am asking.

Also, the boxes are IBM HS 21, XEON 5450 Quad cores, 8GB ram.

Thanks in advance.
 
Cliff's got a ton of icons on that laptop's desktop :eek:

And I thought I was bad in regards to the number of icons I end up having on the desktop at any time. :D

And why can't I comment on the cabling? I only have seven boxen currently running in my living room and your cabling job looks a hell of a lot nicer than mine. :(

 
Virtualizing refers to running a bunch of virtual machines on one big server. It allows you to use your hardware more efficiently. For example, say you have a four-way quadcore machine dedicated to forum searches. When nobody's searching, that box is idle. If you also run an appserver on the same hardware you can properly loadbalance the services and use the hardware you bought all the time. It also makes for incredibly easy backups and new hardware provisioning. You just install the vserver software on the host and load up a file and boom, you've got 4 more appservers or whatever.

Almost everybody colocates their servers. It's simply not cost-effective to do your own hosting until you get very large. It's not just the plug in the wall, you also need raised floors, multiple levels of redundancy on everything, physical security, 24/7 management, and so on.

It always pisses me off when clients name their servers with "cool" names. I had one client a few years back that named their servers after fish; flounder, tilapia, salmon, etc. "Crappie" actually crashed a lot. I name my servers proddb01, stgdb01, qadb01, drdb01, prodapp01, prodweb01, and so on. Boring, but not confusing.

so all that astuff on the rack then is going to be replaced by one single machine virtualizded into several different servers? (becasue in the pictures i still saw multiple machines).

also, i was just clarifying outloud how serving works, as for the longest time i figured it was just some box in a dudes house hooked up to and hosted on whatever ISP he paid to host it (albeit technically in that case you probably could just you your own intenet, just pay for a domain name).

but anyway, most hosts (like the one hardocp uses) has their own location, bandwith/internet hookup access and in some cases (but maybe not hardocp's?) machines you also lease/rent to store your content on (with tiered packages of hardware deoending on how busy the site gets).

i presume though in hardocp's case, they simply rent the space in the ISP's office and utilize their internet pipe to host it (and i guess presumably through them pay for the domain name/url fee...or is that one time?).
 
It always pisses me off when clients name their servers with "cool" names. I had one client a few years back that named their servers after fish; flounder, tilapia, salmon, etc. "Crappie" actually crashed a lot. I name my servers proddb01, stgdb01, qadb01, drdb01, prodapp01, prodweb01, and so on. Boring, but not confusing.

'Amen' to that! Much easier to wrap your brain around sensible names when that alert hits you at 3am :)
 
Thanks for all the details Kyle.

It it really nice to see the behind-the-scene's action.

It is truly amazing how much hardware it takes to run this site and the number of users on at one time can really tax a system.

I also would like to see more information about server software setup and features when the project is complete!
 
Kyle, just curious if you plan to use vmware server (ie the application loaded on either nix or windows) or ESXi since it's free now since it handles memory and time slices for the cpu better?
 
not sure about virtualizing DB servers.
speaking from experience here with MS SQL on VMware. ICKY performance when under load.
 
Yeap, we pretty much use dell exclusivley also. They produce great server products and the price is more than reasonable. Their new R class is pretty nice also.
 
neeto could you write us an article on the virtualization (how you combine servers) and which software you are using?
 
Hrmm, this might be a little off topic, if so move/mod it where ever. ;)

With the increasing popularity of 'cloud computing' did you consider not laying down the cash for actual equipment and move the entire infrastructure into 'the cloud', either like Amazon's offering or the multitude of other cloud services out in the wild?

I have no idea if it makes true business sense to do so, but the fact that at least some of the hardwork would be done for you would be very nice.

Yes we have considered it. Cost has been considered. Our expense for owning the hardware is nominal compared to what we would pay for "cloud computing" or having others host our needs. The deals we got through Dell Outlet were remarkable, just took some patience and looking. The other side of this coin is having full control over own computing environment. We make changes when we want and are not at the discretion or expense at having a host do it. We also get to run our security exactly how we want to. For a smaller setup, I can see where it would be less expensive.

We have had forum issues for the last year, but we should be out from under those now. On the HardOCP side of things, we have been on the front pages of Digg, Slashdot, and other major news services at the same time and our readers have not seen any degradation in response. Being able to handle that kind of demand through a host is EXPENSIVE. Also, our outbound bandwidth has dropped greatly as Google is now host our ads as we are using their Ad Manager tools. So our expense there has dropped as well.
 
Word of advice: If you plan on keeping the Dell servers more than 4 or 5 years, buy spare hard drives. Dell is really irritating when trying to get replacement drives for older servers...they simply tell you they don;t have any replacements because it's not a current model. (One of the many reasons we use HP instead, HP will still give you parts for old Compaq servers).

We build all of our critical servers will full HD redundancy, and SATA SCSI attached SCSI hard drives. And with this current build I would suggest we have a 200% redundancy rate throughout the system.
 
not sure about virtualizing DB servers.
speaking from experience here with MS SQL on VMware. ICKY performance when under load.

The number one reason we usually see for poor DB performance on virtual servers is poor design of the storage system. Using VMware ESX, a virtual server should see at or near equivilent performance to a physical server of the same size if designed correctly.

There are many other things that can bite you too, like oversubscription of memory, not aligning your VMFS partitions to 64k track boundaries, too few ethernet or fibre-channel ports on the backend, etc.
 
I am a system architect/Geospatial engineer right in the middle of putting together our 3rd entperprise as we speak. We use IBM Blades (yay your tax dollars at work baby!) but the idea is the same. Also, if you need stroage, cant say enough good things about Hitachi Data Systems, they have taken care of us very well, both when flush and when we need to go on the cheap.

I am curious which virtualization software you are going to use?

We are tossing around the idea of VMWARE right now, but thats mostly cause of other guys in our organization have used.

Lastly, the thing that made me laugh the most was the Server names, love them, great idea. Our current two are both sports team names of all of favorite teams and our 2nd is an ode to geeks everywhere, Warcraft Lore Names.

Yes we are dorks, but its so much fun building this kind of thing you cannot imagine, good luck on the new hardware and setup.

VMware's ESXi (Which is FREE now and greatly trims the cost of virtualizing) on the servers and we are still evaluating the different client features. Basically we can over triple our cost for the client if we want a GUI that handles ALL our virtual servers at once instead of logging into the physical boxes separately. This lets you move virtual servers between physical hardware more easily, although it is not hard to do logging into boxes separately. Also, since you have a SAN, there are a lot of features available to you that work in VMware that do not work with only local storage. A SAN will likely be our next investment.
 
You're using the new, free ESXi right? I'm still testing it, but I'm pretty sure I'm going to be rolling it into production.

not sure about virtualizing DB servers.
speaking from experience here with MS SQL on VMware. ICKY performance when under load.

Are you talking about ESX/ESXi (hypervisor type) or regular VMware Server/Workstation?
 
i dont get what you mean by "virtualizing" the servers, they seem pretty real and tangible from the pictures.

also, i presume the servers are at the Planet ISP server place? I always for some reason imagined the server cabinet in your house or something, but then i figured any operation bigger than tiny probably would have to put their hardware in the place their host is stationed for the bandwith hookup, right? (and in most cases use/rent their hardware, but in this case you just use the space, but provide your own hardware and then just hook it up to the ISP's bandwith plug in their server room).

is that about right?

VMware - What is Virtualization? With pictures! :)

I work as a DBA and support extremely large enterprise environments on oracle and mysql. My advice is to be really careful about virtualizating the databases.

Trust me, we will not be jumping into anything on the database side. That said, the HardForum primary database is the only database that is truly under load. We have layers of replication on the HardOCP side of things, most of that hardware there is for piece of mind rather than taking care of load.

Not to hijack, but we run Oracle 10G currently on our operational enterprise, in the middle of a new install and we wanna go ESX VMWARE, most likely, then run our Oracle box on a 2008 server 64bit OS. Now we are only doing this on the Oracle box for DR purpases, as we wanna move the VM onto the SAN for our Coop site and have that as a bare metal backup. Are you saying that even with all of the resources that Oracle performance dropped or just if we add a few more VMs, which we are not intending to do. I have to play DBA as we cant afford one, which is why I am asking.

Also, the boxes are IBM HS 21, XEON 5450 Quad cores, 8GB ram.

Thanks in advance.

It can be done. Whether or not you have the expertise to tune for it, I simply do not know.

so all that astuff on the rack then is going to be replaced by one single machine virtualizded into several different servers? (becasue in the pictures i still saw multiple machines).

also, i was just clarifying outloud how serving works, as for the longest time i figured it was just some box in a dudes house hooked up to and hosted on whatever ISP he paid to host it (albeit technically in that case you probably could just you your own intenet, just pay for a domain name).

but anyway, most hosts (like the one hardocp uses) has their own location, bandwith/internet hookup access and in some cases (but maybe not hardocp's?) machines you also lease/rent to store your content on (with tiered packages of hardware deoending on how busy the site gets).

i presume though in hardocp's case, they simply rent the space in the ISP's office and utilize their internet pipe to host it (and i guess presumably through them pay for the domain name/url fee...or is that one time?).

The only things we don't own are the cabinet, the power, and the bandwidth. That is all we pay for on a monthly basis. We handle all of our own domain name purchases and DNS servers.

Here is the basic physical server setup we had planned to move forward with, BUT then ESXi went for free and that has changed the way we are doing Webservers at the moment. Basically 8 physical servers for web that will be hosting 20+ virtual webservers for HardOCP and HardForum. We can scale these as needed.

HOCP&HF_Infrastructure.gif


Thanks for all the details Kyle.

It it really nice to see the behind-the-scene's action.

It is truly amazing how much hardware it takes to run this site and the number of users on at one time can really tax a system.

I also would like to see more information about server software setup and features when the project is complete!

I will tell you know, that outside of the database servers for HardForum (which we think should be able to handle 5000+ users at a time) we have severely overbuilt the configuration. One thing we have learned over the years...redundancy redundancy redundancy. And I still like having it IN our cabinet and having 100% control over it.

We will keep you guys advised as we move along.

Kyle, just curious if you plan to use vmware server (ie the application loaded on either nix or windows) or ESXi since it's free now since it handles memory and time slices for the cpu better?

ESXi because of cost. All our servers are FreeBSD. While moving our webservers back to 32-bit FreeBSD because of ESXi compatibility (which we are still working on) is not a big deal but we need our HardForum Database servers on 64-bit so they have all they RAM they need. So we are not looking at virtualizing the HardForum primary db server yet.

Yeap, we pretty much use dell exclusivley also. They produce great server products and the price is more than reasonable. Their new R class is pretty nice also.

I just bought 4 R300 boxes. Nice stuff, hard to argue with for web servers.

neeto could you write us an article on the virtualization (how you combine servers) and which software you are using?

Will likely do it once we get finished with our migration.
 
I would love to have a SAN as well, but it's a bit too difficult to justify the cost in my situation. I also don't think VMware's infrastructure is worth it with the amount of servers you have. It always seemed to me that it was primarily designed for managing multiple racks of servers over multiple SANs.

That is likely the case with us.
 
Back
Top