• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

MMO servers...

sirusarai

sirusarai

n00b
Joined
Dec 27, 2005
Messages
60
I was wondering if anyone had any kind of insight on what kind of systems and bandwidth you would need run your own MMO on a global scale (i.e. World of Warcraft).

any kind of information would be much appreciated.
 
Why, are you starting one? Steam's status shows 20 gigabits available and 5 used for the content serving machines. As for the machines used, I'd guess 1u dual opteron boxes, lots of them. Lots of ram to keep from killing disks quite so quickly, and big 4p database boxes. Fiber channel storage, redundant stuff on the backend, cheap replaceable stuff on the front end. That's how I'd do it, anyways.

And geographic/load distribution is important - if all your players are physically close together, okay, but otherwise you probably need servers spread over the globe to keep ping times decent. WRT load, what Guild Wars does is create new regions based on load. So every night when the farmers show up in ToA, you end up with 15 US instances of that, compared to 1 or 2 during the day.

There's my speculation for the day. :p

 
Yeah its really expensive to get some good ones up. I played on some "private WoW servers and they were slowwww. Even with like 20 people it lagged crazy. You need some really really good equipment and plenty of it if you want decent.
 
Lazy_Moron said:
Yeah its really expensive to get some good ones up. I played on some "private WoW servers and they were slowwww. Even with like 20 people it lagged crazy. You need some really really good equipment and plenty of it if you want decent.
Click to expand...


laggy because of low bandwidth? or not enough server power?
 
Lazy_Moron said:
Yeah its really expensive to get some good ones up. I played on some "private WoW servers and they were slowwww. Even with like 20 people it lagged crazy. You need some really really good equipment and plenty of it if you want decent.
Click to expand...
The 3rd party servers were run on half decent hardware. The reason they lagged so much and were virtually unplayable was because the server sotware (obviously) is not what Blizzard uses, and so it was TOTALLY full of bugs. Even still, most of those servers had caps on the population. Something like 100-300 people total.

Blizzard's 'new hardware' for WoW are some Dual Core Opteron servers.
 
Coming from an Ultima Online background myself, I'm familiar with a few of the UO server emulators. The best server emulator package available (at least IMO) is RunUO. It is designed for use with Windows and the .NET Framework, though some have been able to get it running using Linux (using Mono) and there are native Linux server packages out there as well, though none quite so full featured and easy to use as RunUO. There are people running servers using low end P4s or your regular run of the mill Athlons. Of course, the way the server software is designed, it's not multithreaded so multiple CPUs usually don't help much for server load, but lots of RAM does help since it appears to use arrays to store items, objects, players, creatures, NPCs, etc. The more of each of these, the more memory you need. I have played on a couple servers where the numbers of items, creatures, and NPCs had to be controlled because if left unchecked, lag would result. Disk I/O is probably the least of the worries since it caches almost everything in memory when running. Every so often (with the time period able to be set by the server op) the arrays in memory are serialized to disk in order to maintain a "hard copy" of the server world. Again, depending on the number of players, creatures, items, etc. currently active in the world and, to some extent the server's hardware itself, the serialization action alone could take a mere fraction of a second or up to a minute or more.

But, even though a person could run a top notch UO shard using top of the line server hardware, the main bottleneck appears to be bandwidth. On a regular DSL connection (2.5MBit down, 256KBit up for example), the speed at which you upload to your players would be your main bottleneck... and on a connection like this, using RunUO, you could expect to support only about 20 to 30 players before lag becomes unbearable. I assume other UO server emulator packages would yield the same results because they essentially all communicate the same with the UO client on the player's PC.
 
Bandwidth is a very big concern. I think I estimated WoW using about 16 kbps client side the last time I checked, and I wasn't doing any thing overly complicated, I was actually in the NE starting area during that, so the population was minuscule. Now, take that 16kbps, and multiply it by say, one hundred. that's about 1.5mbps. Now think, how many people are on each realm, I think its a bit more then 100.

I have no idea about CPU usage or Memory though, and with how many items have to be kept track of, along with all the character data, sounds like the database isn't going to be too small. Last I heard Blizzard was using Oracle for the databases.

EDIT: just did a quick check, looks like its using about 0.013mbps, based on ethereals numbers. That includes running around in Org, and using Auctioneer to scan the AH.
 
Xipher said:
Bandwidth is a very big concern. I think I estimated WoW using about 16 kbps client side the last time I checked, and I wasn't doing any thing overly complicated, I was actually in the NE starting area during that, so the population was minuscule. Now, take that 16kbps, and multiply it by say, one hundred. that's about 1.5mbps. Now think, how many people are on each realm, I think its a bit more then 100.

I have no idea about CPU usage or Memory though, and with how many items have to be kept track of, along with all the character data, sounds like the database isn't going to be too small. Last I heard Blizzard was using Oracle for the databases.

EDIT: just did a quick check, looks like its using about 0.013mbps, based on ethereals numbers. That includes running around in Org, and using Auctioneer to scan the AH.
Click to expand...

pop max on a server is 3,000 I believe, so a filled server would use roughly 39megs/sec of bandwidth using your estimation per player. Filled server isn't uncommon for wow either, good old connecting ques.
 
Sounds like hosting from home is a job for somebody with symetric FIOS and a nice rack available.
Joe Fission: YGPM
 
Blizzard has theirs hosted at AT&T datacenters, at least for the ones here in the U.S.
 
Problem isn't bandwidth, it's processing power. Those servers are performing all the calculations for all your NPCs, mobs, etcetera. That's all done serverside.
Basically, imagine playing the game. Three thousand times over. Yeah. That's roughly what it's like.

Yeah, you have reduced load because you're not drawing and such, but it's still a tremendous amount of CPU load. It's usually broken up across multiple systems, but if that's done wrong on the hardware or software side, you get the clusterfuck that is WoW. (Sorry, I've heard way too many complaints about server problems to believe that there isn't something wrong in their backend. ESPECIALLY having seen it compared to CoH/CoV.)

In reality, most MMOs run on babygrids. Babygrids are pretty much the minimum to get the job done. The problem is that a lot of game developers don't have a lot of experience with this kind of coding, which is very different from typical backend coding. Netcode has significantly less impact on overall performance - it's all about your interconnect capability. No single system can realistically handle any MMORPG. Not even the old stuff like UO.

Here's a basic diagram of a babygrid..

server0-0 -+
server0-1 -+
server0-2 -+--- Gigabit Switch <---> NAS Systems
server0-3 -+
server0-4 -+

Typically the servers are "daisy chained" in addition to being connected to the switch. There's usually a front-end system that acts as a pseudo-load balancer, AKA the login server, which is usually one or two systems at most, since login processing is significantly less load in comparison.

Don't know how WoW did their layout exactly, but typically seperate zones are handled by seperate machines, so that if one box goes down, the majority of the world stays up. Of course, we all know that's not exactly how it works, but that's the basic idea.
 
AreEss said:
Problem isn't bandwidth, it's processing power.
Click to expand...
Never said processing power wasn't an issue, but I find it hard to see how bandwidth isn't an issue. Massive games like WoW require a lot of resources in just about every area of computing; Processing, Memory, Bandwidth. Also the programmers need to know how to use all the resources to their fullest potential.
AreEss said:
Don't know how WoW did their layout exactly, but typically separate zones are handled by separate machines, so that if one box goes down, the majority of the world stays up. Of course, we all know that's not exactly how it works, but that's the basic idea.
Click to expand...
Last rumor I heard, each continent (Their are two) use two different clusters, hence why people can be fine on one side of the world, when the other goes to hell and dies. Also instances are on their own as well. Also they have 2 sets for the back end, one for the databases (Oracle, that's a given due to jobs page on blizzard), and another for handling the rest. Another rumor was that multiple realms actually share one DB cluster (at least at release, this was guessed due to 4 realms going down all at once, and blizzard said something about a database crash taking them out)
 
Xipher said:
Never said processing power wasn't an issue, but I find it hard to see how bandwidth isn't an issue. Massive games like WoW require a lot of resources in just about every area of computing; Processing, Memory, Bandwidth. Also the programmers need to know how to use all the resources to their fullest potential.
Click to expand...

Well realistically, your per-machine Internet bandwidth, 100bT is more than enough. If you're pushing that much bandwidth to clients per machine, well, you're totally fucked. :)

Last rumor I heard, each continent (Their are two) use two different clusters, hence why people can be fine on one side of the world, when the other goes to hell and dies. Also instances are on their own as well. Also they have 2 sets for the back end, one for the databases (Oracle, that's a given due to jobs page on blizzard), and another for handling the rest. Another rumor was that multiple realms actually share one DB cluster (at least at release, this was guessed due to 4 realms going down all at once, and blizzard said something about a database crash taking them out)
Click to expand...

It wouldn't surprise me either way. I really don't know enough about WoW's setup to say for sure. I can say that CoH/CoV has seperate database instances per shard at the minimum, and "local" servers for the east/west coast split, plus euro, etc as well. They definitely have dedicated instances per shard though.

I still haven't seen any MMORPG that makes truly intelligent use of the database backend though. But I'm biased. :)
 
AreEss said:
No single system can realistically handle any MMORPG. Not even the old stuff like UO.
Click to expand...
Quite true. UO's system had, IIRC, 9 seperate "region" servers (I think it was 9) for the main map alone. That number may have increased since I last played a few years ago due to all of the new content and new lands. Crossing server boundaries sometimes represented a night and day difference in lag just due to the population imbalance in the different regions.
 
Joe Fission said:
Quite true. UO's system had, IIRC, 9 seperate "region" servers (I think it was 9) for the main map alone. That number may have increased since I last played a few years ago due to all of the new content and new lands. Crossing server boundaries sometimes represented a night and day difference in lag just due to the population imbalance in the different regions.
Click to expand...
hm. I want to say 6 for the main map (and another set per fel/tram), plus another one for T2A. And yes, the lag crossing server lines was horrendous, as was the possible differences in load between them.

I miss UO now :(
 
GLSauron said:
hm. I want to say 6 for the main map (and another set per fel/tram), plus another one for T2A. And yes, the lag crossing server lines was horrendous, as was the possible differences in load between them.

I miss UO now :(
Click to expand...
Ya, you're right... there are 8 servers in total for each facet (6 main map, 1 for dungeons, 1 for T2A)... heh... guess I've been gone from UO for a long time now too. But, I started playing on player run shards and I'll never go back to the real UO ever... it's just that much better.

BTW, sent you a reply to your PM.
 
Xipher said:
Blizzard has theirs hosted at AT&T datacenters, at least for the ones here in the U.S.
Click to expand...

So are they just leasing the servers instead of actually owning them? and are they using some sort of remote access or do you suppose they actually have employees on-site managing things?
 
JAW said:
pop max on a server is 3,000 I believe, so a filled server would use roughly 39megs/sec of bandwidth using your estimation per player. Filled server isn't uncommon for wow either, good old connecting ques.
Click to expand...
They did something since the last patch, because all my queues have dissapeared, and my ping is now ultra low. (20-60ms)
 
i know FFXI is using like 2p Opteron servers with one for each zone in the game
so each world is realy just one BIG server farm and there all hosted in one place in Japan
 
Blizzard is running Oracle RAC (Real Applications Clusters) - which means they have many many computers powering one database - if one computer happens to crash then the load is balanced amongst the remaining computers in the cluster. When all computers are operating they contribute their own CPU and memory to the total for the database WoW runs on.

So it's not uncommon to have like 32 CPU's and 100's of gb of RAM in a RAC setup.

I believe there's a certain number of Servers running off of one Database setup like this as well. Remember when they had to migrate certain servers to new hardware and other servers had to come down as well? This happened last week - It's most likely all of those servers are running off one RAC setup so they had to bring down the database in order to move servers into or out of that cluster.
 
sirusarai said:
So are they just leasing the servers instead of actually owning them? and are they using some sort of remote access or do you suppose they actually have employees on-site managing things?
Click to expand...
They are probably leasing space for servers they own. I'm far from experienced or knowledgeable but I imagine AT&T is providing the location, space, power, power backup, bandwidth. And Blizzard provides servers and remote manages them.

However; send ppl onsite when shit hits the fan, like the hurricane that hit the AT&T facility during beta.
 
Korialstrasza said:
woah, really? where was that and did it crash the game for awhile?
Click to expand...
5-6 days while they dryed out the servers. Located in Virginia IIRC. One of those east coast states north of GA.
 
Menelmarar said:
5-6 days while they dryed out the servers. Located in Virginia IIRC. One of those east coast states north of GA.
Click to expand...

It was VA, I live near where it happened.
 
Really depends on the game. You can't just ask for information on MMO servers in general.

Well, one thing is for certain: You need your own dedicated server.
 
There's also the size of the playerbase. If you don't have alot of players, you don't need alot of server.

I ran a private Ragnarok Online server for a while. The original code we used was horribly inefficient but very stable. We had two servers splitting duties of handing all the zones, and one server for all logins, housekeeping, databases, etc. Two of the servers were recycled dual P3's, one was a brand new P4-2.8 I build specifically for it's purpose. This was hosted at a datacenter with very high bandwidth. I don't know how many users we could have handled, but I think if we had three brand new servers we could have easily handled a few hundred users in addition to having a high mob and NPC count.
 
ok, so lets say i created my own mmorpg with a group of friends, and we wanted to also have it hosted at a dataenter somwhere, does anyone know who could offer me more then just a celeron and 1 gig of memory? any thoughts on that?
 
sirusarai said:
ok, so lets say i created my own mmorpg with a group of friends, and we wanted to also have it hosted at a dataenter somwhere, does anyone know who could offer me more then just a celeron and 1 gig of memory? any thoughts on that?
Click to expand...


You always have the option of colocation - building your own server and shipping it to a datacenter to host. You pay rent for rackspace/bandwidth/power/etc
 
Managed hosting these days doesn't even rate joke. You're gonna have to colo, and do it with someone good. Not yet another fly by night $9.99/mo dumping ground.
I'm talking Switch & Data, ICG, someone along those lines. On the upside, getting in somewhere like ICG or S&D does a lot more than just give you bandwidth. 8 hours of battery and 2 days of generator without refuel is just the start. Depending on location, you may be able to grab yourself a rate-limited ethernet interconnect with several tier 1's, to provide multihoming and massive expansion capability. You also get stupidly paranoid security in some locations, so you don't have to worry about some bastard sticking a screwdriver in your PSU and causing it to burn out.

Downside? Cost. It's not cheap. Last time I looked, it was about $1200-1500/mo for a 42U cab with commercial AC. (Meaning bring your own UPS and/or line filter.) DC ranges anywhere from $5/Amp to $50/10A blocks, and I always recommend -48DC for switch gear and systems where possible. (Cleaner, higher effeciency, better reliability, and you've got anywhere from 8-24hrs of additional uptime from the batteries depending on site and load.)

The other option is finding a decent tier2 or tier3 who's already multihomed that'll rent by the U. It costs more per U, and you don't get nearly as much backup, but it's cheaper per month. The problem is finding a tier2 or tier3 who doesn't lie through their teeth about their bandwidth. ("Yeah, we have THREE DS3s!!" - all at 6Mbit and oversold by 400%.) You also don't get anywhere near the level of resiliency of other datacenters. Usually it's bring your own UPS, which costs you extra to house, and there's no site backup.
 
Slartibartfast said:
Hey, quit talkin' about my girl! :p
Click to expand...
lOl!

Anyways, MMO servers are crazy... It's nothing someone can pull off at home. Maybe a small MMO or with some really genious coding... It would be easy if you trusted your clients though, cause you could give them a lot of the processing... So if it was with friends it wouldn't be a big deal, but a commericiall MMO can't trust their clients so they have to do ALL the processing.

EVE was talking about 70 processors! Imagine what WoW has to have with 7 Million + clients... Yeah I'm asuming a few thousand of those computers is what they need.

They should just all outsource to Google ;)
 
Stormlifter said:
Anyways, MMO servers are crazy... It's nothing someone can pull off at home. Maybe a small MMO or with some really genious coding... It would be easy if you trusted your clients though, cause you could give them a lot of the processing... So if it was with friends it wouldn't be a big deal, but a commericiall MMO can't trust their clients so they have to do ALL the processing.
Click to expand...

Depends on the way it's written, really. Certain ways of doing it that cut down on processor load dramatically, but I don't wanna get into that whole mess because most of it's theory. I don't have time to write an MMO.

EVE was talking about 70 processors! Imagine what WoW has to have with 7 Million + clients... Yeah I'm asuming a few thousand of those computers is what they need.
They should just all outsource to Google ;)
Click to expand...

Again, depends how it's done. But I'd guesstimate WoW at several full cabs, probably Dell or HP 1Us; say 25 duals per cab plus switchgear plus storage. Maybe 2-3 cabs per shard ballpark? But, really, that's just guessing. I don't play WoW, I don't even like WoW. But that's a fair guesstimate for any MMO in general as far as cab configuration.
But why outsource to Google? MMOs don't work like that; Google has far, far too many hardware and software faults that require human intervention, far too often. Letting Google host something like WoW would be like asking your ISP to upgrade your DSL to tin cans and string. And no, you don't notice all these outages, because the software works around it. Doesn't work that way in MMOs; you lose that much equipment, and things are invariably affected.
 
AreEss said:
Managed hosting these days doesn't even rate joke. You're gonna have to colo, and do it with someone good. Not yet another fly by night $9.99/mo dumping ground.
I'm talking Switch & Data, ICG, someone along those lines. On the upside, getting in somewhere like ICG or S&D does a lot more than just give you bandwidth. 8 hours of battery and 2 days of generator without refuel is just the start. Depending on location, you may be able to grab yourself a rate-limited ethernet interconnect with several tier 1's, to provide multihoming and massive expansion capability. You also get stupidly paranoid security in some locations, so you don't have to worry about some bastard sticking a screwdriver in your PSU and causing it to burn out.
Click to expand...


The flash sucks, but http://www.teamnet.net/ <- that is one of the data centers Fast Servers hosts servers at (and its the one that's about a 5 minute drive away from me right now). The other I know of is Hurricane Electric. Fast Servers does the hosting for http://forums.amd.com/ btw. One downside of Team though, is they use Internap for all their bandwidth (well, they also connect to CFU and UNI, but I don't know if much of their traffic ends up going that way), so your pretty much at least 2 hops away from all your customers, at minimum.
They also do Colocation services, but thats not what they specialize in.
 
Xipher said:
The flash sucks, but http://www.teamnet.net/ <- that is one of the data centers Fast Servers hosts servers at (and its the one that's about a 5 minute drive away from me right now). The other I know of is Hurricane Electric.
Click to expand...

Show me a satisfied HE customer and I'll show you someone who doesn't know the first thing about colocation or routing. HE is below bottom of the barrel; I'd take Verislow over them. Which, for me, is really saying something. (I've thrown Verislow circuits out of more POPs than most people have seen POPs in their lives.) You couldn't pay me to take space from HE. Team, no experience, but the way they advertize alone puts me way off.
Might be able to get a sublet out of 60 Hudson or 111 8th Ave cheap if you're lucky.
 
Xipher said:
I don't know, the AMD forums seem to be working fine :confused:
Click to expand...

5 sl-bb22-chi-6-1.sprintlink.net (144.232.8.81) 27.273 ms 26.259 ms 26.540 ms
6 sl-st21-chi-11-0.sprintlink.net (144.232.20.21) 27.754 ms 26.263 ms 26.733 ms
7 144.232.8.98 (144.232.8.98) 26.087 ms 41.041 ms 42.540 ms
8 hurricane-108836-chi-bb1.c.telia.net (213.248.84.46) 26.604 ms 27.179 ms 27.502 ms
9 pos1-0.gsr12012.sjc.he.net (64.62.227.53) 100.520 ms 110.730 ms 102.629 ms
10 pos1-2.gsr12416.fmt.he.net (64.71.128.182) 151.101 ms 220.143 ms 211.155 ms

Like I said; HE is shit. Watch as latency more than triples entering their network, then doubles again.
(EDIT) We won't mention that the 12016 was EOL'd more than five years ago and is very much an obsolete piece of equipment and then some. Putting a 12012 in front of a 12416 is very much cart in front of horse.
 
Code: 
1  router (192.168.0.1)  0.206 ms  0.111 ms  0.101 ms
 2  10.19.16.1 (10.19.16.1)  8.750 ms  11.577 ms  10.236 ms
 3  12-215-8-97.client.mchsi.com (12.215.8.97)  17.112 ms  23.090 ms  13.856 ms
 4  12-215-1-90.client.mchsi.com (12.215.1.90)  30.667 ms  24.819 ms  22.245 ms
 5  gbr1-p50.cgcil.ip.att.net (12.123.193.154)  24.396 ms  76.979 ms  22.378 ms
 6  tbr1-p013502.cgcil.ip.att.net (12.122.11.33)  26.288 ms  33.929 ms  28.987 ms
 7  12.123.4.13 (12.123.4.13)  25.596 ms  22.892 ms  70.058 ms
 8  chi-bb1-pos1-1-1-0.telia.net (213.248.84.69)  30.892 ms  24.587 ms  26.749 ms
 9  hurricane-108836-chi-bb1.c.telia.net (213.248.84.46)  26.214 ms  23.541 ms  24.261 ms
10  pos1-0.gsr12012.sjc.he.net (64.62.227.53)  79.511 ms  81.248 ms  78.791 ms
11  pos1-2.gsr12416.fmt.he.net (64.71.128.182)  79.313 ms  80.048 ms  78.642 ms
12  g1-0.gsr12008.fmt.fastservers.net (64.71.160.46)  80.983 ms  110.634 ms  95.553 ms
13  forums.amd.com (64.62.232.245)  85.713 ms  81.252 ms  110.756 ms
that was the worst of 3 runs. I see the jump in response time as well, at the telia.net peering point, but >200ms here.


BTW, I'm sure you do have more exprience then I do, I actually see what you mean here, and agree that isn't a good thing. I don't have the experience with the equipment though (only ever worked with 2500 and 2600 series, and never in a real environment). Thanks for the insight :)
 
Code: 
 3  ge-1-43-ur02.flint.mi.michigan.comcast.net (68.87.187.157)  388.869 ms   382.478 ms   376.892 ms
 4  te-9-2-ur03.flint.mi.michigan.comcast.net (68.87.190.82)  385.232 ms   381.448 ms   390.806 ms
 5  te-7-3-ar02.pontiac.mi.michigan.comcast.net (68.87.190.86)  386.876 ms   391.589 ms   387.992 ms
 6  pos-6-3-ar01.pontiac.mi.michigan.comcast.net (68.87.191.173)  387.028 ms   390.175 ms   398.436 ms
 7  12.118.104.49 (12.118.104.49)  407.575 ms   403.614 ms   399.669 ms
 8  tbr1-p011801.cgcil.ip.att.net (12.123.6.58)  46.779 ms   43.235 ms   52.280 ms
 9  12.123.4.13 (12.123.4.13)  44.941 ms   39.928 ms   36.042 ms
10  chi-bb1-pos1-1-1-0.telia.net (213.248.84.69)  32.236 ms   38.143 ms   33.859 ms
11  hurricane-108836-chi-bb1.c.telia.net (213.248.84.46)  30.115 ms   72.011 ms   56.295 ms
12  pos1-0.gsr12012.sjc.he.net (64.62.227.53)  112.880 ms   107.407 ms   103.555 ms
13  pos1-2.gsr12416.fmt.he.net (64.71.128.182)  233.455 ms   229.873 ms   225.921 ms
14  g1-0.gsr12008.fmt.fastservers.net (64.71.160.46)  95.770 ms   91.762 ms   109.044 ms
15  forums.amd.com (64.62.232.245)  105.889 ms   109.476 ms   113.324 ms

I hit 'em slow. :(
 
You must log in or register to reply here.
Back
Top