How does storage like Hotmail or Google actually work? You're allowed 7gigs on a startup gmail account, but does that mean that there is a computer sitting somewhere with seven gigs of space with your name on it (shitload of harddrives with lotsa people!. lol)? I dunno. I just have a few ideas I'm throwing around and was wondering. Thanks!
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
General question about webspace
- Thread starter CruisD64
- Start date
Wow. I assumed so but it just seems a little crazy.
sirmonkey1985
[H]ard|DCer of the Month - July 2010
- Joined
- Sep 13, 2008
- Messages
- 22,412
they use servers.. that used shared storage.. so yes.. every account gets 7 gigs of storage.. but that does not mean there is actually 7 gigs per account of true storage.. take into consideration that the typical person on any account even if flooded with spam will only avg about 100mb.. so why bother having 6.9GB of storage dedicated to that one account.. when you can spread it across all the accounts.. in reality they probably only have about 1-2TB of actual email storage dedicated for gmail.. and the counters just there to make it look like they are adding more.. even though they arent..
(how i know this.. when you load gmail on your login page.. take note of the storage counter.. when the page first loads it will say 2.3GB of storage.. then flash over to the counter that overlays the old number)
and no its not ram crayzjayz.. and yes you can use all 7GB of storage.. i have 2 accounts that ive used for backing up data.. you are able to uploaded 10MB per file to your account by just emailing it to your self.. or using one of the many google storage programs that are on the web..
dont forget google owns the largest data center on the west coast..
(how i know this.. when you load gmail on your login page.. take note of the storage counter.. when the page first loads it will say 2.3GB of storage.. then flash over to the counter that overlays the old number)
and no its not ram crayzjayz.. and yes you can use all 7GB of storage.. i have 2 accounts that ive used for backing up data.. you are able to uploaded 10MB per file to your account by just emailing it to your self.. or using one of the many google storage programs that are on the web..
dont forget google owns the largest data center on the west coast..
Why would it be too volatile? The already have all their metadata stored in RAM for quick caching. I'm almost sure that they have all their caches on RAM or SSDs. A couple years ago they wanted their searches to be the fastest of any provider, and thus made the switch. I'll try to find more info on it.
aaronearles
[H]ard|Gawd
- Joined
- Aug 31, 2006
- Messages
- 2,016
Gee, I hope the power doesn't go out... I might lose all my email.
It's not stored in ram... just no.
It's not stored in ram... just no.
1-2 TB of actual email storage dedicated for gmail?
that's ridiculous!
I don't even send/receive attachments very often and my Gmail account is using 1.3 GB of storage right now. Sure, there are a lot of people who easily stay below the 100MB mark, but when you never have to delete an email because of Gmail's "archive" feature, most people I know are at around 500 MB or so with just regular usage.
Heck, let's say in total, the average gmail account uses 50 MB of space... with only 2 TB total for gmail, that would only be 42,000 email accounts. I'm pretty sure, that Google's Gmail is just a tad more popular than that!
Of course, Google most likely doesn't actually have 7GB * millions of gmail accounts = total storage space allocated specifically for gmail use only. But they do have enough so that people can use up to that maximum storage at a typical rate, because as the need increases, they'll be allocating more storage to gmail. It's not like if you keep sending large attachments to reach that mark that it will suddenly hit a maximum and stop you from using that total 7 GB of storage or whatever it's at right now.
Its a virtual ponzi scheme ha ha ha hahaha
If everyone filled their Inbox to the max all at the same time it would fail because that is not how it works.
There is actual storage usage and theoretical, and I am guessing that with all of the GMail account out there, if all of them became full at the same time they would not be able to handle it.
If everyone filled their Inbox to the max all at the same time it would fail because that is not how it works.
There is actual storage usage and theoretical, and I am guessing that with all of the GMail account out there, if all of them became full at the same time they would not be able to handle it.
yeah most likely but that's pretty much impossible. the odds of that happening are pretty much none.
but saying there are only 1-2 TB of total space for Gmail is pretty insane.
but yeah, i can easily see it all being in RAM and off-site backed up to physical drives or whatever.
Google's tech is pretty crazy and in some cases revolutionary in how they set things up, like having custom motherboards built by manufacturers for use in their web server farm. these are what they use for their web servers.
but saying there are only 1-2 TB of total space for Gmail is pretty insane.
but yeah, i can easily see it all being in RAM and off-site backed up to physical drives or whatever.
Google's tech is pretty crazy and in some cases revolutionary in how they set things up, like having custom motherboards built by manufacturers for use in their web server farm. these are what they use for their web servers.
sirmonkey1985
[H]ard|DCer of the Month - July 2010
- Joined
- Sep 13, 2008
- Messages
- 22,412
the only thing they run on ram is the cache data for google searches.. thus making it quicker.. but as for actual data they do not keep that on ram.. thats all backup on disc.. the only thing they really need on ram is the url data which doesnt take up all that much space..
my actual gmail account i use for mail, i have had since gmail started.. ive never deleted an email and im only up to "You are currently using 181 MB (2%) of your 7328 MB."
so id say the typical user is much lower then me.. not to mention the 1000's and 1000's of accounts people have opened and never used..
hell screw it ill just call google and ask em..
my actual gmail account i use for mail, i have had since gmail started.. ive never deleted an email and im only up to "You are currently using 181 MB (2%) of your 7328 MB."
so id say the typical user is much lower then me.. not to mention the 1000's and 1000's of accounts people have opened and never used..
hell screw it ill just call google and ask em..
but I'd imagine that even if the average user has used 5 megs of their 7gigs, the amount of gmail accounts that exist still means that they need a whole lotta fuckin space!
Anyways, I've been meaning to put my CS degree to use since I'm in the IT industry. I wanna get my hands dirty with some code and SaaS seems like a great way to do that and possibly make some money while I'm at it!
Thanks for the replies so far everyone! More info is also accepted
Anyways, I've been meaning to put my CS degree to use since I'm in the IT industry. I wanna get my hands dirty with some code and SaaS seems like a great way to do that and possibly make some money while I'm at it!
Thanks for the replies so far everyone! More info is also accepted
I don't know why can't people believe that it's all on RAM. Read up on the Google File System and it's redundancies. At the very least they have the data copied 3x for onsite, directly connected systems. If you were to include offline/backup storage they probably have 5 or 6 times the data. I can almost assure you that Google is running mostly everything on the surface through redundant arrays of RAM on their own GFS. They also perform some sort of hard copy backup (whether that may be standard hard drive or datacenters of other sorts).
As far as GMail space, if everyone were to suddenly fill up their inboxes, I think Google may slow down a bit, maybe take some GMail servers offline to offset and manage the influx of data, but I assure you, they have the space required to handle each and every account. You're not realizing that 7.3GB per account is a drop in the bucket for Google. Do some quick math. At 7.3GB per account, a 2TB HDD could potentially contain the emails of 274 people. Let's say there are 5 million people with GMail. That means Google would need 18250 HDDs. At 200 bucks a pop (which is probably much more than Google gets for buying them in bulk), that's around 3.7 million dollars worth of hard drives. As I said, a drop in the bucket for Google. Hence the reason I feel they do most of their popular end-user products on RAM, because they can afford it, and for them, it's a reasonable price for it's speed and operating costs.
Google could potentially add 5 million GMail accounts per month and spend less than 50 million dollars for the required space in a year. If their ad department makes 1 billion a year, that's 5% of their income, only from ads. Think about it.
As far as GMail space, if everyone were to suddenly fill up their inboxes, I think Google may slow down a bit, maybe take some GMail servers offline to offset and manage the influx of data, but I assure you, they have the space required to handle each and every account. You're not realizing that 7.3GB per account is a drop in the bucket for Google. Do some quick math. At 7.3GB per account, a 2TB HDD could potentially contain the emails of 274 people. Let's say there are 5 million people with GMail. That means Google would need 18250 HDDs. At 200 bucks a pop (which is probably much more than Google gets for buying them in bulk), that's around 3.7 million dollars worth of hard drives. As I said, a drop in the bucket for Google. Hence the reason I feel they do most of their popular end-user products on RAM, because they can afford it, and for them, it's a reasonable price for it's speed and operating costs.
Google could potentially add 5 million GMail accounts per month and spend less than 50 million dollars for the required space in a year. If their ad department makes 1 billion a year, that's 5% of their income, only from ads. Think about it.
I have implemented large caching systems, and there is no way in hell it's all on RAM. to be performant you have as much RAM as possible in your cache server, but everything will still reside on disk. you cache in such a way that the most frequently accessed files, followed by the smallest files are cached in memcache and larger files followed by less frequently access files are stored in disk cache. Stuff that nobody looks at, or is hella big is stored on the back end on a SAN.
If I'm wrong, I hope at the very least my Google apps account isn't stored on the shipping container that loses power
If I'm wrong, I hope at the very least my Google apps account isn't stored on the shipping container that loses power
Because RAM requires a constant supply of power to retain data. RAM is never used for permanent storage.
In this 2004 article by Techworld they bring up how Google's servers (in a very basic sense) operate.
Also, the GFS is detailed in this paper published by Google, which basically says, the directions to all the data is concurrently stored as metadata in RAM, while the actually data files are spread throughout Google's hard drives based on their own GFS.
It seems as though, as of 2003-2004, Google was still using standard hard disk drives for storing the majority of their data. I would think by now, they would be moving this to SSD, for increased speed and decreased operating costs.
I do realize this, but keep in mind, do the servers actually turn off?
Also, the GFS is detailed in this paper published by Google, which basically says, the directions to all the data is concurrently stored as metadata in RAM, while the actually data files are spread throughout Google's hard drives based on their own GFS.
It seems as though, as of 2003-2004, Google was still using standard hard disk drives for storing the majority of their data. I would think by now, they would be moving this to SSD, for increased speed and decreased operating costs.
I do realize this, but keep in mind, do the servers actually turn off?
They could. Obviously there are power backup systems, but in a worst-case scenario, when the power goes out, everything would be lost. It would also remove the possibility of moving servers to different locations or to a dataserver, or taking one offline for maintenance, etc. There has to be some sort of tangible data storage.I do realize this, but keep in mind, do the servers actually turn off?
Even though this is potentially possible, I doubt the power would ever go off due to redundancies in generation. Google HAS to have it own power generators just for that purpose. As for the second part about data location, they probably have real-time redundancies which take over in case of primary failure.
Redundancy is taken care by their own file system, Google File System. Read the paper that I linked to in my other post. It contains details about GFS.
But you suggested that Google stores their Gmail data entirely in RAM, which would mean that it has no redundancy. For there to be true data protection, there would need to be a hard copy of the data in some other location.
I read the white paper it actually states that all the client data is not stored in ram.
"Neither the client nor the chunkserver caches file data.
Client caches offer little benefit because most applications
stream through huge files or have working sets too large
to be cached. Not having them simplifies the client and
the overall system by eliminating cache coherence issues.
(Clients do cache metadata, however.) Chunkservers need
not cache file data because chunks are stored as local files
and so Linuxs buffer cache already keeps frequently accessed
data in memory."
Only frequently accessed data is stored in ram. Ie your gmail account is likely not in ram most of the time. Basically according to that white paper their use of ram is fairly typical for a computer system. Now their distributed file system is probably one of the most advanced ever created but they are not using Ram as a storage solution. Ram is still used to hide the latency of going to persistent storage.
Nor does it make sense that they would try, Per that white paper they are using off the shelf components. The maximum amount of ram that you can put in a off the shelf system (as system ram or via an pcie ram storage card) is pretty limited compared to the amount of storage you can get via persistent storage. Most intel or amd based systems max out at 256gb of system ram for the big 4-way systems and this is a function of their integrated memory controllers, ie not alot google could do about it. Itanium and Power5 get you more but still your not going to be able to get anywhere close to the amount possible via persistent storage.
Basically they are running a huge number of systems with lots of Ram for frequently and/or recently used files (some of their data) and massive persistent storage that contains all of the data. And their file system allows them to use 10000's of machines as one unified storage solution. It also allows them to add or remove servers with no visible effects to the software running on the file system.
"Neither the client nor the chunkserver caches file data.
Client caches offer little benefit because most applications
stream through huge files or have working sets too large
to be cached. Not having them simplifies the client and
the overall system by eliminating cache coherence issues.
(Clients do cache metadata, however.) Chunkservers need
not cache file data because chunks are stored as local files
and so Linuxs buffer cache already keeps frequently accessed
data in memory."
Only frequently accessed data is stored in ram. Ie your gmail account is likely not in ram most of the time. Basically according to that white paper their use of ram is fairly typical for a computer system. Now their distributed file system is probably one of the most advanced ever created but they are not using Ram as a storage solution. Ram is still used to hide the latency of going to persistent storage.
Nor does it make sense that they would try, Per that white paper they are using off the shelf components. The maximum amount of ram that you can put in a off the shelf system (as system ram or via an pcie ram storage card) is pretty limited compared to the amount of storage you can get via persistent storage. Most intel or amd based systems max out at 256gb of system ram for the big 4-way systems and this is a function of their integrated memory controllers, ie not alot google could do about it. Itanium and Power5 get you more but still your not going to be able to get anywhere close to the amount possible via persistent storage.
Basically they are running a huge number of systems with lots of Ram for frequently and/or recently used files (some of their data) and massive persistent storage that contains all of the data. And their file system allows them to use 10000's of machines as one unified storage solution. It also allows them to add or remove servers with no visible effects to the software running on the file system.
What I understood from the white paper was that the metadata of files were stored on memory, so the information of the file name and location is on RAM. The actual data is on hard drives.
And Zero82z, why couldn't RAM be redundant. If they use RAM as a ramdrive, and then overlaid their GFS, they could just use RAM like any hard drive.
And Zero82z, why couldn't RAM be redundant. If they use RAM as a ramdrive, and then overlaid their GFS, they could just use RAM like any hard drive.
Well, what I mean by that is that it can't back things up independently. It always requires some sort of external power source, so it can't be transferred between systems or removed and put in storage or anything like that. And I don't consider it to be truly redundant if there isn't some sort of hard copy of the data that can exist as a backup regardless of whether or not the datacenter itself is still functional.
Ram can have redundancy. It doens't have to be other ram though. Thats why tape backups are so popular. Cheap and high capacity, but I wouldn't recommend loading a program off of them! ;P
BTW, to me it would be retarded if they stored everyone's email on RAM. The latency due to the internet negates the advantages of high speed memory. That speed simply isn't required when dealing with such small chunks of data such as emails. 2TB of harddrive = $200 -> 2TB of RAM = who the fuck knows but its alot. Even for a giant like Google that seems a bit ridiculous and unnecessary. 10 years from now though, who knows!
BTW, to me it would be retarded if they stored everyone's email on RAM. The latency due to the internet negates the advantages of high speed memory. That speed simply isn't required when dealing with such small chunks of data such as emails. 2TB of harddrive = $200 -> 2TB of RAM = who the fuck knows but its alot. Even for a giant like Google that seems a bit ridiculous and unnecessary. 10 years from now though, who knows!
Last edited:
CaseyBlackburn
Gawd
- Joined
- Dec 10, 2006
- Messages
- 540
Just think about this really quickly. RAM costs way too much to store everything in it, Google would not do that. They do have a lot of money but they still to this day use regular consumer based products that have been slightly modified. RAM would not be used for storage, nor would SSDs. Google goes for the best GB per Dollar ratio like any of us. The speed benefits and etc you would get from RAM or SSDs would be pointless because with their load balancing and consumer internet connection speeds that the bottleneck in speed is the internet connection not the hardware. It'd be ultimately pointless to use anything besides standard harddrives.
TechieSooner
Supreme [H]ardness
- Joined
- Nov 7, 2007
- Messages
- 7,601
I wish we had an emoticon for backpedaling...
It's common "tech" sense they'd never do this. Why in the world would Google keep a 3 year old Word Document I got as an attachment, that I haven't looked at since I originally got it in my inbox 3 years ago, in RAM? That's just stupid, considering the high cost of RAM opposed to Hard Disks.
RAM is more error-prone, more easily damanged, and obviously you've got the whole issue with needing power 24/7 (A chip on a server's MoBo gets fried... Cya Gmail data.)
Do you even read your own quote?
Two things. A) They still store everything on disk.
B) "Frequently Accessed" means their frequently accessed stuff (like the search algorithm, etc) is cached in RAM. Not stored, just cached. Even if they did do it on-demand, it'd always be cached in RAM anyways just because the sheer amount of people using Google.
If Google "truly" ran everything off RAM, you wouldn't have loading times. Just doing various tasks inside your Gmail account can sometimes give you their little loading bar, which is their servers reading data off the nearest server.
Your reasoning didn't at all prove that RAM would be cost effective, you proved that hard drives would be cost effective, and then at the end mentioned RAM again. Do the same math again with RAM prices (18250 X cost of 2 TB of RAM = $huuuuuge)
Google is known for doing things as cheaply as possible by employing extremely bright engineers designing clever algorithms, and using the cheapest hardware possible. Nobody runs a world class financially feasible mail service using RAM as storage.
Also Gmail has over 100 million users, not 5 million. (according to Wikipedia) Try the math again.
Wow, I just got owned.
Anyway, the more and more I read up on Google's server structure and whatnot, the more and more I'm impressed by what they do using "relatively" weak equipment, that when strung together can perform like a workhorse. So it's been beaten to a pulp now, I'll join everyone else in saying that it's HARD DRIVE based and not RAM based.
In response to TechieSooner, haha, yes an emoticon for backpedaling would be very useful now. Almost everything I'm reading, regardless of when written, points to a hard drive based solution. Now let me just backpedal away.....*pedal* *pedal* *pedal*
And the_b_man, I wasn't trying to prove that RAM would be cost effective, just that RAM doesn't cost THAT much more when you're revenue is more than 1000x as much. But hey, it doesn't run on RAM, so the point is moot.
I suppose I can thank everyone for setting me straight on this one. Perhaps my Google fanboy-ness was just being extremely hopeful that everything was on RAM..... but a part of me still wants to believe so
Anyway, the more and more I read up on Google's server structure and whatnot, the more and more I'm impressed by what they do using "relatively" weak equipment, that when strung together can perform like a workhorse. So it's been beaten to a pulp now, I'll join everyone else in saying that it's HARD DRIVE based and not RAM based.
In response to TechieSooner, haha, yes an emoticon for backpedaling would be very useful now. Almost everything I'm reading, regardless of when written, points to a hard drive based solution. Now let me just backpedal away.....*pedal* *pedal* *pedal*
And the_b_man, I wasn't trying to prove that RAM would be cost effective, just that RAM doesn't cost THAT much more when you're revenue is more than 1000x as much. But hey, it doesn't run on RAM, so the point is moot.
I suppose I can thank everyone for setting me straight on this one. Perhaps my Google fanboy-ness was just being extremely hopeful that everything was on RAM..... but a part of me still wants to believe so