Notes from Vijay: Stats DB Migration in Progress

[Tobit]

February 06, 2010

We've talked about this for some time, but now's the time to start the migration to the new stats db hardware. We are doing it now and everything looks ok so far. We are keeping several safeguards in place in case there is a problem.

IF there is a problem with the stats, please bear with us. There are several links we need to update and it's possible that a link is still pointing to the old db. Also, in case of emergency, we are keeping track of all the new stats from this point in a special place, so even in the worst case scenario, we can just go back to the old db and input all the new stats into it.

So, the stats will be down for a bit and there may be some inconsistencies for a day or so while we get all the links updated. The good news is that we'll have much faster stats soon, which will be great for all of us.

UPDATE 1
The migration is now done and it looks like everything is working. We've tested out the stats pages and done a small manual stats update. All looks good. However, since stats are so important, before diving in and just putting everything back to automatic updates, I wanted to see if donors see any problems. If you do, please report them in our forum (http://foldingforum.org).

UPDATE 2
It looks like everything has migrated well. We have the stats back on normal updates and those updates are going fast (under 10 minutes). With the new hardware, I bet we can make it even faster, but that's for later. We have turned back on certain features we previously turned off (eg CPU counts). We have more ambitious plans for the future, especially ideally getting to the point where the stats are never off line (even during updates), which is now possible with the new hardware.

 

[Elledan]

*rejoices*

Stats are the lifeblood of a folder's motivation

 

[Cypher-]

*rejoices*

Stats are the lifeblood of a folder's motivation

Well some of us at least. Some do it for the actual science...

 

[Elledan]

Well some of us at least. Some do it for the actual science...

You know what I mean

 

[EvilAlchemist]

We have more ambitious plans for the future, especially ideally getting to the point where the stats are never off line (even during updates), which is now possible with the new hardware.

This is very exciting news to hear.
I find it very useful to verify how many processors I am showing on their side.
Adds another level of verificiation to make sure all my systems are returning results.

On the Team Side, give us information to help combat other teams.
Nice to know how many each user has because not every user has nice stat sites

 

[CraftyOtter]

That was a smooth transition for a large scale project such as this. Thanks to the crew at Stanford for keeping our little corner of this folding world running.

Yeah I do it for the science but I also like to see how much I am doing at any one particular time, keeps the motivation factor healthly. The farm doesn't manage itself.

 

[The_Heretic]

Thanks for the info Tobit. Looks like I didn't make it to see this note till it was all done. But none the less we appreciate the info.

*rejoices*

Stats are the lifeblood of a folder's motivation

Well some of us at least. Some do it for the actual science...

WTF, you two are crazy. It's for the Beer and Tamales

 

[Tobit]

WTF, you two are crazy. It's for the Beer and Tamales

I thought it was for the ladies? That's why I wear pink so much these days.

 

[Elledan]

I thought it was for the ladies? That's why I wear pink so much these days.

You mean I could actually get a girlfriend by folding?

*places an order for the farm to out-farm Tiger's farm*

 

[Tobit]

More from Vijay:

We're done with the bulk of our initial update to new hardware. We'll be doing some more work in the future to build up some additional capacity, namely hopefully getting to the point where the stats are never off line. For now I think we're in good shape. The stats are much faster than before, so we've turned back on a lot of the capabilities we previously turned off. Also, stats update are taking about 5 minutes and now are limited not so much by db access than by other issues.

Moreover, we have now set the third party stats to update once an hour (instead of once every 3 hours). It's set to update 10 minutes before the hour, every hour, so checking on the hour should be safe. Note that the pages that are updated are:

http://fah-web.stanford.edu/daily_user_summary.txt.bz2
http://fah-web.stanford.edu/daily_team_summary.txt.bz2

Please do not use scripts to access our main pages (i.e. anything with a cgi-bin in the url). We reserve the right to ban any IP that violates this rule, as it slows down the stats for everyone else.

 

[AgrFan]

You mean I could actually get a girlfriend by folding?