Official YURT Design Thread (Development Discussion ONLY)

[mikeblas]

awe come on.... that's be interesting!!

Only if it was weighted. You know, so Z's count more.

 

[OSUguy98]

exactly... vows and number should count more also


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

Vowels suck.

 

[mikeblas]

I have updated the TODO list with my progress.

I'd like to refresh the beta on Sunday with the fixes I'm making; I'll have to reset the database, since the bug that Unhappy found badly skews the results (by adding a 60000-second-something frmae time, which offsets all the other correct frame times).

 

[MN Scout]

Thanks. Of course, I hope it goes without saying that we're looking for interesting queries, not just all the queries that could be made. I don't think anyone wants to correlate PPD to number of letters in usernames, for example.

Yes, I'm writing some that best work with the strength and uniqueness of the information that we are collecting. Since we are collecting information like cpu speed and frame times that none of the other stats websites are collecting.

-MN Scout

 

[mikeblas]

Great.

I'm also wondering if PPD is the right metric. Maybe PPD/GHz should be used, or something similar.

 

[MN Scout]

Do you have the ability to let the user sort columns? Or would you rather create another page that has sorted columns? Just in general. I don't know anything about how your pages are created.

Also the ppd/ghz and the ppd for the machine are both important. Yes, the user could figure the one off the other, but if we have the ability to display both then we should. In general the ppd for the machine is more widely used currently. Even though it's machine dependent.

I see on the site that you keep track of the last submitted time for a protein one the user pages, and you also keep track of the total number of submissions for that protein on the "All proteins" page. Do you also keep track of the date of each of those submissions?

I was curious because a neat report would be, for the last week display a count of each protein submitted in that time, sorted my most submitted. Also on that page would be a columns for the average ppd and the ppd/ghz. At the bottom would be the total submissions. Other information at the bottom of the page could be the total submissions for the each of the weeks prior and the top 3 proteins each week and there average ppd or ppd/ghz. There could also be the total point submission for each week in that separate table.

This would allow us to look at trends in what wus most people are getting and and how the ppd or ppd/ghz are changing as stanford assigns different proteins. This could be updated every Sunday or Saturday automatically, instead of being dynamic. Often the team's total output fluctuates, and it would be neat to see if a widely turned in protein is getting low average ppd/ghz or is or just takes a really long time to process.

...I think a layout might be better to explain what would be on the report page. Less confusion.

I think this would be really interesting to see how the proteins people are getting change and how this effects the teams - or all the submitters to YURT - daily point totals.

 

[mikeblas]

Do you have the ability to let the user sort columns? Or would you rather create another page that has sorted columns? Just in general. I don't know anything about how your pages are created.

This was answered earlier in the thread, and appears in my TODO list. ORDER BY for the queries can be changed, but it's a ton of work and I haven't hooked it up yet. It doesn't make much sense to do, then, until the queries are stable.

Do you also keep track of the date of each of those submissions?

This was also discussed earlier in the thread, most recently in one of the responses I wrote to you directly, here.

In case it still isn't clear: for each machine, we record the date the last submission, the total number of submissions, and the total time each submission took working. This is done for both work units and frames.

The reason it's done this way is to save database space; I've only got 200 megs to play with. In five days, we've logged 24590 frame submissions. That's about 150,000 per month, and there's only five people in the beta. If we go 1000 people, we can expect 150,000,000 frame entries per month. Each work unit entry costs about 64 bytes of storage on the server (if they were fully historized), so that's 7.2 gigs per month. It would be about 72 megs per month for work units, since most work units are 100 frames.

1000 people might be low, since there's more than 5000 people just in team 33. Or, it might be high; interest in this project doesn't seem to be very strong.

To sum the total number of work units received, I'd have to do 7.2 gigs of I/O. I don't even know what kind of machine my shared hosting gig is running on, but I would be that it can't do more than 40 megs/second of I/O. So that means the query would take at least 180 seconds to run.

Trust me: I'd be thrilled to implement that. It would be lots of fun at all sorts of different levels, including finding alternate ways to aggregate the data so the storage is less or the queries aren't so expensive. But I don't have the hosting solution.

...I think a layout might be better to explain what would be on the report page. Less confusion.

I don't understand what you mean with this one.

I think this would be really interesting to see how the proteins people are getting change and how this effects the teams - or all the submitters to YURT - daily point totals.

Yes, I guess it would be. But we aren't storing that data. If there's strong interest in it, I guess we might start by storing full work unit data only and see how it turns out. That's mostly a server-side change, which means it can wait until after the client software is completed.

 

[OSUguy98]

interest in this project doesn't seem to be very strong.

That's debatable.... interest in development is one thing, interest in using the program is quite another.... I'd wager that most of the active posters in this forum would use the program, but only a few would want to be involved in the development (be it for lack of programming knowledge, the classic lurkers in the forum, feeling like they might get in the way,etc,etc,etc)....

I'd even get that alot of the Aussie's would jump at this program.... alot of us folders like our stats..... some more than others..... I think you'd be surprised at the usage once this thing goes gold...

EMIII is used by a great number of people in the folding community, it had to start somewhere....



Keep on Folding!! For the [H]orde!!

 

[MN Scout]

Sorry for the slight confusion. I meant that I'd draw a diagram of what I thought the columns should be on the page, and what tables are present.

I don't understand fully all the hosting jargon, if somethign isn't currently possible then it's not currently possible. I was thinking that we'd alleviate some of the data strain by only keeping information for a short period of time and then compressing it and only keep a few totals and averages. And in some cases the querries would only poll that data 1 or 2 times a week total. So they'd be a summary of the week, not something that is polled everytime a user clicks on the page, or runs a search.

 

[MN Scout]

I also agree with OSUguy98 in that we need to start somewhere, and the interest should build, and in the development stage people like to lurk and take a backseat. Even though we'd wish they'd do otherwise - sadly I was one of those for the last 2 weeks, but I don't know windows client programming, so I couldn't do much.

 

[AtomicMoose]

I also agree with OSUguy98 in that we need to start somewhere, and the interest should build, and in the development stage people like to lurk and take a backseat. Even though we'd wish they'd do otherwise - sadly I was one of those for the last 2 weeks, but I don't know windows client programming, so I couldn't do much.

X2. Have you seen how many view this thread has?

 

[MN Scout]

So I read through this whole thread, front to back, looking for some inspiration for querries that mikeblas can implement into the website, I didn't find any. Here are my thoughts on querries and the like.

Some general comments,

• It would be great if the user could sort the lists. On every page of the website. For most columns. I know this will be worked on, but I wanted to say it since it's an integral part to the user having a good experience.

Querries:

• 1. The largest individual contributor querry. Ranks people based upon how many results they have submitted through YURT. This I think is somewhat important in spurring people on to install the program on more of their computers. Yes it could push people to abuse the system, but that potential is there wether this querry gets implemented or not.
• 2. The largest machine contributor. Ranks from highest to lowest the machine that has returned the most results.
• 3. A great thing would be if the "Worked Proteins Page" served as a portal to a few reports. This would page would serve as a portale to a per protein breakdown of how current generations of cpus do ppd wise for each protein. This is like a rework of this page, but with the columns in a slightly different order and with the protein name to as a link to the page for the per protein break down of how the cpus do.
  The breakdown page would be somewhat format like this.
  
  Protein 1234 Total Results Turned In: 152 Average PPD: 100
  .................Total results for each type,PPD/Ghz
  Intel:
  Pentium M..............198...........................130
  Xeon.......................20..............................90
  Pentium 4................65.............................100
  Pentium 3...............60.................................80
  Pentium 2................15................................60
  ... Conroe when it comes out
  AMD
  Athlon 64................300..............................120
  Athlon XP................56................................80
  Athlon......................60................................70
  ....
  Other Manufactures
  
  Then each of these breakdowns could Link to a processor maker indepent page for what the best performing machine running that processor type is doing. Essentially it would be like running a search for "Pentium 4" and returning a page that list the machines that have processed that wu and what the ppd/ghz was for it.
  
  A page like this with just averages for all proteins would also be very nice. I know that Pentium M laptops do better than P4s A64s Xeons, but I'm curious as to how much better. I'm sure a lot of other people are also.
• 4. General best performing list. Cpu which on average has the highest ppd/ghz over all proteins. Linked to a specific machine even better, but not necessary. Would be a sorted list with possible other relating columns on the page. It appears like you keep the completed protein information for each machine, so I think this is possible.
• 5. List that show usernames, and what team they belong to, with a click through to a page showing there machines. Currently there is a machine list, but a user list would be nice. Hopefully 1 day there'll be a lot of users and even more machines. So the user list would be a gatway to that individual users computers.
• 6. Front page that welcomes everyone, currently says "Coming soon" so, a few queries for the front page would be: Totals for #of proteins logged, #users using Yurt, #machines. Possible team breakdown, with again #of proteins logged, #users using Yurt, #machines. Try to get some competiveness going so teams spread the word.

This is a rough writing. Not as detailed as I would like, but I thought it was better to give everyone a preview of what I was thinking, and it would allow mikeblas to say if theses querries are even possible or not and to ask me any questions. If any of the other people reading this thread have other ideas please don't hesitate to voice your opinion and other querries you'd like.

 

[mikeblas]

It would be great if the user could sort the lists. On every page of the website. For most columns. I know this will be worked on, but I wanted to say it since it's an integral part to the user having a good experience.

I thought I had made it clear that I was planning on implementing this later. Far more crucial to having a good experience is assuring that the client software works right.

Querries:

Only one r in "queries", by the way.

1. The largest individual contributor querry. Ranks people based upon how many results they have submitted through YURT. This I think is somewhat important in spurring people on to install the program on more of their computers. Yes it could push people to abuse the system, but that potential is there wether this querry gets implemented or not.

I think you'll find this is already implemented. I don't know why it's overflowing, but I'll look into it later. (LATER: it is fixed.)

2. The largest machine contributor. Ranks from highest to lowest the machine that has returned the most results.

It shouldn't be hard to add this to the list of machines page. But because of the data that's already there, we'd have to remove some stuff to make room. What should we remove? Or are you suggesting another report altogether? Work rate by machine, straight up?

The breakdown page would be somewhat format like this.

I'd like to keep the simple report, since it gives me insight into the math that's going on, as well as one of the core entities in the database. But I'm happy to add something like you're describing if it's agreed that it's a good way to represent the results. Perhaps you could cook up some HTML that shows exactly how you want to rows of the report to fit together.

Note that the groups you're using don't make much sense as far as the problem goes. A Conroe is a Pentium 4, as is a Pentium M. These names are very (very!) arbitrary and therefore quite difficult to use for binning the results.

It's easy to get the processor name, like we're already using. But only newer processors support that query. Older processors go by a combination of the manufacturer, plus the family, model, and stepping. There's some incomplete tables of unknown accuraccy and age, like this one. But I'm afraid I'm in for a long road of answering posts from people who bitch about what their processor is registered as, and fixing up the tables endlessly.

A page like this with just averages for all proteins would also be very nice.

I don't understand the difference between this and the existing proteins page.

4. General best performing list. Cpu which on average has the highest ppd/ghz over all proteins. Linked to a specific machine even better, but not necessary. Would be a sorted list with possible other relating columns on the page. It appears like you keep the completed protein information for each machine, so I think this is possible.

It's already there; see the "Rates by Name" page. If you have ideas for specific links, let me know; but the intention is that you can jump to the list that you'd get if you had searched for that processor name.

5. List that show usernames, and what team they belong to, with a click through to a page showing there machines. Currently there is a machine list, but a user list would be nice. Hopefully 1 day there'll be a lot of users and even more machines. So the user list would be a gatway to that individual users computers.

How is this different than #1?

6. Front page that welcomes everyone, currently says "Coming soon" so, a few queries for the front page would be: Totals for #of proteins logged, #users using Yurt, #machines. Possible team breakdown, with again #of proteins logged, #users using Yurt, #machines. Try to get some competiveness going so teams spread the word.

beta.blaszczak.com is my testing site; it's not specific to yurt. As was explained in the documentation, everything will move to a different site soon. If you have specific suggestions for how the front page to look, I'm all ears.

 

[mikeblas]

I have updated my TODO list with the progress I made tonight.

I will be refreshing the beta later this evening. That means the database will be reset, and anyone using the old beta should be kicked out because I have unregistered their machines.

 

[ICE_9]

Will I need to shutdown the monitor and restart it too? I redid the register thing but when I first started it had uploaded some of the frame data already.

 

[mikeblas]

Will I need to shutdown the monitor and restart it too? I redid the register thing but when I first started it had uploaded some of the frame data already.

If you don't shut it down, how will you replace it with the new version?

 

[ICE_9]

Done and done! It pays to read my email.

 

[OSUguy98]

Just a heads up.... I'm not sure if you're in process of tweaking the html or if it was a mistake which was overlooked.... But there is an extra column in the "submitted Work Units" here .... either than or a missing heading


I took a look through the documentation this morning, debated having one of the guys here at work take a look (since he knows absolutely nothing about folding, or any of the programs related) to see if a noob could understand everything..... So far so good....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

But there is an extra column in the "submitted Work Units" here .... either than or a missing heading

Thanks. I had the protein ID twice. I had the protein ID twice. It's fixed now. It's fixed now.

I took a look through the documentation this morning, debated having one of the guys here at work take a look (since he knows absolutely nothing about folding, or any of the programs related) to see if a noob could understand everything..... So far so good....

Thanks for giving it a look. I don't think it is a goal of that document to teach people about folding.

 

[mikeblas]

I've done some more careful analysis of the numbers. (Estimating database storage size isn't as easy as it looks, particularly when indexes are involved.)

From what I'm seeing in the beta, 15 machines finish about 27000 frames in a week; about 270 work units in a week, then. Let me estimate that we go live with 1000 machines participating, and that we'll end up with 1.8 million frames per week and 18,000 work unit records per week.

If we kept all the records for the frames, this means we'd have about 40 megs of frame data per week; 120 megs per month, and 2080 megs per year.

If we keep all the records for the work units, we'd end up with approximately 1/100th of that storage; 0.4 megs of work unit data per week; 1.2 megs per month, and 21 megs per year.

I'm tempted to add a little fudge factor, particularly to the year-long numbers, as the index growth isn't linear. I'm also costing on the index I have now, across (MachineID, WorkUnit) together. I think the index would get bigger because we'd want to have an independent (WorkUnit, Date) index, or a (MachineID, WorkUnit, Date) index. It depends on what queries we want, but I'm sure that date is going to get involved because Date becomes interesting when historizing the data. (That's why you're historizing it.)

Given the new indexes, the size could double, and that could add about 40% to the numbers.

But given this, I'll alter the stored procedures to track both the work unit sum table (so nothing has to change) and create an populate a work unit history table. And we'll see how it goes; if there's too much growth or queries on the resulting table are too expensive to run, I'll shut it down.

 

[mikeblas]

I'd like to put version 1.0 to bed, soon.

There was only one serious bug in the first beta, and I've fixed that for the second beta. Not a peep out of the beta users after more than twelve hours at it with the second beta.

I don't think there's much left to do on the client side. The Last Big Thing is to convert the console application to a service.

On the server side, we still need to look at more queries. I'll try adding sorting to some of the pages over the next couple of days, but I'd like to have more concrete feedback about the site and the queries. I'm gald MN Scout wrote up his ideas, but we still need some design work and details to be thought out.

I'd like to do a very large beta for a week starting April 7th, then RTW on April 14th, barring any showstoppers. The beta on the 7th should include the console version of the monitor and the new website address. I'll be spending some time getting the reporting for older processors (before family F, that is) in place on the site.

Even after those dates, we can still continue to modifiy and change the website, of course, but I want to push for the content now so that if anything additionally needs to be collected from the clients, it can be added before this version is locked-down.

 

[OSUguy98]

I haven't had a chance to really take a look at the latest beta... it's running on 3 of my home PCs, but I didn't get the email until this morning shortly before work... Everything seemed to be running correctly while I was getting ready for work, and looking at the site, I'd say they're sending in data..... but I haven't played around with the different sizes of log files/etc....


Keep on Folding!! For the [H]orde!!

 

[AtomicMoose]

So, what was the decision on the art? I never got an answer...

 

[mikeblas]

So, what was the decision on the art? I never got an answer...

I don't know why nobody's taking an interest in it. Whenever you have something, let me know and I'll work it into the site.

 

[ICE_9]

I have seen no issue on my end. I still want to delve into the manual but I want to make all changes at once. Don't wanna keep hounding you on issues.

 

[sobe]

I'm running the latest (beta010 I think it was called), and had 0 problems installing, and no problems running since... yesterday? I think thats when I installed it.. idk, im at work

 

[mikeblas]

If you started using it any earlier than, say, 7pm Pacific yesterday, I'd be impressed. And a little scared.

 

[OSUguy98]

So, what was the decision on the art? I never got an answer...

I'm not an artistic person by any stretch of the imagination..... But things we'd might need are:

1. A banner
2. a couple YURT icons
3. a couple screenshots of the results of YURT
4. ??


Keep on Folding!! For the [H]orde!!

 

[MN Scout]

Sorry all, I'm going to be away from the computer this weekend, and because of that I've been really busy these last 2 days trying to get things done that I needed the computer for. Little vacation for the weekend. Hopefully I'll have something to contribute Monday.

 

[mikeblas]

The proteins list page is now sortable.

 

[AtomicMoose]

I'm not an artistic person by any stretch of the imagination..... But things we'd might need are:

1. A banner
2. a couple YURT icons
3. a couple screenshots of the results of YURT
4. ??

Mike has the icons already. They are done.

I can just be creative I guess.

Any size limitations for a banner?

 

[OSUguy98]

gotta love sorting

couple things I noticed.......the Protein ID link in the submitted work units section doesn't work quite right... it lacks the protein ID in the link.....

and on the protein list page, can we add a link to the Protein ID column that would take you to a list of all the PCs that have returned that WU? (I'm pretty sure it would be the same link as the fix I mentioned above.....)


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

couple things I noticed.......the Protein ID link in the submitted work units section doesn't work quite right... it lacks the protein ID in the link.....

Fixed.

and on the protein list page, can we add a link to the Protein ID column that would take you to a list of all the PCs that have returned that WU? (I'm pretty sure it would be the same link as the fix I mentioned above.....)

Fixed.

Mike has the icons already. They are done.

Thanks again, by the way. Do you want a credit in the about box or the docs? What should it say?

I can just be creative I guess.

When I grow frustrated of getting no responses here, that's what I do!

 

[AtomicMoose]

Thanks again, by the way. Do you want a credit in the about box or the docs? What should it say?

Sure. Are you guys using your forum names or your real names?

When I grow frustrated of getting no responses here, that's what I do!

I'll see what I can come up with.

 

[mikeblas]

Is anybody here running the Yurt 0.10 beta bits? And actually looking for bugs? Have you noticed any errors in your logs? Say, records that couldn't be sent?

I have seen no issue on my end. I still want to delve into the manual but I want to make all changes at once. Don't wanna keep hounding you on issues.

It's better to "hound me on issues" than to give no feedback at all for a long period. You're assuming that I'll have time in the release schedule to address all your issues, all at once, at the latest possible time. And that all your issues will be solvable quickly.

That backloads the schedule and can cause it to slip.

 

[unhappy_mage]

No problems. I am running 0.10, and have 3 instances of "20 of 20 frame items sent. response was "SUCCESS"" in my log file. I do read the log it prints every once in a while and look for weird things. It'll keep running over this weekend, so by the time I get back I'll have a good-sized pile to dig through.

 

[OSUguy98]

I've not noticed anything weird in the logs... the file size limit on the log is good... and the way you rename it would help keep track of past problems (if they're found later on)....

One thing I noticed on the Machine Details page of mine...... Protein 1481 has a significantly higher ppd (2-,3-,4-times) than the other proteins listed for that machine..... I'll look at my FAHlog on that PC tonight to see if something weird happened....


Keep on Folding!! For the [H]orde!!

 

[AtomicMoose]

First stab at artwork...

Any aspect of this can be changed. Feed back is welcome and desired.

Let me know. I am going to work on some more Monday.

 

[mikeblas]

It would be neat to have a favorites icon.