Official YURT Design Thread (Development Discussion ONLY)

[OSUguy98]

every time a percentage of the WU is finished, it aslo writes to the unitinfo.txt file...... which looks like:

Current Work Unit
-----------------
Name: p1152_L939_K12M_ext_from638
Download time: February 27 05:01:28
Progress: 43%  [||||______]

That, I believe, is the same of all WUs, no matter what type.... no matter what core.....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

What I'm getting is

[19:29:32] Finished a frame (4)
[19:36:01] Finished a frame (5)

and so on. Will different clients write different text? Is it configurable in each client? Does the service write differently than the console? Does the GUI write differently thant the console or the service?

The code to watch the directory is ready to test; I have to sand off a couple of bugs and keep any eye on it to see if it misses ever changes. If the log file writes aren't flushed by the client immediately, that's what'll happen. This is a pretty fragile interface, so I'm happy to discard this code in exchange for some "official" management notification from the folding at home code.

 

[OSUguy98]

Will different clients write different text? Is it configurable in each client? Does the service write differently than the console? Does the GUI write differently thant the console or the service?

As far as I remember, the console and the graphical both keep log files the same way.... the service is just the console running as a windows service (to autoload, no window/etc)....

the only variables I know of, are the different cores, and how they write to FAHlog.txt..... there are 5 or 6 cores, and each is different.... and generally speaking, can be run on Linux/Mac/Windows.....



Keep on Folding!! For the [H]orde!!

 

[unhappy_mage]

I asked on the FCF if there's a FAH-specific way to do this. We'll see what the experts have to say.

Whichever core you're using (5 in production ATM) are responsible for what log output you get. I have perl code for parsing the log files and figuring out things like percent done, time per frame, all that; if you're interested it's available here.

 

[mikeblas]

I've got the file change code working well. The CPUID code is also in good shape. So I'm blocking on more goals and specs; I can't send anything to a website because I don't have any information on communicating with the site. I can't parse anything becasue I don't know what to parse.

So I'm blocked on ya'll.

 

[unhappy_mage]

What to parse: FAHlog.txt. Here's an example of an entire run through an AMBER type protein; do you need examples like this for other proteins, given that any protein will print the line:

[07:18:53] Folding@home Core Shutdown: FINISHED_UNIT

(with a different time, of course) when it finishes?

I've also sent email to enhanced08, so hopefully he can let you know what the other half of the story is.

 

[mikeblas]

What to parse: FAHlog.txt.

Well, of course. My question is: what tokens am I trying to get out of the file? How often? Every change, or only when some event is found when parsing the file? Or is the whole file to be digested and sent to the server every single time it changes? Which fields? Which format?

Some of the information you've said needs to be in the database isn't in the file in the first place -- the score for the block, for example. How is that retrieved from the server? Which server?

 

[OSUguy98]

Seeing as how this changes regularly..... is there any way to download that webpage to the server every day, and add new protiens to a master list of point values/etc/etc? (does that make sense?)


Keep on Folding!! For the [H]orde!!

 

[unhappy_mage]

The value of each protein can be determined from the page OSU linked, or with LPerry's data file which he uses for EM3. It has a header, and then groups of four lines.

"Version"
 4
 6
 6
"2093"
 3888000
 235.0
 100

Version should be obvious; when the file is updated it changes. The "2093" on the first line is the protein number, the 235.0 on the third is the value of it. This pattern repeats throughout the file.

My suggestion would be to have the file change code check to see if FINISHED_UNIT has been reached, then parse the whole thing and send results. We have to wait on e08 to see what fields need to be sent, unfortunately.

 

[mikeblas]

Well, I'm becoming afraid of this is turning into just another simple project that is going to go nowhere because the requirements are not crisp.

Perhaps a reset is in order.

Right now, I have a program that can sniff out the CPUID info (and a few additional data) to describe the hardware where it's running. I also have some code that waits to notice a change in a particular directory. When the directory changes, it figures out which files changed in which ways; it can detect deletes, new files, writes, and length changes.

I can use the detection code to find out when FAHLOG.TXT changes. I'm running the client on one of my machines, and it hasn't yet finished a work unit; it's finished many frames, and I think I'm at 235 of 400 frames -- so maybe it'll be done tomorrow morning. That will give me a single example of the file I'm supposed to parse.

I know that ya'll want something parsed out of FAHLOG.TXT each time it changes. I don't know what that is. I guess I'm to extract a list of data that will lead to the time it is taken to process a work item. Or do we need the time taken to process each frame in a work item? Each one, or an aggregate of them all? And I also don't know what other information from the file is interesting.

If I did, I could start writing the parsing code.

Here's what I'm worried about; for completeness, it includes the issues I alaready raised above.

* No description for what's to be done with the data to eventually be parsed. Sent to a website, I guess; don't know how.

* Don't know what data to pull out of the log file.

* Don't know when to pull it. Every change? Every frame? Every work unit? Every day at 3pm local time?

* Log file format is very unstructured, so the parser is going to break all the time. I don't see any dates in the file, so if it takes longer than 24 hours to do something, I'm not sure I can latch on to the timing. Is there a doc describing the log format? An option to generate a richer or more structured log?

* Might need a high-water mark. My understanding is that the file is cumulative. I have a 25 kilobyte file after running the client for about 24 hours, so someone who's been running the client for a year is going have a file more than nine megs. The parsing code will be fast, but I'm not sure you want to do nine megs of I/O every five or six minutes as frames get finished. If the file is cumulative, then we'll probably want a high-water mark.

* No plan for determining the score of a work unit. The EMPROTZ.DAT file has some interesting information, but since that often updates, it seems better to do it server-side.

* I don't see how to get the protein number from the log file to later find it in EMPROTZ.DAT. Where is it in the script?

* Bootstrapping. Say you've been running the client for a long time and have a huge log file. Do we report all of your data to the server the first time you run the client? Does the client have a conversation with the server about what it has and hasn't seen before from this client? Or do we just throw it at the server, and just sort it out later?

* Interspersed data. If I'm running more than one instance of the client, do those instances write to the same log file? When reading the log file, how do I discern the output from instance #1 apart from instance #2?

* Forging. No plan to avoid unauthenticated claims of work, performance, and so on. That is, no way to prevent ballot stuffing or falsifcation.

How shall we work to resolve these issues?

 

[Mohonri]

* No description for what's to be done with the data to eventually be parsed. Sent to a website, I guess; don't know how.
I know how to fill out a form and submit it using Java (super easy), but not C or C++. I can send you some sample code if you want.

* Don't know what data to pull out of the log file.
I think all you need to worry about is the protein name and the time per frame/step. If there's more than, say, half a dozen line between "Finished a Frame" lines, disregard that frame.

* Don't know when to pull it.
Since the log file is cumulative, I think once a day would be fine. I would just check to see if there's a new WU that hasn't been processed yet. If it's still working on the same WU as last time (and you got a good time/frame average the last time), just exit and don't do anything more.

* Log file format is very unstructured, so the parser is going to break all the time. I don't see any dates in the file, so if it takes longer than 24 hours to do something, I'm not sure I can latch on to the timing. Is there a doc describing the log format? An option to generate a richer or more structured log?
Just worry about the lines you can actually use (Finished a Frame, Protein name), and just ignore anything that doesn't fall in that category.

* <snip> My understanding is that the file is cumulative. <snip>
Actually, once the file reaches a certain size, it gets renamed to some old log, and the fahlog.txt gets wiped and starts over.

* No plan for determining the score of a work unit. The EMPROTZ.DAT file has some interesting information, but since that often updates, it seems better to do it server-side.
Yes, let the server worry about it.

* <snip>Say you've been running the client for a long time and have a huge log file. Do we report all of your data to the server the first time you run the client?
Why not?
Does the client have a conversation with the server about what it has and hasn't seen before from this client?
I think the client should keep track of what it has and hasn't done.

* Interspersed data. If I'm running more than one instance of the client, do those instances write to the same log file?
Nope, separate files

* Forging. No plan to avoid unauthenticated claims of work, performance, and so on. That is, no way to prevent ballot stuffing or falsifcation.
Each datapoint submitted should be associated with a specific user. If that user starts misbehaving, just exclude all datapoints from that user on the server.

See? I can give specifications....

 

[mikeblas]

I have perl code for parsing the log files and figuring out things like percent done, time per frame, all that; if you're interested it's available here.

I can't download this; the MIME type ain't right. I'll try again at home, but maybe it's something on your server.

 

[OSUguy98]

I can't answer all the questions... but I can help with what I know......

* No description for what's to be done with the data to eventually be parsed. Sent to a website, I guess; don't know how.

* Don't know what data to pull out of the log file.

* Don't know when to pull it. Every change? Every frame? Every work unit? Every day at 3pm local time?

I assumed we planned on doing this at the end of every WU...

* Log file format is very unstructured, so the parser is going to break all the time. I don't see any dates in the file, so if it takes longer than 24 hours to do something, I'm not sure I can latch on to the timing. Is there a doc describing the log format? An option to generate a richer or more structured log?

Can you get all the info you need from the unitinfo.txt file?

* Might need a high-water mark. My understanding is that the file is cumulative. I have a 25 kilobyte file after running the client for about 24 hours, so someone who's been running the client for a year is going have a file more than nine megs. The parsing code will be fast, but I'm not sure you want to do nine megs of I/O every five or six minutes as frames get finished. If the file is cumulative, then we'll probably want a high-water mark.

Yes, the file is cumulative... upon restart, the client will rename FAHlog.txt to FAHlog-Prev.txt if the file is over ____kb in size.... but if you have it running for a year with no restart, it's hard to tell how big it would get (I've never done this, the biggest I've seem have been in the 300-400kb range....)

* No plan for determining the score of a work unit. The EMPROTZ.DAT file has some interesting information, but since that often updates, it seems better to do it server-side.

* I don't see how to get the protein number from the log file to later find it in EMPROTZ.DAT. Where is it in the script?

If you take a look back at the Tinker log example I posted, it has a couple lines that say:
"[08:43:46] Protein: p1152_L939_K12M_ext_from638
[08:43:46] - Run: 120 (Clone 65, Gen 3)
[08:43:46] - Frames Completed: 0, Remaining: 400"

I think other cores make the same lines, or similar..... EMIII's .dat file is based on the number after the p, so 1152 in this case.... the other crap is just... well... Stanford's way of getting more info out of the name....

* Bootstrapping. Say you've been running the client for a long time and have a huge log file. Do we report all of your data to the server the first time you run the client? Does the client have a conversation with the server about what it has and hasn't seen before from this client? Or do we just throw it at the server, and just sort it out later?

If we're on the same page............ I assumed this would run at the end of a WU, and send back that WU's info.... not a list of last 2 or 6 or 45 WUs the log file has in it....

* Interspersed data. If I'm running more than one instance of the client, do those instances write to the same log file? When reading the log file, how do I discern the output from instance #1 apart from instance #2?

Users are supposed to use seperate directories for each client.... each client will keep it's own log file...

* Forging. No plan to avoid unauthenticated claims of work, performance, and so on. That is, no way to prevent ballot stuffing or falsifcation.

Not sure how to get around this.... didn't really think about it until you raised that question........ Is there a way to tell the difference between a copy/paste and FAH writing to the file?

Hope that helps a little.....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

See? I can give specifications....

Don't celebrate yet; we're still not done. That's a great help and a big whack at it, tho!


I know how to fill out a form and submit it using Java (super easy), but not C or C++. I can send you some sample code if you want.
It's not that hard from C++. What will make it difficult is not owning the server myself. Testing, then, will be pretty dodgy especially if a non-production version of the system isn't available. (EG, while I'm testing, I'm polluting the publicly vieable data.)

I think all you need to worry about is the protein name and the time per frame/step. If there's more than, say, half a dozen line between "Finished a Frame" lines, disregard that frame.
That's acceptable to me, but this notion doesn't match the current form, or the database I would presume to be under it.

Since the log file is cumulative, I think once a day would be fine. I would just check to see if there's a new WU that hasn't been processed yet. If it's still working on the same WU as last time (and you got a good time/frame average the last time), just exit and don't do anything more.
I'm agreeable to that. How does it get scheduled to run? Will we trust users to set up an AT job for themselves? Or should I write the code to wake up at a random time in the day and go?

Just worry about the lines you can actually use (Finished a Frame, Protein name), and just ignore anything that doesn't fall in that category.
So we're counting frames only and not whole proteins? Does that mean the scoring info is not relevant? Is a frame fractional to the score?

unhappy's sample file doesn't have any "Finished a Frame" lines in it. At all. But he said it was a complete run. Certainly, I must need to find things more than "Finished a Frame".

Actually, once the file reaches a certain size, it gets renamed to some old log, and the fahlog.txt gets wiped and starts over.
With what naming convention? Then, we'll want to find all these and upload their info -- at least, on the first run, right?

Yes, let the server worry about it.
But I still, then, have to find the protein number in the log file because the server will need it to do its lookup. And I don't see that. Or are we just going by name, as you imply (but don't make explicit) a couple of bullets above?

Why not? [...] I think the client should keep track of what it has and hasn't done.
Again, I'm fine with this. But I have the hunch that users will want to submit and know they submitted, and submit again because it makes them feel better if they think their data disn't show up on the site.

Nope, separate files
With what naming convention?

Each datapoint submitted should be associated with a specific user. If that user starts misbehaving, just exclude all datapoints from that user on the server.
Will the user name be authenticated with a password? If so, then how will the password be stored? We can't store it in plain text as a part of the AT job on the command line; that's not secure.

Thanks for the fast answers; this will help make progress. (Until I go racin', anyway.)

 

[OSUguy98]

looks like Mohonri beat me to alot of things.... but I think we provided a couple viewpoints on your questions.....

There are typically 2 log files in the folding Dir..... FAHlog.txt is the current log, and if it gets too big, then it deletes the existing FAHlog-Prev.txt and renames FAHlog.txt to FAHlog-Prev.txt....... Then, creates a new FAHlog.txt....

Each client will have the same nomenclature for the logs... so the only way to tell the difference would be to include the directory.... C:\FAH1\FAHlog.txt..... C:\FAH2\FAHlog.txt..... etc....


Keep on Folding!! For the [H]orde!!

 

[unhappy_mage]

That will give me a single example of the file I'm supposed to parse.

I posted one above; here's a link. If you want more examples, from other core types for example, let us know.

I know that ya'll want something parsed out of FAHLOG.TXT each time it changes. I don't know what that is. I guess I'm to extract a list of data that will lead to the time it is taken to process a work item. Or do we need the time taken to process each frame in a work item? Each one, or an aggregate of them all? And I also don't know what other information from the file is interesting.

Per work unit, we need to find:

• work unit number (p1152_L939_K12M_ext_from638 -> 1152; it's always the number immediately after the p)
• cpu usage the client was set to (this is a value in client.cfg, which is laid out like an .ini file. [core] cpuusage=34 means this value should be reported 34)
• hours/day the computer was running while this WU was in progress. This is mind reading; just report 24 unless you've got a better idea.
  
  Per computer, we need to find:
  • the Stanford Machine identifier: HKLM\SOFTWARE\PandeGroup\Folding@home\UserID is a REG_BINARY containing this value. It's bytewise reversed from how the value is displayed by the client:
    

    In log: - User ID: 437ECFDC405F93C0
    in registry:       C0935F40DCCF7E43

    so my suggestion is to flip it, do the equivalent of sprintf(result, "%0X", bytes), and send that result to the server as the machine name.
  • whatever data you can to describe the machine it's running on. This detection will only run once.
  
  * No description for what's to be done with the data to eventually be parsed. Sent to a website, I guess; don't know how.
  My suggestion is an http request. Do something like ask for:
  http://fah-database.com/addmachine?machname=437ECFDC405F93C0&speed=2400&mem=1024&whatever=theother
  http://fah-database.com/addwu?wunumber=1209&timeperframe=1357&this=that
  or, of course, a POST with equivalent information in it.
  
  * Don't know when to pull it. Every change? Every frame? Every work unit? Every day at 3pm local time?
  Once a WU is finished.
  
  * Log file format is very unstructured, so the parser is going to break all the time. I don't see any dates in the file, so if it takes longer than 24 hours to do something, I'm not sure I can latch on to the timing. Is there a doc describing the log format? An option to generate a richer or more structured log?
  You can increase output by running the client with "-verbosity 9", but it doesn't really give any more useful information.
  
  * Might need a high-water mark. My understanding is that the file is cumulative. I have a 25 kilobyte file after running the client for about 24 hours, so someone who's been running the client for a year is going have a file more than nine megs. The parsing code will be fast, but I'm not sure you want to do nine megs of I/O every five or six minutes as frames get finished. If the file is cumulative, then we'll probably want a high-water mark.
  The client takes care of this; once the log file passes 300k it renames it to FAHlog-prev.txt, overwriting that file if it exists. I've been running FAH on a system for a few months and have only 600k of logs for each of two instances.
  
  * No plan for determining the score of a work unit. The EMPROTZ.DAT file has some interesting information, but since that often updates, it seems better to do it server-side.
  Server-side does make more sense.
  
  * Bootstrapping. Say you've been running the client for a long time and have a huge log file. Do we report all of your data to the server the first time you run the client? Does the client have a conversation with the server about what it has and hasn't seen before from this client? Or do we just throw it at the server, and just sort it out later?
  I'd say just forget whatever's in the log and take only new work units that show up.
  
  * Interspersed data. If I'm running more than one instance of the client, do those instances write to the same log file? When reading the log file, how do I discern the output from instance #1 apart from instance #2?
  They get two seperate directories.
  
  * Forging. No plan to avoid unauthenticated claims of work, performance, and so on. That is, no way to prevent ballot stuffing or falsifcation.
  I guess we could create a whole private/public key system, so when the users sign up they get a key, save it, the client signs results with it, the server decrypts it.... but I don't think this is critical enough to worry that much about it.
  
  * unhappy's sample file doesn't have any "Finished a Frame" lines in it. At all. But he said it was a complete run. Certainly, I must need to find things more than "Finished a Frame".
  It's from an Amber. Believe it or not, the different cores generate different logs (!). They're written by different people, and they didn't standardize it.
  
  Thanks for making your needs explicit, and thanks again for putting the work into this.
  
  

 

[mikeblas]

Hrm. So, now I'm thinking of a different approach. Do most people run the console, or the service? Is there a screensaver version?

The answer that Unhappy got back was that you could run the console with -oneonly. So I could write a program that would:

Label:
  HANDLE h = CreateProcess(FoldingConsole.exe -oneonly);
  Start timer
  WaitForSingleObject(h);
  Stop Timer
  Parse file for last work item type/name only
  Submit data
  goto Label

but it seems that's only useful for the console version. Do the other operating modes write a log? If so, then I'm probably better off just spying on the log. The problem with the log is that I have to wake up more often and read the file; I can see when it changes, not when it now contains the "finished a block" text. This isn't a big deal -- it's just not an air-tight and all-right solution. It's not nearly as pretty as the above.

 

[unhappy_mage]

I do like that approach, with one caveat - would the process that does the start-wait-process loop be able to run as a service? The service mode on the console is terribly convenient, and if this isn't going to run independent of FAH it'd be best to keep all the original functionality around, even if it's implemented in a different way.

I think most people use the console running as a service. The graphical and screensaver versions take too much CPU to draw the pretty pictures, so they get avoided.

 

[mikeblas]

I do like that approach, with one caveat - would the process that does the start-wait-process loop be able to run as a service? ]

Yeah, sure. The only problem is that I'd have to write something to configure the service. Then again, I guess even the sniffing version of the application needs to know enough configuration in order to work that it'll want some sort of configuration, too.

 

[mikeblas]

Well, duh. My spawn-and-time idea doesn't have much merit because the process might be shut down and then restarted. (If the user does it explicitly, or if there's a reboot, or ...) Of course, this is a problem even for sniffing the log file changes. The paring code will need to notice that the client was shut down between the time the work unit started and the time it finished, and disregard that work unit?

I've got code that reads the whole file scanning for "finished". At each update on a Athlon 2100+ system I have here at work, it takes about 1 mS to read my active 1200 line file in a debug build. So I'm figurin' that is fast enough.

If someone has a very large 300K-limit log file I can use to test, that would be helpful.

whatever data you can to describe the machine it's running on. This detection will only run once.

Here's what I have already:

• Processor count
• For each processor, all the CPUID output, which can be parsed, filtered, and digested on the website.
• Machine network name
• Operating System
• Operating system version
• Physical memory size
• Stanford Machine identifier
• Processor speed from the registry
• Processor speed from timing loop
  

This is far to much data for a GET, so we'll have to POST it.

I'd say just forget whatever's in the log and take only new work units that show up.

You and Mohonri aren't in agreement about this. Can ya'll decide and let me know where you land? I don't think OSUGuy98 expressed an opinion about it.

Thanks for making your needs explicit, and thanks again for putting the work into this.

No problem; it's an amusing little project. I'm glad ya'll could provide good answers rapidly so I could keep flowin'.

I doubt I'll do much else until after the enduro, so I'm probably dark until Monday.

 

[OSUguy98]

I've got a 342K log file here at work I can email you.... just send me a PM with an email address....

--------------------------------------------------------

I'd say just forget whatever's in the log and take only new work units that show up.

You and Mohonri aren't in agreement about this. Can ya'll decide and let me know where you land? I don't think OSUGuy98 expressed an opinion about it.

clarifying it in my mind....... are you asking whether the client should grab all the WUs in the log file each time? or only worry about the last one?


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

I've got a 342K log file here at work I can email you.... just send me a PM with an email address....

YGPM.

clarifying it in my mind....... are you asking whether the client should grab all the WUs in the log file each time? or only worry about the last one?

We're saying the client contacts the server only when there's a new workunit. When the client program is run the first time, there might be finished workunits lying around, but that the server hasn't seen them yet. Should these be parsed and sent, for instant gratification, or should the client wait until it watches the first unit finish.

See? So, I've finished a workunit and it is in my log, and I'm half done with the current one. I hear about this program, and install it. Should the program report that finished work unit, or wait until the half-done one completes?

 

[mikeblas]

What's this effort called? What's a good name for the program? HardMonitor.EXE, say?

 

[mikeblas]

Thanks for the file, OSUGuy98. A release build processes 9800 lines in 7.5 mS, which I guess is good enough. Let's set 10 mS as a goal (for my awesome 2100+ reference machine, and one of these maxed-out 300K files).

 

[OSUguy98]

email sent... to both addresses, just in case......

-----------

Are we making a program that runs 24/7 like EMIII does? or are we doing a "every time the user feels like it" parse of the log?

Can we make a program that grabs everything in the log file, gets the date, protein, run,gen,etc,etc,etc,etc and compares that against the list of protiens and adds the new info?

you could have a date/time that the data was last sent to the server, and grab all the protein info since then and send it to the server

So how about this..... the user opens the program, it reads the log file... grabs all the important info, compares that to the master list of protiens already in the user's DB.... adds any new proteins.... and then monitors 24/7 for more info

Have a setting in that program to return WU info to the server every ___ days at a certain time/etc.... user-settable

maybe? could?


For the name...... I say we do what we always do in the DC forum when it comes to naming things...... leave it up to the users..... make a poll.... let people make suggestions, and then vote.....

------------
One major thing we need to consider, is EARLY_END_UNITs...... happens on unstable OCs, bad RAM, unstable proteins (usually beta proteins from Stanford),etc.....

That, and outlier data....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

Are we making a program that runs 24/7 like EMIII does? or are we doing a "every time the user feels like it" parse of the log?

It's always loaded, but sleeping until it sees the log file change.

Can we make a program that grabs everything in the log file, gets the date, protein, run,gen,etc,etc,etc,etc and compares that against the list of protiens and adds the new info?

You'll need to be more specific: Which list of protiens? New compared to what state?

you could have a date/time that the data was last sent to the server, and grab all the protein info since then and send it to the server

Yep. And that would lead to sending the data that was in the log file before the program was installed.

So how about this..... the user opens the program, it reads the log file... grabs all the important info, compares that to the master list of protiens already in the user's DB.... adds any new proteins.... and then monitors 24/7 for more info

What is a master list of protiens? A list of protiens that have already been sent from this computer, that's persisted by the program I'm writing? So only one record is ever sent for each type of protien?

Have a setting in that program to return WU info to the server every ___ days at a certain time/etc.... user-settable

I don't see the point of this one.

For the name...... I say we do what we always do in the DC forum when it comes to naming things...... leave it up to the users..... make a poll.... let people make suggestions, and then vote.....

Great. Can I trust you'll take care of that and inform me of the results?

One major thing we need to consider, is EARLY_END_UNITs...... happens on unstable OCs, bad RAM, unstable proteins (usually beta proteins from Stanford),etc.....

That, and outlier data....

I'll need details about them, then -- if they're major things we need to consider, it should be a major priority for ya'll to tell me about them. Scope creep and forgotten requirements will make this more like work, and nothing will cause me to lose interest faster.

 

[OSUguy98]

You'll need to be more specific: Which list of protiens? New compared to what state?

The way I saw it, the program would run 24/7, but would only send info to the server once a week... or however often the user would like.... this would mean creating a list on the local computer for each client it was tracking, which contained all of the proteins ever tracked by the program. When a new WU finishes, it would look through this list, find the end and add the data collected....

Lemme see if I can show you my line of thinking.....

1. I open the program for the first time, set a few options (when I want to return data to the server/ where my FAH clients are/etc)

2. after initial setup, the program scans log file, gets info for all of the completed WUs in the log and adds them to a text file.

3. the program continues running, monitoring for completed WUs....

4. WU finishes, program collect info from the log file and adds it to the text file which contains all other completed WU data....

5. repeat 3 and 4 until given "send to server" time

6. program sends completed WU info to our server

continue 5......

Yep. And that would lead to sending the data that was in the log file before the program was installed.

yup.... but with EARLY_ENDs weeded out,etc....

Great. Can I trust you'll take care of that and inform me of the results?

Yup... I'll start a thread once we've finalized what all this thing will do... I guess I want a basic idea of the features/etc so that I can give the potential users an idea of what it'll do... which will help them with names/etc....

I'll need details about them, then -- if they're major things we need to consider, it should be a major priority for ya'll to tell me about them. Scope creep and forgotten requirements will make this more like work, and nothing will cause me to lose interest faster.

I'll look tonight in my other clients' logs and see if I can find an EARLY_END example... there are a handful of errors which the core will post in the log.... EARLY_END_UNIT is one of them.....



Keep on Folding!! For the [H]orde!!

 

[mikeblas]

The way I saw it, the program would run 24/7, but would only send info to the server once a week... or however often the user would like.... this would mean creating a list on the local computer for each client it was tracking, which contained all of the proteins ever tracked by the program.

Nope. We're contacting the server every time a work unit completes. In this case, the log file is the list you're talking about. There's no reason to keep a list around, since we know when the current work unit finishes. Any previously completed work unit is considered sent.

What's the motivation for only connecting once per time duration? It's possible the client hasn't done any work in that time. And since the communication between the server and the client is very small (and no smaller when batched), I don't see a reason to wait.

Yup... I'll start a thread once we've finalized what all this thing will do... I guess I want a basic idea of the features/etc so that I can give the potential users an idea of what it'll do...

What goals do you think are still not finalized? While there might be a couple of details to settle, I think the goal is quite clear.

Keep on Folding!! For the [H]orde!!

 

[OSUguy98]

I guess, the once-per-week idea was because I thought some might be on dialup/etc...

Is it better to have an "offline" way of doing things? or just assume everyone is connected to the net 24/7? (or that we can do a netrequest that kicks off the dialer/etc).....



As for things not being finalized, maybe it's just that I don't have a full understanding of how we're going about things.... I know the goal: get WU stats from user, send to server, server compiles list and makes a viewable page for the world to see... I guess I just need a better understanding of the how in the steps....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

I guess, the once-per-week idea was because I thought some might be on dialup/etc...

Is it better to have an "offline" way of doing things? or just assume everyone is connected to the net 24/7? (or that we can do a netrequest that kicks off the dialer/etc).....

I think our choices are either to declare that dialup users aren't supported, period; or that, if a connection isn't avaialble, the outgoing communication is queued until a connection is available. When a connection becomes available, the queue is depleted.

I'd prefer the latter, since it takes care of the case where the server is otherwise unavailable. But doing so significantly increases the testing effort I've got to do, since I'll need to knock my machine off the net to test, trigger the update code, then make sure something good happens when connected and nothing bad happens if a connection isn't available for a long time.

Connecting per unit of time doesn't solve anything for dialup users. If they're not connected when the scheduled work comes due, they're in the same boat wishing the outgoing notitications were enqueued.

As for things not being finalized, maybe it's just that I don't have a full understanding of how we're going about things.... I know the goal: get WU stats from user, send to server, server compiles list and makes a viewable page for the world to see... I guess I just need a better understanding of the how in the steps....

I'm writing a service.

At installation time, the user will tell the service what directories have active Folding at Home clients. It will also scan the user's machine and upload the list of inforamtion provided a couple of notes ago. That submission is keyed by the client number and machine number; the location of that info is on the previous page of this thread. This activity probably happens in its own application.

Once running, the service will monitor the log files in each of the directories that it was configured to watch. When any log file changes, the service scans it and tries to find completed work. If it finds completed work, it reports the time taken to complete the work item, as well as the name of the protein that was processed. If there's a startup/shutdown note in the log between the start and stop of the work unit, the work unit isn't reported.

Back on the server, the rather raw data is parsed, validated minimally, and shoved into a database. The website exposes that database to visitors as a set of lists: time per protein type by processor, by frequency, by OS, and so on. Perhaps the other way, too: selecting a processor lists time by frequency and protein type.

Does that clear it up for you? Any questions or comments?

Keep on Folding!! For the [H]orde!!

 

[OSUguy98]

I'm writing a service.

At installation time, the user will tell the service what directories have active Folding at Home clients. It will also scan the user's machine and upload the list of inforamtion provided a couple of notes ago. That submission is keyed by the client number and machine number; the location of that info is on the previous page of this thread. This activity probably happens in its own application.

Once running, the service will monitor the log files in each of the directories that it was configured to watch. When any log file changes, the service scans it and tries to find completed work. If it finds completed work, it reports the time taken to complete the work item, as well as the name of the protein that was processed. If there's a startup/shutdown note in the log between the start and stop of the work unit, the work unit isn't reported.

Back on the server, the rather raw data is parsed, validated minimally, and shoved into a database. The website exposes that database to visitors as a set of lists: time per protein type by processor, by frequency, by OS, and so on. Perhaps the other way, too: selecting a processor lists time by frequency and protein type.

Does that clear it up for you? Any questions or comments?

Yup, clears things up.... a couple questions though..........

What if I add a folding client after the initial setup? do I just run the setup again? or so there some way to add it in while it's running?

How are you grabbing time/frame or time/WU? (start time and end time)/total time? find the mode? median?


Is there anyway I can help? (other than brainstorming and answering general F@H questions)

I'll start a thread to search for a name for this program/project....


Keep on Folding!! For the [H]orde!!

 

[unhappy_mage]

FAH has an option for dialup users to dial the internet when a WU finishes. Thus, if they're using this they'll be automatically be connected when the program tries to do its thing.

mikeblas, that's an excellent summary of what I thought was planned. Glad to see we're on the same page.

 

[mikeblas]

FAH has an option for dialup users to dial the internet when a WU finishes. Thus, if they're using this they'll be automatically be connected when the program tries to do its thing.

Sorry, but I won't implement that. I'm just not interested in doing it or testing it.

mikeblas, that's an excellent summary of what I thought was planned. Glad to see we're on the same page.

One thing that seems to be variant is that the current database does track partial workunit progress, while we're saying we won't. Should we do so?

 

[mikeblas]

The database that has the list of proteins and points already seems out of date; it doesn't match the project list page that OSUGuy linked to, and it turns out one of my two machines is already doing work against a protein that isn't in the database file.

Is there any rhyme or reason to how points are assigned? I got 239 points for something my 2100+ did in about 36 hours, and I'm scheduled for 206 points for something that my Opteron will take 48 hours to complete. Is it more about importance and less about work?

 

[unhappy_mage]

Points are assigned based on how long a p4 2.8 using the Linux client with SSE turned off takes to do the work unit; it's given 110 points per day worth of work. Arbitrary but well-defined.

The proteins and points page is incomplete, which is something that a lot of people writing programs struggle with. The page only shows WUs that are currently being handed out, so anything that gets handed out and then they wait on can immediately disappear from the page. In addition, sometimes the difficulty of a WU is re-assessed so the value of it changes. That makes maintaining a database of things really a difficult task; I'm not surprised it's out of date.

In any case, that's hopefully a server-side concern only. I haven't heard from e08 via email, and he obviously hasn't posted back, so I don't know what's going on with him. Hopefully soon we'll know what the deal is.

 

[OSUguy98]

e08 said he only has net access on the weekends.... so I don't think we'll ever hear from him during the week....

Maybe we should talk to LPerry about using his protein data files? he updates them as needed.... Assuming it was okay with him, we could use that format to get the point data.... The other option would be going through my Excel Spreadsheet that has completed protein data for about 2 years..... This would be tedious to do, but it'd be a good start.....


Keep on Folding!! For the [H]orde!!

 

[mikeblas]

I got bored waiting for webspace, so I finished the client reporter and wrote some simple webpages for it. Here's the results, though they might hiccough while I'm playing around: http://beta.blaszczak.com/ShowListOfMachines.asp

 

[mikeblas]

the Stanford Machine identifier: HKLM\SOFTWARE\PandeGroup\Folding@home\UserID is a REG_BINARY containing this value. It's bytewise reversed from how the value is displayed by the client:

In log: - User ID: 437ECFDC405F93C0
in registry:       C0935F40DCCF7E43

This is called "UserID" in the registry, but it seems different for each machine, even though the machine is registered to the same user. You're positive I can use it for the unique machine identifier, yes?

 

[Mohonri]

This is called "UserID" in the registry, but it seems different for each machine, even though the machine is registered to the same user. You're positive I can use it for the unique machine identifier, yes?

Yep. That's how Stanford keeps track of how many boxes you have, how many boxes each team has, how many boxes are folding world wide, etc.

 

[enhanced08]

Ok guys, I'm here, finally..... I'm sorry about not being around, I'm trying everything in my power to get some internet where i'm staying durring the week but its not going well....

I havnt read all 4 pages of this thread yet but am working on that now. You mentioned that instant messenger isn't an option so, is IRC an option for everyone? Anyone have a channel we could chat in (if I ever get internet)?

I'll be posting answers to whats looks to be tons of questions as soon as I finish reading.

Once again, sorry for my absence.