Become a Patron!

New fold-server release

Discussion in 'Distributed Computing' started by unhappy_mage, Mar 2, 2005.

  1. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    Hey guys,
    For those of you interested in farming, a new release of my fold-server distro has just hit the virtual shelves at sourceforge. If you've looked at the idea in the past but gave up on it or tried and failed, this is the one to try it on. Instructions on installing and using fold-server are available here, or you can email me for help.

    I hope this will make things a bit easier for those of you with farms, and enjoy!

    Fold on!
    Unhappy_mage
    [​IMG]
     
  2. Schmave

    Schmave [H]ard|Gawd

    Messages:
    1,718
    Joined:
    Jan 2, 2001
    Sweet. If I had the time and money, I would definately do it. However, college doesn't pay the bills (yet). Someone else try it!
    [​IMG]
     
  3. Oldbenwa

    Oldbenwa Gawd

    Messages:
    741
    Joined:
    Dec 22, 2003
    Any reason for me to upgrade?

    Oldbenwa
    [​IMG]
     
  4. nerk01

    nerk01 Limp Gawd

    Messages:
    187
    Joined:
    Jan 15, 2002
    So, does this use samba yet so I can monitor the clients using EM3 ?
     
  5. NetJedi

    NetJedi n00bie

    Messages:
    44
    Joined:
    Feb 27, 2005
    I was really looking for something like this back in the day when G@H was still going.

    I still have most of that old crap. Too bad it's too slow for Folding.
     
  6. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    I'd suggest it. If you've got a spare disk sitting around, install it on that instead of overwriting so you've got a fallback position if I screwed something up, but it should go smoothly.

    Reasons: the client dns issue is resolved.
    the netconfig script is run over SSL (and everything http, too) now so it isn't quite as hackable as before.
    it's based on DSL 0.9.3 instead of 0.9.2.
    nerk: it does indeed use EM3! go to \\192.168.1.15 in a my computer window to check it out. everything's in share/, based on client name.

    If you need help setting stuff up for this, I should be on around 9 or 9:30 at the latest.
    [​IMG]
    [​IMG]
     
  7. nerk01

    nerk01 Limp Gawd

    Messages:
    187
    Joined:
    Jan 15, 2002
    SWEET, I'm just waiting for the CPU of node #1 to arrive, once things are running smoothly the orders for #2-4 go out.
     
  8. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    Q: The saved work that is stored on the server, is that in a "Linux" only format?

    Could I move the WU's from this to a XP platform?

    I know this sounds wierd, reason is because I can fold 24/7 longer (2 weeks) with my corp boxes if they are under XP (I need my app/drivers ect. loaded to burn in other gear).
     
  9. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    I don't know, I'll try it. If it works at all, it'd be really easy to do: the clients all save their data to a samba share, so you could just copy over the entire directory. The client itself isn't copied over every time, so there's no linux client executables or anything. It does create an empty FAH500-Console.exe for EM3 to notice, but it's easy enough to just skip copying that file over.
    [​IMG]
     
  10. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    One thing I don't know about is how Stanford keeps track of whom it sent what. The reg keys in XP for example. I'm not sure if the box that uploads the data needs to be the same one which down loads the data? Make any sense? I'm thinking it don't matter.
     
  11. SmokeRngs

    SmokeRngs [H]ard|DCer of the Month - April 2008

    Messages:
    16,415
    Joined:
    Aug 9, 2001
    It doesn't matter. Stanford will take the unit back no problem. I've done this many times when I was stuck on dial up and WUs wouldn't upload properly on one box but work fine on another. They were both WinXP boxen, though they did have the different IDs. I've also done XP->2k and 2k->XP although that is the extent of what I've done. I have no Win98 boxen or anything else right now to test anything like that.

    [​IMG]
     
  12. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    Nice sig, I just killed off a few brain cells... It looked like I was talking to myself, took me a few to figure things out.
     
  13. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    I tried copying over the WU this morning, but I hadn't set the date on my fold-server box, so it thought it was 2002 and the box I copied it to said "Oh, deadline passed, try for a new WU". So I set the date right on fold-server, and it said, Oh wait, deadline passed, get another WU. I suppose the smart thing to do would be set the XP box's time to 2002 temporarily and see if it works, but I've never been known to do that. :rolleyes:

    And then I had to leave for school :( . Oh well. More tests to come this afternoon.
    No [H]'s here because of school.
     
  14. Herulach

    Herulach Gawd

    Messages:
    998
    Joined:
    Dec 24, 2004
    Is it possible to get the scripts to do all this running on a standard linux install? That would be top class, as i could get it running on my file server, and save myself some grief when i eventually get round to starting the farm im planning.

    Also, sorry to thread hijack, but as farm as proc choices for farming goes, is it worth getting the hyperthreaded chips over say, an equivalent speed celeron D? In essence, does hyper-threading add ~50% to the points per day, for maybe a 10% cost increase?
     
  15. OSUguy98

    OSUguy98 [H]ard|DCer of the Month - April 2006

    Messages:
    4,000
    Joined:
    Oct 8, 2003
    I think the HT adds about 10-20% to your points... There are others here who can better answer that (and probably back it up with data)...

    back ontopic... I need to find some time to give this a try... I need another PC though... I don't want to switch mine or my mom's over to linux just yet... she has a hard enough time as it is....


    Keep on Folding!!

    [​IMG]
     
  16. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    It is *possible*, and if you want to get this going, email me and I'll send you more information, but the distro is definitely 10x easier.

    Like OSUguy said, 10-20% increase in points. This is also true of non-celeron chips. Stanford prefers that you only run one instance per real processor, though, so if you're in it for the science celeron D's are a win.

    Osuguy: keep in mind this takes almost no machine to get going. You can run the server on a pentium machine, it doesn't take anything fancy. My server is an duron 1200, but I've run it in a virtual machine in Bochs before, which is really slow. I want to try it on a real box at this kind of speed, though. I'll have to dig through all my old junk.

    Come to think of it, I should look into integrating smoothwall/m0n0wall features in this. Router machines would make a really easy place to set this up.
    [​IMG]
     
  17. OSUguy98

    OSUguy98 [H]ard|DCer of the Month - April 2006

    Messages:
    4,000
    Joined:
    Oct 8, 2003
    Well, there are 5 PCs in the house.... 3 are running.... one is dead right now (KT7A that won't post)... and one is my mom's old P-133.... I'm just looking for an excuse to get another folding box in the house ;-)



    Keep on Folding!!

    [​IMG]
     
  18. Herulach

    Herulach Gawd

    Messages:
    998
    Joined:
    Dec 24, 2004
    Ok cool, prob gonna go for the digging an old machine out option though, Just so i understand everything right, it goes
    network->NIC1
    then the farm all connects into NIC2 right?
    Also how well does it play with dhcp? Is it a sort of fine in principle but maybe a bit of a ball ache kind of thing? Or a definate no go area? I ask because experimenting in the past ive had trouble with setting up static ips on the lan, mainly because for some reason my isp decides to change DNSs every couple of days, same 4 or 5 ips, just cycles them, annoying as hell.
    [​IMG]
     
  19. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    The network is set up like you said.

    As long as the router (gateway) you point fold-server to (a linksys box, m0n0wall, whatever) knows how to do DNS you should be OK. I think. Maybe. If it doesn't work automatically, then you'd have to set the DNS for fold-server itself manually, but it does dns for all its clients, so as long as it knows where to look, they'll know where to look too.

    BTW, it has to have a static IP in order to route properly. Imagine there's someone relaying all conversation between you and a friend. In order to talk to your friend, you have to know where to find the middleman. If he keeps moving around, then you'll have no idea where to locate your friend.

    OSUguy: I know the feeling. Right now we're down to 7 computers in the house. I feel lonely ;)
     
  20. Herulach

    Herulach Gawd

    Messages:
    998
    Joined:
    Dec 24, 2004
    Cool, d/led the iso, reading the notes, i assume external netmask is the network side, and internal the farm side? Think thats about it i was confused by. FOund an old p2 in the loft its going on after tea
    [​IMG]
     
  21. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    Yep, you got it. I'll clarify that documentation when I get a chance. Right now it's sort of a Frankenstein of stuff I wrote for the first release and stuff I've tacked on since, so it may be a little inconsistent. I'll get to it :cool:
    [​IMG]
     
  22. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    Hey!!! Don't forget about me!!!!

    I'm not sure is I missed the answer, possible to move WU's from Linux server to XP enviroment?
     
  23. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    I remember you! You were that guy...

    I'm still waiting on Stanford for this, basically. Squid (my mandatory proxy server :( ) is still not completely working, so it takes a while to get a WU. When I get one I'll try, but until then I really can't do anything to check. Updates will be coming when I get a WU...

    Edit: woohoo, got a WU. Copied over the files, everything worked. Finished a frame with no problems. Didn't wait for an entire WU to finish (this machine lacks th3rmal 3ffici3ncy) but it looks good to go.
    [​IMG]
     
  24. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    So just move everything over(/work logs config ect.) and replace the fah502-???.linux thing with the fah502-console.exe (winblows deal) and all should be cool?

    Ahh second question: What server should I use with 50 nodes/2.53 gzh each........

    Ahh third question: You think this would work for 126.5 gzh of boxen?

    If I got the 50 boxes, they will be set to LAN boot. I can have them shipped w/ blank hard drives as they will net boot. Open box, place on shelve and fold. Then ~ 2 weeks before deployment I'd just load my XP image, move the WU's over and keep going till the boxes go out the front door.
     
  25. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    Actually, the problem there is that the Linux end makes a fake (empty) FAH500-Console.exe so EM3 will find it as a valid folding@home directory. Since it isn't the real executable, you wouldn't want to overwrite the real one with the empty one.
    Anything really. The only limiting factor I can tell in terms of expansion is the network bandwidth. Eventually you'll have so many systems sending/receiving WUs that you'll swamp the t1... Oh, and don't turn them all on at once. Each machine has about 12 MB of data to get before it's completely booted. You can opt to make this smaller by removing some of the cores from one of the files (every core in use by Stanford is sent over the wire on boot!) since you'll probably never use most of them. At 100mbit speeds, that means about 1 second (at least) has to elapse between client boots. 3 or 4 would be better, since the traffic isn't all at once.
    Basically, the server just handles a few simple network things that it can do with half its attention, and the real work it does is to fold. A pentium II could probably handle 500 boxen, if it had gigabit ethernet.
    It damn well better. ;)
    How can I get into your line of work? I need the points... :D
    edit: and the $ :rolleyes:
     
  26. rogue_jedi

    rogue_jedi [H]ardness Supreme

    Messages:
    4,857
    Joined:
    Sep 6, 2002
    true dat... where DO you work?

    oh and congrats on the update (don't think i told you yet:p)
    [​IMG]
     
  27. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    One last question (yea not really but...)

    My "reason" for running these systems 24/7 is to break them in. Yea, yea it's a 1/2 way valid reason...sort of.

    Say you take 10 boxes, boot them off the server (they grab DHCP IP's) and one craps out, how am I going to physically know which one of the boxes are dead? All headless boxes.

    My last run of these, ~ 30 or so, I sat them on a bench, imaged them with an XP build-installed the folding doo hicky, set to a static IP and labled them on the outside of the box with IP. It's not bad, just not great either.

    So, in theroy if I paid attention to EM3/Remote desktoped into them I can tell what's working. Somethings EM3 would complain about a box, I'd just reboot it.

    My plan this time was remove from packaging, net boot and let run, install in end use envirmont with XP image move the work WU's over and jam some more till the last minute possible when I need to image with the real XP build.

    And no, the MAC's are not listed on the outside of the box. Maybe I could ask/pay extra to have the MAC's listed onthe outside since the builder needs to power the systems with a monitor anyways.

    Thanks for all you help!
     
  28. rogue_jedi

    rogue_jedi [H]ardness Supreme

    Messages:
    4,857
    Joined:
    Sep 6, 2002
    the only way I can think of is put mac addresses on the outside of the boxen. then compare ip address on the non-responding box to the dhcp logs.

    that, or put them in a row, and turn them on by going down the row, starting one every few seconds... it should get sequential IP addresses.
    [​IMG]
     
  29. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    Power loss would really screw up the IP scheme from the one by one power up.

    IE ummm now how many can I plug in here....
     
  30. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    The dhcp server will remember them for 2 days, I think *checks* yep. So as long as you turn them back on within 2 days they'll get the same IP. I gotcha covered, man.

    You can usually get those power strips at Radioshack @ 2/$10... that's my suggestion. As for lan connectivity, you're on your own. Cheap 24-port hubs from ebay?
     
  31. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    Does your server act as the dhcp server? (I'm too lazy to look it up).

    I've got plenty-o surge strips (spendy ISOBAR ones...hehehe, crap I've got enough voltage regulators if needed too). Before I was running one surge out of another, way not cool.

    I've got an order in to get a few more 20 amp circuits installed, now that's sad. I should have around 80 amps across 4x20 amp lines. Think I can run a few boxen from that?

    I dunno know about all this. I'll use this for sure with the "personal" farm, on these corp boxen, XP is included so it's almost why not use it. It's gonna be 5-6 long hours to image/setup all these under XP, vs. your insto farm. Huh, insto farm I like the sound of that.

    Not if I can figure out how to make a ghost bootable CD with nic drivers my life would be better. Right now, it's two floppies to boot ghost for a ghost cast session. The actual image lay is 1.5 minutes over the wire.
     
  32. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    Yeah, it does dhcp on its own subnet so it won't screw up your existing network.

    I have in the past figured out ghost bootable CDs, but I only have the home edition so you'd need to type in the serial # every time you imaged a machine. Ghost 9.0 (i believe) comes with a bootable WinPE cd that has NIC drivers, but because it's booting Windows from a CD it takes like 5 minutes to start up. And I don't think you can script it so it would immediately start in on the imaging... Oh well.
     
  33. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003
    I'm close to the bootable CD, I'm just gonna finish it before these arrive.

    I got a guys CD form the [H], just won't connect to the session.

    I've got a bootable USB drive, crapper is it's works great w/o a hard disk in the system. Config the bios priorty and it works, till you remove the USB drive. So to boot from USB, config bios, boot every time, config bios....boot.

    yuk
     
  34. nerk01

    nerk01 Limp Gawd

    Messages:
    187
    Joined:
    Jan 15, 2002
    If i've having some problems with it, what's the best way to get some help?

    I'm not a *nix noob, but I'm not much better. As far as I can tell the modprobe to set the nics up are failing. I'll post the exact error msg tonight.


    [​IMG]
     
  35. Oldbenwa

    Oldbenwa Gawd

    Messages:
    741
    Joined:
    Dec 22, 2003
    PM Mage. That's what I've been doing. What type of NICs do you have?

    Oldbenwa
    [​IMG]
     
  36. nerk01

    nerk01 Limp Gawd

    Messages:
    187
    Joined:
    Jan 15, 2002
    I have a few to try, one is an intel pro 100, the other two are 3com 905s ( one is the netboot version, the other the non-netboot).

    I've tried all three combinations of them.
     
  37. Oldbenwa

    Oldbenwa Gawd

    Messages:
    741
    Joined:
    Dec 22, 2003
    Hmmm, funny, The two NICs in mine are both 3c905s. Mage will probably be the one to ask.

    Oldbenwa
    [​IMG]
     
  38. unhappy_mage

    unhappy_mage [H]ard|DCer of the Month - October 2005

    Messages:
    11,534
    Joined:
    Jun 29, 2004
    Those should definitely work, all 3 of them... What do you get if you type "ifconfig -a" at a console?

    This thread is old, wow.
    [​IMG]
     
  39. nerk01

    nerk01 Limp Gawd

    Messages:
    187
    Joined:
    Jan 15, 2002
    ok, nm, NICS are fine, other than my home network is 192.168.0.0

    but i still get these errors:

    Configuring network interfaces.... route: SIOC[ADD|DEL]RT: Network is unreacable

    route: SIOC[ADD|DEL]RT: File exists

    Starting HTTP Server : Socket: Address family not supported by protocol

    fold start called
    tar (child): /foldsave/fold.tar.gz: Cannot open: no such file

    any clue?
     
  40. marty9876

    marty9876 [H]ard|DCer of the Month - February 2006

    Messages:
    4,920
    Joined:
    Jun 11, 2003

    Correct me if I'm wrong.

    Change your network to 192.168.1.x

    Fold server will be at 192.168.1.15

    Need to do a static route throw the router "Network" 10.x.x.x to 192.168.1.x or vice versa on that one.