Getting all miffed

[marty9876]

This is really starting to make me mad. I'll admit I don't monitor the boxen as much as I should- it's just very hard when your actually working your butt off. At the end of the day, I really don't feel like pounding on these things but anyways...

Found 7 of ~40 boxen doing this. Seems that once a week if I let things go about the same amount end up pissing away.

EM3 reports "Unk"

FAH log shows-

[22:15:54] + Attempting to get work packet
[22:15:54] - Connecting to assignment server
[22:15:55] Couldn't send HTTP request to server (wininet)
[22:15:55] + Could not connect to Assignment Server
[22:21:00] Couldn't send HTTP request to server (wininet)
[22:21:00] + Could not connect to Assignment Server 2
[22:21:00] + Couldn't get work instructions.
[22:21:00] - Error: Attempt #87  to get work failed, and no other work to do.
             Waiting before retry.

Over and over again. It's the "older" client, 5.02 non beta. Stopping the service and restarting it fixes the problem every time (no system reboot).

Any ideas here? I'm thinking about nightly reboots but that's kinda harsh. Plus using scheduled tasks coming out of sysprep you need to enter a new password everytime which kinda blows.

Or I'll just move the whole mess outside my network and let someone else pound on it.

 

[BillR]

This is really starting to make me mad. I'll admit I don't monitor the boxen as much as I should- it's just very hard when your actually working your butt off. At the end of the day, I really don't feel like pounding on these things but anyways...

Found 7 of ~40 boxen doing this. Seems that once a week if I let things go about the same amount end up pissing away.

EM3 reports "Unk"

FAH log shows-

[22:15:54] + Attempting to get work packet
[22:15:54] - Connecting to assignment server
[22:15:55] Couldn't send HTTP request to server (wininet)
[22:15:55] + Could not connect to Assignment Server
[22:21:00] Couldn't send HTTP request to server (wininet)
[22:21:00] + Could not connect to Assignment Server 2
[22:21:00] + Couldn't get work instructions.
[22:21:00] - Error: Attempt #87  to get work failed, and no other work to do.
             Waiting before retry.

Over and over again. It's the "older" client, 5.02 non beta. Stopping the service and restarting it fixes the problem every time (no system reboot).

Any ideas here? I'm thinking about nightly reboots but that's kinda harsh. Plus using scheduled tasks coming out of sysprep you need to enter a new password everytime which kinda blows.

Or I'll just move the whole mess outside my network and let someone else pound on it.

It’s not you, it was Stanford. Every once and a while their assignment servers screw the pooch and it happed again.

Thankfully it’s not often and as a rule doesn’t last long when it happens.

 

[marty9876]

The problem is it's ~ 1 hour between retries I think. On this box that's 87 hours of folding time. Across 7 that's lots.

I know eventually it should catch up with the assignment server, however some of these seem to bang their head against the wall for days and days. If the stop/start on the service did not fix the issue, then I could understand more (assignment server down ect.)

Too bad EM3 can't scan for this in the log, once it occurs issue a stop/start to the service.

Hey mage, wanna program under windows for a bit? Little program, scans log every 5 minutes, finds two of these and reboots the box/stop-start the service?

 

[unhappy_mage]

The problem is that it'd be unable to tell the difference between this and a real server outage.

That, and I have no idea how to write this program for Windows.

 

[rogue_jedi]

The problem is it's ~ 1 hour between retries I think. On this box that's 87 hours of folding time. Across 7 that's lots.

I know eventually it should catch up with the assignment server, however some of these seem to bang their head against the wall for days and days. If the stop/start on the service did not fix the issue, then I could understand more (assignment server down ect.)

Too bad EM3 can't scan for this in the log, once it occurs issue a stop/start to the service.

Hey mage, wanna program under windows for a bit? Little program, scans log every 5 minutes, finds two of these and reboots the box/stop-start the service?

no idea on the program. and it's ~48 minutes between server checks, btw.

i get a lot of those stupid squid. i wonder if i get get 8080 opened and if that'll let it work?

 

[marty9876]

I think I can talk a guy at work into writing it. Need help on the logic-

Program to scan fahlog.txt every ~5 minutes

If see this string of txt, stop/start service

Simple enough logic?

Wirst case is if Stanfords server is down, hammers on it a little more often?

 

[unhappy_mage]

Quite a bit more often - FAH only checks once an hour, so hitting it every 30 seconds (that's about how long it takes to restart FAH) instead is 120x more often. If a server actually goes down, a lot of clients will start piling up and you could accidentally flood Stanford with requests. Even on just a t1.

What you need to add is a delay code - only restart FAH every 59 minutes (so it won't send its second query and then get restarted and send another one) at most. This should prevent the flood potential. Also, you'll want to look at just the last line of FAHlog, or remove the file so it doesn't go into endless loop mode. You could also do what "tail -f" does:

        while (follow) {
                sleep(sleep_period);
                i = 0;
                while ((nread = tail_read(fd, buf, sizeof(buf))) > 0) {
                    tail_xbb_full_write(buf, nread);
                }
        }

Code loosely taken from busybox. Basically, you don't want to just read the file once and then die, but rather read the file continually and check for updates. I'm sure whoever you ask will get it.

 

[marty9876]

The real thing that I think is happening here, forgive the marty logic, is the thread which FAH has running to check again is getting stuck.

FAH can't get a WU>ok I'll retry in 48 minutes> reality is I'll post into the log I tried but really I'm stuck in the bathroom>rinse and repeat.

Where this does not hold true is sometimes, rarely, FAH does actually pick thing up and grab another WU.

Just a endless loop checking for this error, error occurs kill service/start, write a flag to itself saying when it did this and don't do it again for ~10 minutes (prevent death loop)

 

[unhappy_mage]

I have no idea what you just said.

 

[Hito Bahadur]

Something definately has been going on recently. I came back from being gone for 2 weeks and I had 4 systems that had been doing that for days. I stopped and restarted the process and all was good, but one of the systems started doing that again. At least one of the systems was usng the 5.03 client and strangely enough it was an X2 system. The othe process was still chugging along. I've probably lost about 10k over the past 2 weeks to this problem.

 

[Dark Ember]

Hmm, I should check on all of my systems... there are a couple I haven't looked at in quite a while.

 

[marty9876]

I have no idea what you just said.

Me neither

Short form: FAH hits bump, gets stuck, don't recover

Think the issue is locally (FAH program) vs. something @ Stanford server end

 

[rogue_jedi]

it may actually be something on their end... i'd imagine that if they change their server code, they need a new client, or new assignment servers. if they do one without the other, then they get problems... like you getting stuck on an assignment server that doesn't work with your client. or something.

(<- has no idea how stanford really does it)

does the 5.04 beta client do it?

 

[marty9876]

after finding 7 yesterday and fixing them all back up--- just found two more today.

Maybe I'll try the beta client on the next load.

 

[unhappy_mage]

I've always been amazed at the way Stanford doesn't have a redundant server system set up. The *first* thing I would've looked at would be LVS or something similar. Set up a WU DB server (or a few redundant ones with Infiniband between them), a few frontends, and that's all it takes. Not one server per project. No more can one server go down and take out the entire infrastructure. You can set up redundancy in every part of the project's servers, so there's no single point of failure. This seems like the simplest way to implement a big system like this.

 

[marty9876]

My thing is I don't think the server is hosed in 99% of my case.

Stopping/starting the service works (which in theroy FAH console was doing my itself)

I think the client is worse off than the servers. IF the server was fubared then the stop/start locally would do nothing

 

[Codegen]

Yeah, it does piss me off at times when it does that.

Why can't it just try another server?

Actually, it's 1059 now.