Tyan s8812 build: red blinking LED problem

[mxa]

Hi everyone,

Excuse my noob-y-ess here, I thought I'd ask about an error I'm having with a Tyan s8812 server build. The system is for academic drug discovery /medicinal chemistry (parallel "docking" of millions of small molecules into proteins associated with human disease, as a means to discover new potential drugs). The server doesn't require extreme reliability, like some of the industrial servers discussed here. Little expenses here and there in the future are not a big deal. I will be the only user of the system. Overall I spec'd out the system to save money, a key restriction being the money left in my grant

Specs:
[a] TYAN S8812WGM3NR mobo
COOLER MASTER HAF 932 case (slightly modified to fit the case better, nice to have access to a machine shop)
[c] Kingston 24GB (3 x 8GB) 240-Pin DDR3 SDRAM DDR3 1333
(PC3 10600) ECC Registered Server Memory Model [72 gb total]
[d] hard drive (1tb) and SSD-SATA (256gb). SSD will only be used while performing parallel operations.
[e] Noctua NH-U9DO A3 heatsink/fans, extremely quiet [x4]
[f] AMD Opteron 6174 12-Core Magny-Cours 2.2GHz [x4]
[g] Athena Power AP-P4ATXK110FEP 1100W EPS-12V Server Power Supply (with dual 8-pin connectors, brand new).
(total ~$6600)

(no special video card needed for my app).

The problem I'm having is the same one described by "jebo_4jc" on these forums. I'm initially testing the system with one CPU. When I turn the system on, the bios does not post, a couple LEDs flash, and then the red LED by the PCI slots starts blinking. As jebo pointed out the error was probably due to the voltage regulation problems on the mobo. I repeated the test with or without memory (1 just 8gb stick for the single CPU), and even without the CPU. No dice, just a blinking red light.

I did one experiment to suggest the Athena PSU might be bad: I took another system I built (8 core dual CPU cluster with an Antec 650w power supply, with a different Tyan mobo), and tried booting that system using the Athena supply, and it did not post. Returning the 8-core system to the Antec supply caused it to post/boot just fine. I was unable to test the Antec supply on the the new server (with red blinking LED) since it doesn't have the 8 pin power connector. (and I don't have an adapter). Trying the Antec supply but without any 8 pin connectors attached, caused the blinking red LED symptom. But I probably can't conclude anything from that.

I guess my experiment showing the new PSU causes the old system to stop booting probably suggests the new Athena PSU is bad. This seems unusual since it's not like the PSU is "dead", and indeed it is brand new. Perhaps the PSU is producing voltages not within the tolerance of the Tyan mobo?

Just wanted to bounce this off the forum before I pay $270 for another supply, and then RMA the Athena supply. I'm thinking about switching for a Corsaire AX1200W based on some of the negative opinions I've heard about Athena, and positive opinions on the Corsaire PSUs. I may eventually move to a redundant supply too, I just need to get the basic thing working first. Any other experiences with robust PSUs for this mobo?

other thoughts? thanks for listening.
-mxa

 

[colinstu]

I think a 1000w+ PSU for this build is a bit... too much. I think you would be able to get away with a ~850w power supply.

With that said, this looks like a much better choice for you. Very efficient, top rated, enough power for these processors, and from a reputable brand. (also modular design). But if you'd like something with a bit more kick, check out other 80+ Efficency rated products from Antec, SeaSonic, Corsair, and a couple others.

Also, just a bit of advice... I hope you bought those processors used. You can get them for half off from ebay vs. buying them new. Would've saved you $2000~

Hope any of this was of use to you, I hope your issue can be resolved.

 

[mxa]

Thanks for your advice colinstu. Yeah, the 1100w PSU is probably overkill for the system I'm building, without a video card, and rather simple hard drive usage. I think the SeaSonic one you mentioned might be perfect, thanks for the tip. Now I'll just try to be hopeful that replacing the PSU solves the problems with the mobo.

I didn't buy the processors used, I guess I didn't know better at the time. (I have little experience with these 12 core CPUs). Also my grant doesn't let me use sites like ebay, so I pretty much had to buy everything new. (so many rules where I work).

-mxa

 

[colinstu]

Thanks for your advice colinstu. Yeah, the 1100w PSU is probably overkill for the system I'm building, without a video card, and rather simple hard drive usage. I think the SeaSonic one you mentioned might be perfect, thanks for the tip. Now I'll just try to be hopeful that replacing the PSU solves the problems with the mobo.

Yeah, exactly. If you had an elaborate hard drive set up in there or a powerful videocard, then a 1000w+ psu would have been more appropriate. Hopefully this will fix the problem.

I didn't buy the processors used, I guess I didn't know better at the time. (I have little experience with these 12 core CPUs). Also my grant doesn't let me use sites like ebay, so I pretty much had to buy everything new. (so many rules where I work).

-mxa

Ahh, alrighty, well I don't feel nearly as bad any more Enjoy all those cores!

 

[musky]

On a side note, I would love to see some pics of the case mod and what the system looks like. I have been kicking around the idea of getting a HAF 932 myself and stuffing an S8812 in it.

 

[jebo_4jc]

mxa,

Sorry to hear about your trouble. As you mentioned, I know first hand how difficult this issue can be. If, indeed, you are experiencing the same problem I experienced, the only fix might be an RMA to Tyan. Unfortunately, that process took a couple/three weeks to complete.

Your PSU issue sounds suspicious however.

I agree that a higher end unit from Antec, Corsair, or Seasonic would be suitable replacements.

 

[Linden]

Just a few weeks ago I attempted a G34 build with a Tyan S8812. My troubleshooting was exhaustive: 1 & 2P, swapped RAM, swapped power supplies, IPMI, checked all jumpers (about 100 times)..and about 20 other things. It had the same symptoms you describe and no beep codes. No BIOS POST, not once. I returned the board and replaced it with an Asus G34 2P, which was instant success.

You are connecting an 8-pin power plug to each of the two EPS-12 board connectors, right?

 

[mxa]

Thanks to everyone for all of the useful comments.

Musky: sure, will do in a little bit. The basic mods (in text for now, pix later):
(a) break off two little metal flaps, that were intended to support a power supply. This left behind some metal roughness. The rest of the steps were by the machine shop guy on campus:
(b) smooth down the metal roughness from the broken flaps. He also used some tool to bend it all down into the case
(c) drill a single hole into the case. (most of the other holes lined up perfectly).
(d) construct an aluminum strip to fill a gap left in the case, where three more holes are required. The metal strip was attached into the case with double-sided tape. I did find someone on the net who accomplished this, without the custom metal stick, using zip ties:

http://www.amdzone.com/phpbb3/viewtopic.php?f=521&t=137455&start=42
(see the top post by user "decentfan". look for the zip ties attaching the motherboard to the case).

------------

Jebo: thanks. I hope it's just the PSU, which I think my experiment determined is bad. I'm also hopeful that usually when I receive one part that's bad, the odds of another part being bad (mobo) are pretty slim. (That's me being naively optimistic

That said .....

-------------------------------

Linden: What you're saying definitely concerns me, it looks like you really looked at everything. Did you ever try contact Tyan? Their customer support looks pretty frustrating, yet I may try contacting them depending on the effect of the new PSU.

Since I have all the CPUs, memory, etc., it crossed my mind to just create two 24 core systems rather than one 48 core system. (maybe with the same mobo you are using, which I can still parallelize with my application). I have a feeling that in the long run, they might be more robust than the 1x48 solution, but you never know.

Yes: two (or one) of the 8-pin power plugs were connected to the EPS-12 board connectors. Another reason I am suspicious of the Athena PSU, is when I unplugged one of them (not while the system was running of course), I tested the voltage levels with a multimeter, and they were sort of fluctuating between 12.3 v and like 10v. To me that suggests a PSU problem, and supports what we think the red LED is telling us (something about unstable power).

Right now I'm just hoping the new PSU solves everything, I should have it tomorrow I hope!
-mxa

 

[Vaulter98c]

I'm kinda worried that everyone seems to be having the same problem with these boards

 

[firedfly]

I'm kinda worried that everyone seems to be having the same problem with these boards

I'm not having issues with mine. I *believe* mine was acquired as an open box from newegg.

 

[tear]

No issues here either.

 

[jebo_4jc]

I'm not having issues with mine. I *believe* mine was acquired as an open box from newegg.

mine was open box from newegg as well. It worked fine for about a month, but then suddenly was doing the no-post routine. Sent it to Tyan and they replaced a bad capacitor or something...I forget exactly. It's been solid since.

Oddly, while mine was exhibiting this behavior, I had IPMI access...I just couldn't wake the thing up.

 

[Linden]

Did you ever try contact Tyan?

Having spent a week of time, on and off, essentially most of my spare time, and more troubleshooting than I've done with any board in 12 years of building, I declined to extend the period of frustration. No, I did not contact them.

Having dealt with an excellent, honest seller, there was no problem at with the return. @Jebo, yes, I was able to connect to the board via IPMI, but it returned no logs and no sensor readouts, and most of the options were greyed out.

@ mxa:

it looks like you really looked at everything

Believe it, including a new Corsair AX 750 ("needed" that anyway) and voltage testing with a voltmeter.

 

[pjkenned]

mxa where are you located? If you were at Cal or Stanford let me know as I have a few that worked with the S8812 around.

 

[mxa]

mxa where are you located? If you were at Cal or Stanford let me know as I have a few that worked with the S8812 around.

Hi Pjkenned-
Actually that was a pretty good guess. SF State University Chem Dept, I live in San Francisco. You're saying you have a couple good power supplies?
thanks
-mxa

 

[pjkenned]

PM'ing!

 

[mxa]

Hey everyone -
I received the Seasonic 850w supply suggested by colinstu. It's a very nice power supply, and I'm happy I got it, but this does not fix the problem (no post, just an annoying red blinking LED). Based on jebo's statement above of having the same problem, RMA'ing, and getting a good board back, I decided to do that.

I was considering RMA'ing via Newegg and ordering a replacement, except they're sold out. I took the plunge and called Tyan. (The online tech support / chat feature / knowledge base is completely FUBAR for me, and just crashes).

Surprisingly, the Tyan tech support guy was great. We did a little bit of over-the-phone troubleshooting, and then I convinced him that I just needed to RMA it.

Hopefully I will have a similar experience to jebo, and have a beautifully working mobo in a couple weeks! I'll post a followup when I receive the new board. I can't imagine it being anything else at this point.

-mxa

 

[colinstu]

Hey everyone -
I received the Seasonic 850w supply suggested by colinstu. It's a very nice power supply, and I'm happy I got it, but this does not fix the problem (no post, just an annoying red blinking LED). Based on jebo's statement above of having the same problem, RMA'ing, and getting a good board back, I decided to do that.

I was considering RMA'ing via Newegg and ordering a replacement, except they're sold out. I took the plunge and called Tyan. (The online tech support / chat feature / knowledge base is completely FUBAR for me, and just crashes).

Surprisingly, the Tyan tech support guy was great. We did a little bit of over-the-phone troubleshooting, and then I convinced him that I just needed to RMA it.

Hopefully I will have a similar experience to jebo, and have a beautifully working mobo in a couple weeks! I'll post a followup when I receive the new board. I can't imagine it being anything else at this point.

-mxa

Sorry to hear the new PSU didn't help hope the RMA'd board will fix it! Good luck!

Are you going to keep this new PSU over the Athena? I would, and so would others.

 

[mxa]

Sorry to hear the new PSU didn't help hope the RMA'd board will fix it! Good luck!

Are you going to keep this new PSU over the Athena? I would, and so would others.

Yep, keeping the Seasonic 850, already RMA'd the Athena. I like the modularity of the Seasonic PSU, and in general it has wonderful ratings. It turns out that the Athena was probably fine in terms of generating power, and I think my previous experiment demonstrating the Athena was defective, had a flaw. But since it's easy to RMA right now, I'm sending the Athena back.

Anyway, hopefully the RMA'd S8812 mobo fixes the problem.
-mxa

 

[mxa]

Hi everyone -
Just reporting that the flashing red LED problem with my Tyan S8812 mobo, the subject of this thread, was definitely just a problem with the mobo. After replacing it (convenient since Tyan is in Fremont which is close by), the earlier problems went away and now I have a beautifully working system. Thanks to everyone who helped me troubleshoot this.

The Tyan tech said it is an occasional problem they see, and has to do with one faulty resistor (I think is what he said), which prevents the CPU from receiving power. I'm hoping it might be something that occurs in shipping, and is not an issue once the system is stably mounted in a case. So far I am very happy with the motherboard, and hope it remains reliable.

One caveat: it's supposed to be a 48-core system (4 x 2.2 ghz 12-core magney cours processors). My first attempt at starting this system with the new mobo, with a single CPU and 1x8gb memory stick created another interesting problem: no red LEDs, a selection of nice green LEDs (good!), however, the system would not post, and no output occurred to the monitor. I thought the "VGA-disable" jumper might be closed, but that wasn't the case. I decided to swap the CPU, and swap the memory stick, and then the system worked perfectly, and posted. Then I sequentially installed the other two CPUs, they worked. Then I installed all 72 gb memory, that all worked. And then I installed the suspected-dead CPU in the remaining socket, and then the system wouldn't post again. Conclusion: that one CPU was received DOA, and I am in the process of getting another one.

So for the time being, I have a 36 core system, I installed OpenSuse 12.1 is working beautifully for my application. I am using Noctua NH-U9D0 heatsinks/fans, which keep the CPUs nice and cool (and are incredibly quiet!). I'm maintaining like 35-40 degrees at load. And the HAF-932 case, with minor modifications, is perfect for this box. I'm still optimizing the cooling in the case. I'll try to get pictures up soon.

One other minor note, which I'm sure other people have noticed: If you're using three CPUs, they need to be installed in sockets CPU0, CPU1, and CPU3 (the third one should not be in CPU2). I did put the third processor into CPU2 initially, and it caused another red LED problem, but then I saw in the manual that CPU3 needs to be used instead. Crap I should read things more carefully.

final specs:
--------------------------
[a] TYAN S8812WGM3NR mobo
COOLER MASTER HAF 932 case (slightly modified for the huge mobo).
[c] Kingston 24GB (3 x 8GB) 240-Pin DDR3 SDRAM DDR3 1333
(PC3 10600) ECC Registered Server Memory Model [72 gb total]
[d] hard drive (1tb) and SSD-SATA (256gb).
[e] Noctua NH-U9DO A3 heatsink/fans, extremely quiet [x4]
[f] AMD Opteron 6174 12-Core Magny-Cours 2.2GHz [x4] (once I get a replacement ...)
[g] SeaSonic X series X-850 PSU (thanks for the tip on this great PSU).

-mxa

 

[musky]

Maybe it is a Tyan thing, but I ran a SuperMicro 4p board with the three processors in CPU sockets 0, 1, and 2. One other thing - with 36 cores, you can only use 32 for one instance of F@H (multiples of 8 - 24, 32, 40, 48, 56, 64, etc.) We do have a custom version of the kraken tear made to let the remaining 4 cores run smp, if you are interested. I don't know when you plan on getting your 4th CPU. Shoot me a PM or jump on irc if you are interested.