• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

MC ES Setup Help

402blownstroker

[H]ard|DCer of the Month - Nov. 2012
Joined
Jan 5, 2006
Messages
3,257
Swapping out the 6128 for a set of ES 1.6GHz. First issue is that all the memory is not showing up in the BIOS, only 15 of 15 slots. I have played the DIMM swap party game and it does not seem to make any difference. All 16 DIMMs where showing up and working when I had the 6128s in it. I did apply a 205 OC setting to get the memory flashed. Here is the tpc -dram results:

Code:
[root@ancalagon ~]# tpc -dram
Turion Power States Optimization and Control - by blackshard - v0.43


DRAM Configuration Status

Node 0 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=46
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 1 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=4 TrwtTO=3 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 2 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=47
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 3 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=4 TrwtTO=3 Twtr=5 Twrrd=4 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=53
LDIMM0=EMPTY/EMPTY LDIMM1=FAILED/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 4 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 5 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=4 TrwtTO=3 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=44
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 6 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=46
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=46
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 


Node 7 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=4 TrwtTO=3 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY 



Done.

Looks like tpc is seeing all 16 DIMMs. memtest runs clean without errors. Interesting thing is that only 10MB of L3 cache is reported. Isn't there 12MB? Also memtest does not report quad or dual channel for the memory. You think it is running single ranked mode?

When it boots, the 'Node Interleaving requested, but not enabled error is showing up?

The second big issue is that it is not stable under load. It acts very much like the 6166HE rig when it does not have enough vcore voltage to support the OC. Basically the fah client fires up and after about 30 seconds, the system reboots. It does this at the stock 1.6GHz frequency with the stock vcore value of 1.0500. I have tried bumping up the vcore voltage to 1.150 at the stock frequency and it still does the same. I would expect the system to be stable at stock vcore and frequency let alone a slight vcore bump. To set the vcore voltage I have been doing the following:

Code:
tpc -set vcore 1.150
tpc -fo 1
tpc -fo 0

At this point I am looking for some ideas/pointers to figure out the memory issue and get it stable( even at stock frequency ).
 
According to the result of tpc, one DIMM of Node 3 is failed, you could replace it with a new one and see if it would work.

Node 3 ---
DCT0: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=5 TrwtTO=4 Twtr=5 Twrrd=3 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=45
LDIMM0=EMPTY/EMPTY LDIMM1=OK/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY

DCT1: memory type: DDR3 frequency: 1066 MHz
Tcl=6 Trcd=6 Trp=6 Tras=17 Access Mode:1T Trtp=5 Trc=25 Twr=10 Trrd=4 Tcwl=7 Tfaw=16
TrwtWB=4 TrwtTO=3 Twtr=5 Twrrd=4 Twrwr=4 Trdrd=3 Tref=2 Trfc0=0 Trfc1=3 Trfc2=0 Trfc3=0 MaxRdLatency=53
LDIMM0=EMPTY/EMPTY LDIMM1=FAILED/EMPTY LDIMM2=EMPTY/EMPTY LDIMM3=EMPTY/EMPTY
 
Hi 402,
When I started up my 4P, sudo tpc -dram showed 2 failed memory slots like your Node 3 DCT1, it was traced to TIM on the pads of 2 CPU's. I am a noob, hope this helps. Best regards, Charlie
 
Swapping out the 6128 for a set of ES 1.6GHz. First issue is that all the memory is not showing up in the BIOS, only 15 of 15 slots. I have played the DIMM swap party game and it does not seem to make any difference. All 16 DIMMs where showing up and working when I had the 6128s in it. I did apply a 205 OC setting to get the memory flashed.
Er.. for the record. Setting 200 or more doesn't cause SPDs to be altered. It only makes
OCNG use XMP. Memory is unchanged.

Here is the tpc -dram results:
(...)

Looks like tpc is seeing all 16 DIMMs.
Quickz and thinklet covered this already :)

memtest runs clean without errors.
Possibly, but it certainly sees one DIMM less.
Interesting thing is that only 10MB of L3 cache is reported. Isn't there 12MB?
There is but 2MB is internally used for directory cache and is not counted towards available L3.
Also memtest does not report quad or dual channel for the memory. You think it is running single ranked mode?
G34 chips are not designed or validated to run multi-channel (aka ganged). Each DCT
operates individually. More than that, "single channel" works faster [sic!].

When it boots, the 'Node Interleaving requested, but not enabled error is showing up?
That could be BIOS bug (side effect of one missing DIMM). Make sure
Node interleaving is disabled in the BIOS. With OCNG just load optimal defaults
and save them.

The second big issue is that it is not stable under load. It acts very much like the 6166HE rig when it does not have enough vcore voltage to support the OC. Basically the fah client fires up and after about 30 seconds, the system reboots.
Theare are too many variables -- it's difficult to determine what the problem is.
With ES you should not be setting any OC w/OCNG at all. At least not initially.
Run sudo ./smocng.sh reset for starters and test things at stock clock/timings.

It does this at the stock 1.6GHz frequency with the stock vcore value of 1.0500. I have tried bumping up the vcore voltage to 1.150 at the stock frequency and it still does the same. I would expect the system to be stable at stock vcore and frequency let alone a slight vcore bump. To set the vcore voltage I have been doing the following:

Code:
tpc -set vcore 1.150
tpc -fo 1
tpc -fo 0
This looks all right and yes, stock should work...

I would go about:
1. Turning the OC off
2. Figuring out missing DIMM
3. If that doesn't help, bumping vcore a bit (as you've been doing)
4. If that doesn't help, flashing lastest official ROM
5. If that doesn't help, trying to identify "problematic" CPU+RAM "pair" (run FAH on each CPU individually)
 
Thanks for all the tips/ideas everyone. Look slike I know what I will be doing tonight ;)
 
OK, I traded down the slot and tried swapping around different DIMMs to it, and that does not make any difference. I even tried shifting the DIMMs from the blue slots to the white. Same issue with Node 3 DTC1. Anyone think I board might be bad? This was after I reset the OC.

Actually when I shift all the memory up to the white slots, Node3/DTC1 is now empty.

I am starting to really think the MB is bad. Even with the 6128s, I was getting the "Node Interleaving requested, but not enabled" error.
 
Last edited:
I am starting to really think the MB is bad. Even with the 6128s, I was getting the "Node Interleaving requested, but not enabled" error.
I sometimes see that "error" on boot as well. On all my SM rigs.
I don't think that is a real problem.
As stated here before, my worst problem when swapping CPUs was due to pin contamination from stray TIM.
I would pull the corresponding new CPU (based on the memory slot failure) and examine pads and pins carefully.
Then you could try swapping back to a 6128 for that socket alone to see if the issue persists.
 
As stated here before, my worst problem when swapping CPUs was due to pin contamination from stray TIM.
+1

Another possibility is foreign body in DIMM slot. Check both.
 
I am really starting to think bad MB. I pulled the CPU in socket 2, pads looked clean, clean them again with rubbing alcohol, reseated the CPU, fired it up again, and now only 28GB instead of 32GB. Reseated the ram, still 28GB.

I swapped that CPU out for 6128, boots with 30GB. If I pull the DIMM closest to the socket, 30GB.

Anyone got anything else?
 
Crap..... I was thinking I might be able to limp along with CPUs 1,3, and 4, but I can not get it to boot and it has a weird beep code. Is this config supported?

Looks like it is support, but guess which socket controls the PCIe slots?

God I am so ready to toss the fucking thing into the trash.
 
Last edited:
Time to clean the CPUs again, check the pins... again.

That is strange that the PCI-e slots are controlled by only one socket, very strange. You should be able to put a CPU into socket 0 and boot and run the system entirely.
 
I have cleaned a checked the pins/pads three times now on that socket, I doubt that is the problem.
 
I meant all the CPUs, but I have a feeling that you have done that. Could be the dimm slot has been used too much and is failing. Can happen, but not likely.
 
I got it to boot up with 3 CPUs and 12 DIMMs. BIOS reports 22GB and tpc reports all slots OK except Node 3 DTC1 which is not populate?!?!?! I am still guessing bad MB.
 
Node 3 DTC1 which is not populate?!?!?!
Pic is only usable when all CPUs are populated. If you skip a CPU, numbers change.

You swapped CPU in Socket #2 and that did not improve things. Is this correct?

If nothing stands out in neither CPU #2 nor respective DIMM sockets after thorough
visual inspection then I'd say something's up with the board...
 
Hey tear,

Correct I swapped a 6128 in socket 2 and had a ES in socket 1, 3, and 4. With that 30GB only showed up and Node 3 DTC1 was getting an error.

I am playing around with a couple of other things, but I am guessing it is the board.
 
Well doing a CPU swap party( which I enjoy slightrly more than having bloody diarrhea ) with just socket 1, I found one CPU that has errors with DTC1 in the second node. If I install a good CPU socket 1 and then move the broken CPU around, those corresponding banks develop errors also.

Right now I have a pair running stable at 2.7GHz in sockets 1 & 2. I will be added the third CPU here shortly. These chips took a lot more vcore voltage to run stable than what most other people have had to use.

I will be getting the four CPU, the broken one, swapped out here, so soon I will have 48 cores running at 2.5GHz or higher :)

Thanks for all the help everyone.
 
which I enjoy slightly more than having bloody diarrhea

Welcome to my world about 6 months ago, and my one machine I "affectionately" call demon. An RMA to SuperMicro and 3 sets of chips later, it finally decided to be a 4p. If you are into good omens, I can tell you that after I got it working, it has been rock stable for many months. Hopefully you will see similar results once you get the intial issues worked out.
 
Welcome to my world about 6 months ago, and my one machine I "affectionately" call demon. An RMA to SuperMicro and 3 sets of chips later, it finally decided to be a 4p. If you are into good omens, I can tell you that after I got it working, it has been rock stable for many months. Hopefully you will see similar results once you get the intial issues worked out.

I too had similar issues. Worked a treat out of the box for months, then instability, hard lockups, the works. After cleaning two of the CPUs and a "bloody diarrhea" swap party that lasted for a couple of weeks, it all came good. Now, thanks to tear's help, my 4P 6176SE is solid at 230, 16.
 
The third chip seems to be doing well now. 36 cores at 2.5GHz with 1.300 vcore voltage. Hopefully I can get the issue with the 4 CPU straighten out with the seller.
 
Ugh. Bloody diarrhea swap party....I think most of us have been there.....

Glad to hear things are getting straightened out.
 
Is it that much of a performance hit to run 3 sticks instead of 4 pr CPU?

What "part numbers" do you have, btw? Just curious so i can compare to mine :)
 
Supermicro 4P boards only support dual or quad channel memory. 3 sticks performs as dual channel. And yes, there is a big different between dual and quad channel memory.
 
With 3 sticks per CPU you would probably be able to get a memory performance between two channel and four channel.

Is it that much of a performance hit to run 3 sticks instead of 4 pr CPU?

What "part numbers" do you have, btw? Just curious so i can compare to mine :)
 
With 3 sticks per CPU you would probably be able to get a memory performance between two channel and four channel.

That is not correct. 3 sticks per socket would be around dual channel performance.
 
In the manual of H8QG6(H8QGi), there are several descriptions of the tri-channel configuration:
Thirty-two (32) single/dual/tri/quad channel DIMM slots supporting up to 128 GB of ECC/Non-ECC UDIMM or up to 512 GB of ECC RDIMM DDR3-1600/1333/1066 SDRAM
...
The H8QG6/i-F serverboard supports thirty-two (32) single/dual/tri/quad-channel, DDR3-1600/1333/1066 registered ECC/Unbuffered ECC/non-ECC SDRAM memory modules.
...
Single/dual/tri/quad channel configuration are supported. See Section 2-4 for details on installing memory.
Do you mean that the tri-channel configuration's performance is only as fast as the dual-channel configuration's? Have you tested for it?

Supermicro 4P boards only support dual or quad channel memory. 3 sticks performs as dual channel. And yes, there is a big different between dual and quad channel memory.
That is not correct. 3 sticks per socket would be around dual channel performance.
 
Yes you can plug 3 stick per socket and have all of it usable, but the performance is the same as dual channel. It has been tested and verified in the past.
 
I got it. Thank you.
And if I have time I will perform a test myself and see if there are any tricks for it.

Yes you can plug 3 stick per socket and have all of it usable, but the performance is the same as dual channel. It has been tested and verified in the past.
 
Yes you can plug 3 stick per socket and have all of it usable, but the performance is the same as dual channel. It has been tested and verified in the past.

....twitch...
G34 is odd...

you can technically run in "tri" channel but it isn't tri channel...
There will be a performance increase but it will also induce some nasty load imbalance.

Here is why.
G34 is 2 dies. Each with dual channel...
So you can run dual on 1 and single on the other.
You can call it tri... but it really isn't ... quad channel is dual-dual channel.
Tri is 1 die in single and 1 in dual.
So it technically increases performance...and it does I have tested it...
but not by much because of the load imbalance.
 
Patriot have you tried populating all 32 slots with 1 dead slot does it make a difference or do you know.
 
I thought I remember tear saying that populating more than 4 DIMMs per CPU hurts performance, even with 8 DIMMs.
 
Once you have set the frequency with tpc, is there any way to verify the current CPU frequency. /proc/cpuinfo still shows the original boot frequency.
 
Back
Top