Editing/Engineering Build: Cosmos II - 3960X Sandy Bridge-E Platform

[SonataSys]

What is your crysis 3 peformance like for single player?

Sorry for the delay, but I promise to gather Crysis 3 benchmarks in single and multi-player no later than this coming weekend...

 

[SonataSys]

Are the titans showing pcie 3?
http://1pcent.com/?p=257

Good question!

Yes, all three TITANS are running in faster PCI-E Gen-3 bus lanes:

However, PCI-E Gen-3 was not enabled by default on the MSI Big Bang XPower II mainboard when the cards were first installed.

Note, to enable PCI-E 3 I did NOT need to run the NVIDIA "force" utility (hack) program.

Instead, I simply upgraded to the latest (1.5) BIOS for MSI Big Bang XPower II, then used ClickBIOS to enable PCI-E Gen-3 (because Gen-2 is the default):

Those are the only steps necessary to enable Gen-3 bandwidth across the bus lanes with TITAN cards using MSI Big Bang XPower II.

Based on what I've read, there is no measurable improvement in any benchmark comparing PCI-E Gen-2 and Gen-3 with a single TITAN on X79 LGA 2011 platforms. However, there are benefits when running 2/3/4-way SLI configurations under heavy load.

This is when PCI-E Gen-3 actually comes in handy.

I will need to perform another round of benchmarks across the board, since most of the numbers I have gathered thus far were obtained before I enabled PCI-E Gen-3.

Expect more updates here later this week sharing "final" benchmarks with PCI-E Gen-3 and Precision TITAN overclocking in place...

 

[signalpuke]

I think you will be pleased with the gains.

 

[SonataSys]

I think you will be pleased with the gains.

You are correct, and yes, I am pleased!

Enabling PCI-E GEN3 improved the 3DMark 11 score by 3%, producing a score of 25,921. The prior PCI-E GEN2 score was 25,177.

Although not a big gain, the faster GEN3 bus lanes did produce a noticeably higher 3D score. I'll take whatever gains I can to insure a fast and smooth frame rate:

 

[signalpuke]

Do you recommend the Corsair link system?

I was thinking about getting it, but the Asus board I am looking at seems to have many of the same features and it will support the number of fans I am planning on using.

edit: The PCI-E improvements should be higher with higher resolutions. Have you ran the Unigine 3.0/4.0 benchmarks yet? I found that running that in loop in addition to the GPU-Z render test put a nice load on the entire system.
http://forums.guru3d.com/showthread.php?t=375347

 

[SonataSys]

Do you recommend the Corsair link system?
I was thinking about getting it, but the Asus board I am looking at seems to have many of the same features and it will support the number of fans I am planning on using.
edit: The PCI-E improvements should be higher with higher resolutions. Have you ran the Unigine 3.0/4.0 benchmarks yet? I found that running that in loop in addition to the GPU-Z render test put a nice load on the entire system.
http://forums.guru3d.com/showthread.php?t=375347

I didn't realize Corsair offered monitor/control hardware. Something like the link system is better than having no form of external/hardware monitoring and control-- don't like flying blind. Several cases and some motherboards don't have adequate fan/temp headers, monitoring or control capability, which explains why Corsair is offering its link system. However, higher-end X79 boards and cases should provide enough control without integrating something like Corsair's link system. Then again, as for lighting, perhaps going with Corsair's product makes more sense if you need to control a light show on top of everything else.

I usually prefer to work in the dark (totally dark), so I actually turn off all lighting, including on the motherboard and even the "GEFORCE GTX" logo on each TITAN. Between all the fan headers on the Big Bang XPower II and the stock fan controller in the Cosmos II, all of monitor/control needs were satisfied. The only thing I felt was missing was good case temperature monitoring, so I did add two analog temperature diodes/gauges to quickly and simply monitor case temperatures.

Regarding the PCI-E GEN3 benchmark, as you indicate I must not be generating enough load on the bus while running the standard 3DMark 11 benchmark to see more than a 3% boost. I will probably have to wait until I move from a single panel to a multi-panel configuration. Perhaps going to 4-way SLI would also take more advantage of PCI-E GEN3 bandwidth. Both would multiply bus traffic to considerably higher busy levels...

 

[signalpuke]

Oh, I thought I saw you using the Corsair link system to control the fans. I think I will just use the bios for controlling the fans.

 

[SonataSys]

That's probably the right move-- adding a big external control system usually means an equally big mess of cables to manage...

 

[refraxion]

Sorry for the delay, but I promise to gather Crysis 3 benchmarks in single and multi-player no later than this coming weekend...

Awesome, eagerly awaiting the results!

 

[SonataSys]

Awesome, eagerly awaiting the results!

Sorry for the delay. Here are some Precision/OSD numbers while playing Crysis 3 with maximum rendering quality at 2560 x 1600.

There is no doubt Crysis 3 puts the heaviest load yet on this SLI configuration. The levels are tremendously gorgeous to play through, with frame rates remaining smooth while maintaining a steady 59.8 FPS.

As the HardForum 3-way SLI TITAN review indicated, moving to any higher (multi-monitor) resolution while maintaining smooth game play will require giving up some rendering quality, probably lowering the anti-aliasing down from MSAA 8x.

However, for those satisfied with 2560 x 1600 resolution, Crysis 3 can be played cleanly at 60 FPS with totally maxed-out rendering quality in a 3-way TITAN SLI configuration:

 

[SonataSys]

I decided to turbocharge the system's folding performance by tuning BOINC with release 0.40 software from Lunatics. The default configuration assigns one work unit (WU) per TITAN, leaving each card with a zillion latent CUDA cores. The Lunatics configuration allows threaded GPU-assist on multiple WUs. After a few hours of testing, the best throughput appears to be four WUs per TITAN.

The system has remained stable after running at 100% utilization for the past 24 hours, while the average work credit rate (RAC) jumps into the realm of turbocharged folding!

The BOINC initial log spew:

5/14/2013 5:04:28 PM | | Starting BOINC client version 7.0.64 for windows_x86_64
5/14/2013 5:04:28 PM | | log flags: file_xfer, sched_ops, task
5/14/2013 5:04:28 PM | | Libraries: libcurl/7.25.0 OpenSSL/1.0.1 zlib/1.2.6
5/14/2013 5:04:28 PM | | Data directory: C:\ProgramData\BOINC
5/14/2013 5:04:28 PM | | Running under account SonicAgamemnon
5/14/2013 5:04:28 PM | | Processor: 12 GenuineIntel Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz [Family 6 Model 45 Stepping 7]
5/14/2013 5:04:28 PM | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss htt tm pni ssse3 cx16 sse4_1 sse4_2 popcnt aes syscall nx lm vmx tm2 dca pbe
5/14/2013 5:04:28 PM | | OS: Microsoft Windows 7: Professional x64 Edition, Service Pack 1, (06.01.7601.00)
5/14/2013 5:04:28 PM | | Memory: 31.95 GB physical, 79.88 GB virtual
5/14/2013 5:04:28 PM | | Disk: 906.10 GB total, 524.75 GB free
5/14/2013 5:04:28 PM | | Local time is UTC -7 hours
5/14/2013 5:04:28 PM | | VirtualBox version: 4.2.4
5/14/2013 5:04:28 PM | | CUDA: NVIDIA GPU 0: GeForce GTX TITAN (driver version 314.22, CUDA version 5.0, compute capability 3.5, 4096MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | | CUDA: NVIDIA GPU 1: GeForce GTX TITAN (driver version 314.22, CUDA version 5.0, compute capability 3.5, 4096MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | | CUDA: NVIDIA GPU 2: GeForce GTX TITAN (driver version 314.22, CUDA version 5.0, compute capability 3.5, 4096MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | | OpenCL: NVIDIA GPU 0: GeForce GTX TITAN (driver version 314.22, device version OpenCL 1.1 CUDA, 6144MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | | OpenCL: NVIDIA GPU 1: GeForce GTX TITAN (driver version 314.22, device version OpenCL 1.1 CUDA, 6144MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | | OpenCL: NVIDIA GPU 2: GeForce GTX TITAN (driver version 314.22, device version OpenCL 1.1 CUDA, 6144MB, 4096MB available, 4989 GFLOPS peak)
5/14/2013 5:04:28 PM | SETI@home | Found app_info.xml; using anonymous platform
5/14/2013 5:04:28 PM | | Config: report completed tasks immediately
5/14/2013 5:04:28 PM | SETI@home | URL http://setiathome.berkeley.edu/; Computer ID 6857388; resource share 100
5/14/2013 5:04:28 PM | SETI@home | General prefs: from SETI@home (last modified 19-Mar-2013 05:20:57)
5/14/2013 5:04:28 PM | SETI@home | Computer location: home
5/14/2013 5:04:28 PM | SETI@home | General prefs: no separate prefs for home; using your defaults
5/14/2013 5:04:28 PM | | Reading preferences override file
5/14/2013 5:04:28 PM | | Preferences:
5/14/2013 5:04:28 PM | | max memory usage when active: 16360.19MB
5/14/2013 5:04:28 PM | | max memory usage when idle: 29448.34MB
5/14/2013 5:04:28 PM | | max disk usage: 25.00GB
5/14/2013 5:04:28 PM | | max download rate: 999004 bytes/sec
5/14/2013 5:04:28 PM | | max upload rate: 999004 bytes/sec


For those interested, I am including the various BOINC/Lunatics configuration files below:


cc_config.xml

<cc_config>
<options>
<max_file_xfers>24</max_file_xfers>
<max_file_xfers_per_project>12</max_file_xfers_per_project>
<max_tasks_reported>2000</max_tasks_reported>
<report_results_immediately>1</report_results_immediately>
</options>
</cc_config>

mbcuda.cfg

[mbcuda]
[bus1slot0]
processpriority = abovenormal
pfblockspersm = 16
pfperiodsperlaunch = 200

[bus1slot1]
processpriority = abovenormal
pfblockspersm = 16
pfperiodsperlaunch = 200

[bus1slot2]
processpriority = abovenormal
pfblockspersm = 16
pfperiodsperlaunch = 200

app_info.xml

<app_info> 
    <app>
        <name>setiathome_enhanced</name>
    </app>
    <file_info>
        <name>AK_v8b2_win_x64_SSE3.exe</name>
        <executable/>
    </file_info>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>603</version_num>
	<platform>windows_intelx86</platform>
        <file_ref>
           <file_name>AK_v8b2_win_x64_SSE3.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>603</version_num>
	<platform>windows_x86_64</platform>
        <file_ref>
           <file_name>AK_v8b2_win_x64_SSE3.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>
    <app>
        <name>astropulse_v6</name>
    </app>
    <file_info>
        <name>AP6_win_x86_SSE_CPU_r555.exe</name>
        <executable/>
    </file_info>
    <app_version>
        <app_name>astropulse_v6</app_name>
        <version_num>601</version_num>
	<platform>windows_intelx86</platform>
        <file_ref>
            <file_name>AP6_win_x86_SSE_CPU_r555.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>    
    <app_version>
        <app_name>astropulse_v6</app_name>
        <version_num>601</version_num>
	<platform>windows_x86_64</platform>
        <file_ref>
            <file_name>AP6_win_x86_SSE_CPU_r555.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>
    <app>
        <name>astropulse_v505</name>
    </app>
    <app_version>
        <app_name>astropulse_v505</app_name>
        <version_num>505</version_num>
	<platform>windows_intelx86</platform>
        <cmdline>-v505</cmdline>
        <file_ref>
            <file_name>AP6_win_x86_SSE_CPU_r555.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>    
    <app_version>
        <app_name>astropulse_v505</app_name>
        <version_num>505</version_num>
	<platform>windows_x86_64</platform>
        <cmdline>-v505</cmdline>
        <file_ref>
            <file_name>AP6_win_x86_SSE_CPU_r555.exe</file_name>
            <main_program/>
        </file_ref>
    </app_version>
    <app>
        <name>setiathome_enhanced</name>
    </app>
    <file_info>
        <name>Lunatics_x41g_win32_cuda32.exe</name>
        <executable/>
    </file_info>
    <file_info>
        <name>cudart32_32_16.dll</name>
        <executable/>
    </file_info>
    <file_info>
        <name>cufft32_32_16.dll</name>
        <executable/>
    </file_info>
		<file_info>
			<name>mbcuda.cfg</name>
		</file_info>    
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>610</version_num>
	<platform>windows_intelx86</platform>
        <plan_class>cuda_fermi</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>609</version_num>
	<platform>windows_intelx86</platform>
        <plan_class>cuda23</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>608</version_num>
	<platform>windows_intelx86</platform>
        <plan_class>cuda</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>610</version_num>
	<platform>windows_x86_64</platform>
        <plan_class>cuda_fermi</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>609</version_num>
	<platform>windows_x86_64</platform>
        <plan_class>cuda23</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
    <app_version>
        <app_name>setiathome_enhanced</app_name>
        <version_num>608</version_num>
	<platform>windows_x86_64</platform>
        <plan_class>cuda</plan_class>
        <avg_ncpus>0.080000</avg_ncpus>
        <max_ncpus>0.080000</max_ncpus>
        <coproc>
            <type>CUDA</type>
            <count>0.25</count>
        </coproc>
        <file_ref>
            <file_name>Lunatics_x41g_win32_cuda32.exe</file_name>
            <main_program/>
         </file_ref>
        <file_ref>
	       <file_name>cudart32_32_16.dll</file_name>
        </file_ref>
        <file_ref>
            <file_name>cufft32_32_16.dll</file_name>
        </file_ref>
			<file_ref>
				<file_name>mbcuda.cfg</file_name>
			</file_ref>        
    </app_version>
</app_info>

 

[SonataSys]

The system has received an average of 117,000 credits per day after 3 days crunching at 100% with this configuration.

It has remained solid throughout this period, so I am ready now to declare all three TITANS fully burnt in and operational.

 

[signalpuke]

Never understood the credits thing, what are they 'worth' ?

 

[SonataSys]

Never understood the credits thing, what are they 'worth' ?

The credit thing is nothing more than a vehicle to compare relative throughput across various systems . . . and it probably encourages obsessive/compulsive nerds to go broke even faster as they build ever larger folding empires...

;->