No progress on clusters / parallel setups / what have you?

F

fastgeek

[H]ard|DCOTM x4 aka "That Company"
Joined
Jun 6, 2000
Messages
6,520
So we have a lot of old servers sitting around and in the near future I'll have permission to experiment with them in regards to messing around with "clustering". I've seen some pretty old threads on this subject, but was wondering if there was any news? Or is going with something like VM-Ware the best way to go? We do have an issue with a product here at work that could use more "oomph" than currently practical with a single server; so the best part of this experiment is adding some more points to #33 and applying that knowledge to work problems as well. ;)

Just can't help walking around here and seeing old racks of Dell 1650's (~16 per rack?) gathering dust and thinking "Hey, I wonder how those old guys would do acting as one big ass computer"... just want to tinker!

Just musing...
 
I would recommend a solution like VMware for that situation, however you have to remember that with VMware you can "cluster" hosts however each running vm still operates on a specific physical server and is limited to the available CPU and Memory resources of said server.

Clustering in vSphere allows you to take advantage of VMware's High Availability (Delayed Failover), DRS (Load Balancing), vMotion (Live Migration) and Fault Tolerance (Instant Failover) technologies to name a few.



Is the product in question web-based? If so there are other load balancing alternatives there that might net great results with or without VMware.
 
In order to get a single folding client to play nice accross multi systems it needs a HUGE amount of bandwidth.

Infiniband is the only option I know of that can move that much data.

We have some budding experts on it.
 
Most Infiniband will leave you with no money to power the machine on after you have it built however. :( Unless you have a ton of money to burn you might just want to run a VM hypervisor.
 
metallicafan said:
Most Infiniband will leave you with no money to power the machine on after you have it built however. :( Unless you have a ton of money to burn you might just want to run a VM hypervisor.
Click to expand...

There are options on fleabay ;)
 
I'm interested in this as well.....I really want to know what kind of throughput is needed from the infiniband?
 
I cant tell you the specific bandwidth required because that would take testing. However infiniband + vSMP have shown 70%+ scaling. How it responds to F@H is another story. The main issue is the cost of licencing. We can get dirt cheap copper based infiniand cards and cables on ebay. I have no doubt the hardware will do it (if we meet bandwidth requirements) however software is a different story. Finding a cheap or free scaling enviorment would be the best thing to do. I have not had the resources as of late to begin experimenting, however if any of you choose to do this I would love to hear how it goes!
 
402blownstroker said:
Yes, cards can be bought cheap on crapBay, but the switch is the expensive part. A half way decent 8 port one will run about $5K :eek:
Click to expand...

Can't you configure these just like two interconnected ethernet ports? One machine directly connected to another?
 
NobleX13 said:
Can't you configure these just like two interconnected ethernet ports? One machine directly connected to another?
Click to expand...

Kinda, and actually, there are a bunch of cards out there that are inexpensive and dual port meaning you can make one master, then have two slaves per PCIe slot.

The big issue is subnetmanager

If someone makes an awesome guide for it, I will host on STH and make sure they get a hook up for some fun hardware.
 
pjkenned said:
Kinda, and actually, there are a bunch of cards out there that are inexpensive and dual port meaning you can make one master, then have two slaves per PCIe slot.

The big issue is subnetmanager

If someone makes an awesome guide for it, I will host on STH and make sure they get a hook up for some fun hardware.
Click to expand...

I bet I could figure out the subnetmanager piece; but it's the actual configuration of some sort of FOSS SMP clustering software that might stop me due to time constraints. I will hunt on eBay for some infiniband cards and see what my budget allows for.
 
Dirt cheap but PCI-X cards so they won't work in current motherboards. Maybe an option for older Socket-F and socket 771 boards which are cheap if someone figures out the clustering side.
 
There are cheap HTX, and PCI-E x8 / x16 cards on eBay :) look for the DDR 4X infiniband cards. Those cards that were linked were 10gbps, the DDR 4X cards are 20gbps and are around the same price and come in different interfaces.

cables ($20):
http://www.ebay.com/itm/New-DELL-GO...665642332?pt=COMP_EN_Hubs&hash=item4601119d5c

20gbps PCI-E x8 card dual port ($30)
http://www.ebay.com/itm/MELLANOX-IN..._EN_Networking_Components&hash=item3f1214934a

Cheap ass 10gbps HTX card ($19)
http://www.ebay.com/itm/Qlogic-Path...2?pt=COMP_EN_Workstations&hash=item439958bf4e
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Untitledone said:
I cant tell you the specific bandwidth required because that would take testing. However infiniband + vSMP have shown 70%+ scaling. How it responds to F@H is another story.
Click to expand...
I would say its closer to 80% and for a Virtual 16P that would pay big time. seen how the steep the point curb is.


Untitledone said:
The main issue is the cost of licencing. !
Click to expand...
I have contacted Mr. Benzi Galili about this subject. they have 2 type of license, the cheap on is the vSMP foundation and in the foundation license they also have CPU levels, and only the lowest class(acording to their standards) have a $400 per socket license..

saddly any 6c/12T Intel CPU is $800 per socket.

lucky for some that the new 6272 is $400 per socket(the only 16 core processor with this price per socket the rest are $800 or more)


I am thinking on building a Proof of Concept(PoC) to test how well does it scale(was able to confirm with Mr. Benzi that is possible to build such small system)

Proof of Concept small build(using the certified hardware list http://www.scalemp.com/spec )

2- H8DGU-F from
2- 8 core Magnycours based CPUs off FleeBay
2. SRD Infiniband Connection Cards and Cables off FleeBay too
2. vSMP license($400 each)

run a few test comparing a native 2P system(2 CPU in one board) vs a vSMP virtual 2P system(1 CPU per Node) now if this scale well(75%+) it will be worth it for virtual 16P systems as they don´t really need an InfiniBand rounter because they have the DirectConnect feature: Connecting up to 4 nodes without InfiniBand switch (back-to-back).

now if you want to contact Mr. Benzi you can email him to [email protected] he has the full price list for each processor(I lost my copy)
 
Scalability would be a huge factor for ScaleMP. If it where 75%, I do not think it would be worth the cost unless you got into 4 or more configurations.

Say a 64-core machine could process a 6904 at 20m30s and make 467K PPD. Even it scale at 75%, 2 64-core combined would make a 128-core and would be process the 6904 at around 12m50s and make 994K PPD. Just the two 64-core machines by themselves would make 934K PPD. It would take $3200 ( 8 x $400 ) to connect the machines via ScaleMP. The $3200 could buy a 3rd 64-core machine to make 1401K PPD for the machines running separately.

If you connected 4 64-core machines, the 6904 could be done around 5m40s and make about 3217K PPD. To connect the 4 machines together would cost ( 16 x $400 ) $6400. Now the $6400 could buy you 2 more 64-cores and have to total of 6. Them 6 of them on 6904s would net about 2802K PPD. The ScaleMP version of the six machines would net about an extra 415K PPD.

Bigger configurations than 4 machines would in theory produce more points than the sum of individual machines. Keep in mind this is assume perfect scalability with the increase of cores and the only loss is in ScaleMP. Lets get real, I am guessing combos over 128 cores efficiency most likely drops around 60% at best. The amount of money and more machines needed to make up in the loss of efficiency would be staggering.... at least for most people I know including myself.

Just my $0.02 as I can not afford any more :D
 
Since your paying per socket for the licensing you might just be better off with cheap 2600k systems chained together. Food for thought.
 
vD_Edition said:
Since your paying per socket for the licensing you might just be better off with cheap 2600k systems chained together. Food for thought.
Click to expand...

Not really. A 4P costs about $800ish. 4 'cheap' ScaleMP licenses is $1600. Better to get a 4P setup. A Xeon 8P board ( Supermicro X8OBN-F X8OBN-CPU X8OBN-BR1 ) runs about $9600, now the ScaleMP solution starts to look a little where licensing will run about $6400( but there is the over head of 8 of everything else to buy and having a pretty fast network ).
 
vD_Edition said:
Since your paying per socket for the licensing you might just be better off with cheap 2600k systems chained together. Food for thought.
Click to expand...

there is no support for single socket system yet. and future SandyB EP license will be $1600 per socket..

so far the 6272 are the most powerful CPU with the lowest price($400 per socket)

I believe the scalability will be close to 85%+ for Virtual 16P System.. but we can´t be sure untill somebody builds one and test it..
 
I think of using vSMP or any other transparent clustering solution as reversing all work
that made multithreaded GROMACS possible (vide: current SMP FahCores)...

I'm thinking ... how big of a cluster you'd need to have to have the project agree
to build "approved" MPI GROMACS for you...

EDIT: and of course, one sure could evaluate cluster performance of GROMACS
over IB/10GbE/whatnot without involving the project
 
Is there no way to cluster boxes using some sort of a linux distro and gigabit network cards?
 
There are many ways, but just about all of them are at the application level. The two major libraries are MPI and PVM. There are very few that do it at an OSish level. ScaleMP is one of the only ones I know of, but I have heard there are others.

In fact the older version of the FAH SMP client used MPI.
 
Last edited:
tear said:
I think of using vSMP or any other transparent clustering solution as reversing all work
that made multithreaded GROMACS possible (vide: current SMP FahCores)...
Click to expand...

As usual tear gets right to the heart of the matter => A2 processes with their own memory spaces vs A3/A5 threads with shared memory.

Back in the good old A2 days when we were playing with recompiled mpiexec's, there actually was very good scaling on a transparent clustering solution I was evaluating. Too bad that company went bankrupt.

Now with multithreaded GROMACS there appears to be much more data in memory that is shared between the threads, meaning any solution with significant node-to-node latencies is going to perform poorly if any writes to that shared memory are involved. ScaleMP has a part of each node's memory reserved for a "cache" of memory on remote nodes, and it works great for read-only accesses, but when writes start happening, the IB latency gets in the way.
 
It would be nice if ScaleMP could offer a limited trial version to test out how bad that latency would be. Obviously 1GB ethernet would be a lot higher and 40GB IB, but how bad would the 40GB IB be?
 
It is around 1 micro second latency on infiniband. Not sure what the latency is on ethernet. However 1gbps ethernet is only around 125MB/s. I dont think that would be enough bandwidth. The slowest infiniband (SDR 1x) is 2gbps (1.6gbps actual) and the cheapo ebay ones are the 10gbps/20gbps variety (8gbps/16gbps real respectively).
 
As an eBay Associate, HardForum may earn from qualifying purchases.
As an eBay Associate, HardForum may earn from qualifying purchases.
402blownstroker said:
Did a cat pee on it?
Click to expand...

1. All equipment is sold-as-is. We are surplus equipment dealers and do not have the ability or knowledge to test equipment for functionality beyond plugging it in and describing what we observe. Buyer is responsible for calibrations, repairs, or upgrades if needed.

thats in their bottom notes, i'm pretty sure they have no clue what it actually is or how to price it.
 
Lord_of_Doors said:
And this is why we love [H]

"Hey, look what I found! Might not work though..."

[later that day]

"Bought one. I'll let everyone know."
Click to expand...

LOL

First test:
Does it smell like pee?
If no-> plug it in. Does it trip a breaker or smoke?
If no-> are there shiney lights?
 
You must log in or register to reply here.
Back
Top