Evaluate my overkill SMP project, please.

Aegri: I have looked into both of those RAID levels and am still unsatisfied with the I/O throughput (and this is only because of my speed fetishism and the desire to make everything faster). I've been trying to figure out how to implement a RAID 7-type system but effing SCC doesn't even deal with hardware anymore and they will not respond to my emails about how their RAID 7 system worked.Also, this doesn't really have anything to do with media although if someone wants to rent computer time from me to do something media based I won't say no. :)
 
Order said:
Aegri: I have looked into both of those RAID levels and am still unsatisfied with the I/O throughput (and this is only because of my speed fetishism and the desire to make everything faster). I've been trying to figure out how to implement a RAID 7-type system
Click to expand...

I hear you, and I know the feeling. However, from what I've read, RAID 10 will give you slightly better read performance.
 
I don't have many concerns about data integrity. I'm going for pure I/Os. I can back things up manually should the need arise. Thats why I'm opting for lots of small drives instead of a few bigger ones. I don't know, though. I'm still a few months from building this.
 
Then tell me more about the type of data you will be storing and processing. The best solution might be to read whatever you need to process in large chunks, hold it in a massive RAM store, and process it that way. Any output, if any is to be created, would then be serialized out to your long term storage.

I need to know more about your plans (if any) for the algorithms you need to run.
 
Order,

With all this talk of massive arrays including the very rare RAID Level 7, why not go solid state?? Budget?
 
I guess it has to do with budget, yes.
I haven't really investigated the cost of solid state recently but last I heard was that it is like $50,000 for 90GB or something obscene like that.
 
get a few 74 GB fugitsu 15k rpm SCSI drives if you want real mean performance.
 
Yeah, I was looking at the 18gb MAU line from Fujitsu...
They have gotten excellent reviews.
 
Update:

I'm going with the 37GB MAU line from Fujitsu. They've shown tremendous performance ability above all other brands and models.

Also, the Tyan K8We is more than likely the final decision for the motherboard because of the staggering amount of options and possibilities that it offers.

Tell me why I'm right or wrong. Please. :)
 
When I saw the proposed system specs..


I touched myself, in a special place.


This is sexy, even if it is overkill. :D
 
you'd be better served with just going with 5 or 6 scsi drives in a raid 5 array for operational stuff (current workflow data storage, os, etc...) and saving your money. you can get a good raid 5 scsi 320 card for $200 used as well as 15,000 rpm scsi drives can be found cheap if you get 8-9 gb models (about $100 apiece) that still gives you about 35 gb of room for things that need extreme hard drive speed

then getting 2-3 or maybe 4 large capacity drives @300 gb to sit off the sata controller for backup/storage of finished work

skip 8 gb of ram right now your stated tasks won't need that much
go for 4 gb (ram will always get cheaper and you can add more later at half the cost)

rack chasis for raid 5 array of 5-6 drives $125 used
chasis for rack to hold dvd burner (for backup and software loading) and your3-4 drives $200
power supply get cheaper here as well as you have cut down on your drive usage

go with the 2 steps down from the 250's @ 246 as you can pick those up used and when dual core comes out sell em again on ebay and buy dual core 850's
 
how big is the data you're working with? Why not just bump up the ram to 16gb (yes costly) and have say a 10gb ram drive. Won't be huge but will be fast. You'd need to have a UPS of course to backup that ram drive in a power loss. Not sure how much data you're working with though.
 
Wetware Interface: First off, your username is awesome. Secondly, I'm going to post a new version of the full system specs because the previous ones have mutated into something different in the last few months. Quite a bit of what you mentioned is now part of the configuration.

Sojuuk: It wouldn't be [H] if it wasn't overkill :).

LittleMe: I'm still unsure as to what the data will be like, unfortunately. This whole thing is based on an experiment attempting to make RISC and CISC processes interoperable at the compiler level to take advantages of the high points of both systems simultaneously. This system will share some weird symbiotic connection to a dual IBM Power5 machine and I need to have pretty much all bases covered with the x86 with regards to fast speeds on all levels.
 
Order said:
LittleMe: I'm still unsure as to what the data will be like, unfortunately. This whole thing is based on an experiment attempting to make RISC and CISC processes interoperable at the compiler level to take advantages of the high points of both systems simultaneously. This system will share some weird symbiotic connection to a dual IBM Power5 machine and I need to have pretty much all bases covered with the x86 with regards to fast speeds on all levels.
Click to expand...


Not sure how much data there will be then. But again, if you can get enough ram into a machine to fit the data on a ram drive. That will be your fastest bet.
 
I'm already annihilating any sort of budget that I might have set for myself. I'm poor, lol. I can't afford 2gb DIMMs of PC3200 ECC.
 
LittleMe said:
Not sure how much data there will be then. But again, if you can get enough ram into a machine to fit the data on a ram drive. That will be your fastest bet.
Click to expand...
If you use a RAM Disk, you still do an I/O operation from the RAM disk to system memory before a program can work with it. The I/O operation is just a copy, but it's still a request, a kernel transition, driver work, and so on. And memory-to-memory copies aren't free.

Why don't you think it's faster to just load the data straight into memory and give the program direct access to it?
 
mikeblas said:
If you use a RAM Disk, you still do an I/O operation from the RAM disk to system memory before a program can work with it. The I/O operation is just a copy, but it's still a request, a kernel transition, driver work, and so on. And memory-to-memory copies aren't free.
Click to expand...

That's still faster than pretty much anything else.
 
LittleMe said:
That's still faster than pretty much anything else.
Click to expand...

It's not faster than doing no work at all. If the data is needed, load the file into program-accessible memory; why waste the memory on a virtual disk, and have to copy the data around anyway?

If the program in question can't handle the volume of data in a single run, working around it with a RAM disk might be the only solution. But in that case, re-writing the program to be more capable is the true solution.
 
I really don't need to go with all SSD.
It would be nice and [H] as fuck but impractical. (Not that other aspects of my configurations are practical, mind you.)

Someone help me out with a gigabit NIC...should I get (2x) 2gb Intels or just (1x) 4gb? I know I could just use the onboard ones but I don't want to hit the processors with a full load when things really get going.
 
mikeblas: said:
It's not faster than doing no work at all. If the data is needed, load the file into program-accessible memory; why waste the memory on a virtual disk, and have to copy the data around anyway?
Click to expand...
Can you elaborate on this? I'm not certain I fully understand what you're saying.
 
The information that I need will already be loaded into the huge amount of RAM from the HDs anyway. I trust whatever the system believes to be the most frequently accessed files to keep in memory. There would be no recognizable difference to load everything to RAM versus allowing the OS or programs to cache what it needs in memory from the drives. A large array of 15k SCSIs will be fast as hell. I really don't think there will be a discernable difference with regards to IOs in my realm of work. Of course, this doesn't mean that I won't test it both ways if I can find a RAM-Drive program that supports partitions larger than 2GB.
 
Order: said:
Of course, this doesn't mean that I won't test it both ways if I can find a RAM-Drive program that supports partitions larger than 2GB.
Click to expand...
I'm aware that there used to be RAM drive applications with a non-limited user-configurable amount of allocatable memory for RAM disks. However, I couldn't find any when I looked for one myself recently. The apps I came across all seemed to have their limitations.
 
I'm sure that with the proliferation of 64bit memory addressing it will only be a matter of time before the RAMDrive storage ceilings are shattered.
 
APOLLO said:
Can you elaborate on this? I'm not certain I fully understand what you're saying.
Click to expand...

It's true what he is saying, although the data would be loaded into a RAM disk, the OS would still copy that data from the RAM disk, into system memroy(RAM). The quickest way would be to code the program to load the data directly into system memory and have it access everything directly from memroy.

Order, for the nic, I love my Intel Pro/1000+ MT Quad port nic. If you have a Cisco switch that supports it, you can setup an ether channel to bind them together for 4gbps (8gbps full duplex).
 
LittleMe: I don't have a Cisco but my Dell 5324 allows aggregate linking. ;) Thanks for the recommendation. I wonder when/if the PCI.E version will come out.
 
LittleMe said:
It's true what he is saying, although the data would be loaded into a RAM disk, the OS would still copy that data from the RAM disk, into system memroy(RAM). The quickest way would be to code the program to load the data directly into system memory and have it access everything directly from memroy.
Click to expand...

Right.

Using a RAM Disk:

1) Create RAM DISK. 10 gigs for RAM Disk, 2 gigs for programs and OS.
2) Copy from magnetic disk to RAM Disk.
3) Run program to use data from RAM Disk. Probbly can't do work in more than, say, 1.5 gig chunks.
4) Loop to 3 until all data processed
5) Copy from RAM Disk back to magnetic disk.

Not using a RAM Disk:

1) 12 gigs for OS and programs.
2) Run program to use data from disk. Probably can work on, say, 11.5 gigs chunks.
3) Write results back to magnetic disk.
 
You must log in or register to reply here.
Back
Top