Project: Galaxy 5.0

Wow... :eek:

I've just spent the better part of three hours reading your worklog, and I must say that I'm very impressed by your "personal" server. It must certainly must be nice to have something like that to play around with. :D
 
Just because of the nice compliments, another couple of teaser pictures.

Intel PRO/1000 PT Quad Server PCIE adapter
dsc06467ji6.jpg


dsc06469kf5.jpg
 
Ockie said:
Just because of the nice compliments, another couple of teaser pictures.

Intel PRO/1000 PT Quad Server PCIE adapter
Click to expand...

I do not know if I can take many more teasers man!!! I might j-j-j-u-u-s-s-s-t-t-t-t fall over!!!
 
Hey Ockie - nice quadport NIC - been thinking about that same NIC myself - so I assume you're going to put them in a bonded configuration, right? Does that require a special switch that understands bonded sets of NICs, or does it tend to work with any switch?

In other words even though presumably all four nics show up to the switch as a single MAC, I'm curious how the switch distributes traffic to those four physical connections on the switch side. Perhaps I should just go read up on bonding :)

Also, Hillary for president. :)
 
odditory said:
Hey Ockie - nice quadport NIC - been thinking about that same NIC myself - so I assume you're going to put them in a bonded configuration, right? Does that require a special switch that understands bonded sets of NICs, or does it tend to work with any switch?

In other words even though presumably all four nics show up to the switch as a single MAC, I'm curious how the switch distributes traffic to those four physical connections on the switch side. Perhaps I should just go read up on bonding :)

Also, Hillary for president. :)
Click to expand...

Some switches have had issues with it but generaly you are fine with anything modern that is semi good.

Rob Black said:
What does that NIC do? allow for faster transfers? :confused:
Click to expand...

Yes. With dual or quad nics(in the OPs case he might be able to bind it with the onboards as well) you can setup them with redundancy so if you loose a nic or port on a switch the server stays up for for higher bandwith as the server will tied them together as one.

Just another way of keeping uptime high in enterprise grade hardware or keeping the speed up. In this case it it cheaper for him to go this route the just get 10gb hardware.
 
protias said:
Holy cow, how much did that set you back?
Click to expand...

More than I would have liked. This time it felt like an arm.

gjvrieze said:
I do not know if I can take many more teasers man!!! I might j-j-j-u-u-s-s-s-t-t-t-t fall over!!!
Click to expand...

Don't hurt yourself :p

odditory said:
Hey Ockie - nice quadport NIC - been thinking about that same NIC myself - so I assume you're going to put them in a bonded configuration, right? Does that require a special switch that understands bonded sets of NICs, or does it tend to work with any switch?
Click to expand...

Yes, these will be teamed up to provide a quad path to the network. If you have a very modern and good switch, it may auto aggregate the ports, but most wont. You need to manually set up this configuration with your switch. If you do not setup this config with your switch, it will basically act as a failover system instead of a multi path fail over system. So this has to be a switch supported function.

Keep in mind, this is from my experience and my switches, it may be diffrent for other switches or other peoples configurations.

odditory said:
In other words even though presumably all four nics show up to the switch as a single MAC, I'm curious how the switch distributes traffic to those four physical connections on the switch side. Perhaps I should just go read up on bonding :)
Click to expand...

This is why it has to be properly setup on both ends so that the switch can send data down all the availible paths. I intend on maxing this nic out this weekend, maybe if I'm nice I'll post some benchies.

Now if I can only get a 10GigE port :)

odditory said:
Also, Hillary for president. :)
Click to expand...

Wtf? You mean the suitpants wearing kankle queen? Okay okay I kid I kid. I'll keep politics out of this :)

Kinda loss to the comment.


Rob Black said:
What does that NIC do? allow for faster transfers? :confused:
Click to expand...

Yes, approximatley 4 times the bandwidth if I only bond the 4 ports on that nic.

swatbat said:
Yes. With dual or quad nics(in the OPs case he might be able to bind it with the onboards as well) you can setup them with redundancy so if you loose a nic or port on a switch the server stays up for for higher bandwith as the server will tied them together as one.
Click to expand...

Yes, since these are both supported Intel chipsets, I can trunk all 6 ports. I will most likley keep it to just the controller ports of 4, so I am able to utelize the other two ports for other functions.

swatbat said:
Just another way of keeping uptime high in enterprise grade hardware or keeping the speed up. In this case it it cheaper for him to go this route the just get 10gb hardware.
Click to expand...

Yeah, it's cheaper, thats the main reason. Rest assured, galaxy 5.0 will see 10gb :) I'm just still trying to find a reasonable priced switch with 10gbe.
 
All I can do is hope that someday I have the money to have toys like that. But very nice work. I'm here trying to figure out the best way to get some kind of a server with out spending to much money.
 
Ockie said:
This is why it has to be properly setup on both ends so that the switch can send data down all the availible paths. I intend on maxing this nic out this weekend, maybe if I'm nice I'll post some benchies.

Now if I can only get a 10GigE port :)

Yeah, it's cheaper, thats the main reason. Rest assured, galaxy 5.0 will see 10gb :) I'm just still trying to find a reasonable priced switch with 10gbe.
Click to expand...

You know I have a pallet of Infiniband switches, as well as dual port 10Gbps duplex Infiniband HCAs, right? :)
 
pissboy said:
You know I have a pallet of Infiniband switches, as well as dual port 10Gbps duplex Infiniband HCAs, right? :)
Click to expand...

Yeah I remember us talking about them. Glad to see you found me :)

How are they working out for you? IM me.



Update:

I am a little dissapointed to inform you that the final component (Areca Battery Backup) will only arrive on monday, not today as I had hoped. The shipper took a while to get the part out so I'm without it for the weekend. The other parts arrives today and tomorrow.

I'm planning for some performance numbers for you guys and pics this weekend.


On a side note:

I have been working on mulling ideas for the datacenter project for galaxy and I have been looking for good deals on some racks. Also, I did find the ideal switch that I've been looking for that features 10gige ports... but this component can wait, they only get cheaper the longer I wait.
 
Ockie said:
I have been working on mulling ideas for the datacenter project for galaxy and I have been looking for good deals on some racks. Also, I did find the ideal switch that I've been looking for that features 10gige ports... but this component can wait, they only get cheaper the longer I wait.
Click to expand...
DATACENTER!!! Man, can't wait for the end result.

I hope you're going with an HP cabinet/rack! :)

Those 42U cabinets are very sharp looking. :)
 
When you have the system fully up and running (OK, so now you do), I'd like to see the power usage numbers. :eek:
 
By the power of Greyskull!

You have the storage!!! :eek::eek:

:D:D

Awesome work so far, like your other Galaxy threads. :D

Keep it up Ockie & co. :)
 
beowulf7 said:
When you have the system fully up and running (OK, so now you do), I'd like to see the power usage numbers. :eek:
Click to expand...

It's about 4 amps at 120v. I will do more detailed checks when I get the time :)
 
Okay, so my weekend has been interesting. I'm just glad I am here and alive today... got into a snowmobile accident at 100mph :eek: To say the least, my brand new Yamaha Apex GT is trashed... so thats going to be my new project for the next while. :(

Anyways, I managed to get awesome transfers through galaxy this weekend, I had the quad nic do some load balancing, I consistently ran into the problem that the other machines peaked out before I could saturate my entire quad aggregate pipe :( However, I did get a consistent 1.2-1.8Gbps for extended periods. Peaked out at 2.8Gbps via the network... if I had a more ideal switch setup configuration, I'm sure beyond a doubt that I would be able to saturate all 4's.

The system ran awesome, no problems, it's just a dream. I will post final pictures tomorrow night, I wanted to post some this weekend, but other little more serious things consumed my time.
 
Ockie said:
Okay, so my weekend has been interesting. I'm just glad I am here and alive today... got into a snowmobile accident at 100mph :eek: To say the least, my brand new Yamaha Apex GT is trashed... so thats going to be my new project for the next while. :(
Click to expand...

Geez, 100mph, at that speed you are lucky to be alive!!! That explains why there was no updates on Galaxy 5!!!
Anyways, I managed to get awesome transfers through galaxy this weekend, I had the quad nic do some load balancing, I consistently ran into the problem that the other machines peaked out before I could saturate my entire quad aggregate pipe :( However, I did get a consistent 1.2-1.8Gbps for extended periods. Peaked out at 2.8Gbps via the network... if I had a more ideal switch setup configuration, I'm sure beyond a doubt that I would be able to saturate all 4's.

The system ran awesome, no problems, it's just a dream. I will post final pictures tomorrow night, I wanted to post some this weekend, but other little more serious things consumed my time.
Click to expand...


Very cool, cannot wait to put my network behind the new switch, so I can see what mine will do, it is really hard to hit the network at those speeds, just do not have enough load around home:)

Sweet deal, cannot wait to see the final pictures!!!!
 
Ockie, sorry to hear about your snowmobile accident. :eek: Glad you are OK. Your snowmobile is replaceable; you are not. ;)
 
Ockie: can you give me any idea on how long OCEs were taking for you with your 1TB drives?
 
When I went from 8 to 12 drives, it took me 7 hours with 500gb drives to expand the array and another hour or so to expand the volume. Will probably be faster on your card as I have the IOP331 processor.
 
Blue Fox said:
When I went from 8 to 12 drives, it took me 7 hours with 500gb drives to expand the array and another hour or so to expand the volume. Will probably be faster on your card as I have the IOP331 processor.
Click to expand...

Blue Fox, was that one 500GB at a time, or in a bunch??

It has been taking me around 24hrs a 750GB to expand, was wondering what Ockie's times are like with 1TB drives......
 
Ockie said:
if I had a more ideal switch setup configuration, I'm sure beyond a doubt that I would be able to saturate all 4's.
Click to expand...

What switch setup are you going to?? Also, is this just for at home, or do you take your switches to LAN parties??
 
gjvrieze said:
Blue Fox, was that one 500GB at a time, or in a bunch??

It has been taking me around 24hrs a 750GB to expand, was wondering what Ockie's times are like with 1TB drives......
Click to expand...
That was for all of them at once. Are you running in background mode with a very low priority?
 
Blue Fox said:
That was for all of them at once. Are you running in background mode with a very low priority?
Click to expand...

I am in background mode with 80% Priority... I just checked, at 80% HDTach is 227MBps sustaining, at 5%: 512MBps..... I array is getting used all this time, very lightly, maybe, 1000KBps, nothing hard at all......

When you did all those drives in a few hours, was it in background or foreground rebuild??
 
gjvrieze said:
I am in background mode with 80% Priority... I just checked, at 80% HDTach is 227MBps sustaining, at 5%: 512MBps..... I array is getting used all this time, very lightly, maybe, 1000KBps, nothing hard at all......

When you did all those drives in a few hours, was it in background or foreground rebuild??
Click to expand...
I did it in foreground mode. Since this is just a large media server for me, I could go without it overnight. I'm in the market for a 1280ML myself now as my 1130ML is full. This forum costs me a lot of money. :p
 
Blue Fox said:
I did it in foreground mode. Since this is just a large media server for me, I could go without it overnight. I'm in the market for a 1280ML myself now as my 1130ML is full. This forum costs me a lot of money. :p
Click to expand...

Ah, I bet foreground would be a lot faster, should try it with one drive, kind of scary, if it were to take 24hrs, we could not live without it that long!!!

I know, [h] costs me a ton too, and Ockie is the worst of it; so I bought a 24 port managed switch off of him this past week, then this morning, he is like "Oh, btw, take it from me... this switch is AMAZING with aggregate performance. Get yourself some quad nics " Now, I really want to grab one!!!
 
gjvrieze said:
Ah, I bet foreground would be a lot faster, should try it with one drive, kind of scary, if it were to take 24hrs, we could not live without it that long!!!

I know, [h] costs me a ton too, and Ockie is the worst of it; so I bought a 24 port managed switch off of him this past week, then this morning, he is like "Oh, btw, take it from me... this switch is AMAZING with aggregate performance. Get yourself some quad nics " Now, I really want to grab one!!!
Click to expand...
After checking the price on the quad gigabit NICs, it would be cheaper to just pick up a decent server board with 4 already built in, or just going for a 10gbit NIC. They're really cheap on ebay and the switches for them aren't too bad either. $40 for a NIC (link) and $400 for a 12 port switch (link). I've considered picking up 3 of the NICs and running a ring network (sorta).
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Blue Fox said:
After checking the price on the quad gigabit NICs, it would be cheaper to just pick up a decent server board with 4 already built in, or just going for a 10gbit NIC. They're really cheap on ebay and the switches for them aren't too bad either. $40 for a NIC (link) and $400 for a 12 port switch (link). I've considered picking up 3 of the NICs and running a ring network (sorta).
Click to expand...

God, you got my interest with Infiniband, will have to do some research!!
 
As an eBay Associate, HardForum may earn from qualifying purchases.
As an eBay Associate, HardForum may earn from qualifying purchases.
Blue Fox said:
Sorry, I didn't have a choice. :p
I only thought of Infiniband because my card uses Infiniband connectors instead of the usual multilane cables that the rest of the Areca cards use.
Edit: Also, there is a better version of that NIC with upgradable RAM: http://cgi.ebay.com/eBayISAPI.dll?ViewItem&item=300203358374
Click to expand...

I have no experience with Infiniband, how does it link with normal switches, if I just want to carray TCP/IP traffic, and not do say hdds or other cool things..??
 
As an eBay Associate, HardForum may earn from qualifying purchases.
gjvrieze said:
I have no experience with Infiniband, how does it link with normal switches, if I just want to carray TCP/IP traffic, and not do say hdds or other cool things..??
Click to expand...
I have no real experience with it either, but I know you can do IP over Infiniband (topology is a bit different...I think the same as fibre channel), so hopefully everything should work...maybe Ockie can chime in on this?
 
*** Self Proclaimed Storage King ***
Click to expand...

Self proclaimed? Yeah, right. I can't think of anyone who comes near to your [H]ardcore fileservers, Ockie.

And, God help me, I'm tempted to ignore my plans for a car and get me a fix^H^H^H bunch of HDDs myself. I've got the stacker 831 (empty), all I need is a mobo, cpu, raid card, drives, more cages... and I've wanted a fileserver ever since a mate built a terabyte one out of 160GB drives back when that was leet.

You're a terrible man, Ockie, and I envy you greatly.
 
You must log in or register to reply here.
Back
Top