starting to play with 10gb - Mellanox and need help

Why do people.jump into infiniband and mellanox etc.... for a first time.

Get some Intel XFSR cards in Ebay dirt cheap and direct wire an LC to LC fiber patch between the two machines and you dont need a switch.
 
Why do people.jump into infiniband and mellanox etc.... for a first time.

Get some Intel XFSR cards in Ebay dirt cheap and direct wire an LC to LC fiber patch between the two machines and you dont need a switch.

That is what he was already doing only with cheaper Mellanox cards.Only now he also bought a switch to increase the connectivity capabilities.
 
Ok. Post is growing large. Hard to absorb all the detail.

Lol yeah.. i kept it all bottled in here.

Yeah i got 8 cards...15 cables..so started screwin around. Now ive bought a switch and a conx-2 to try... i get it. Not sfp+....
Im happy with what i got. Need some treaks but im less than $75 in and 6 computers filling my switch...hah
 
Lol yeah.. i kept it all bottled in here.

Yeah i got 8 cards...15 cables..so started screwin around. Now ive bought a switch and a conx-2 to try... i get it. Not sfp+....
Im happy with what i got. Need some treaks but im less than $75 in and 6 computers filling my switch...hah
You got all of that for $75 ? Wow thats a good deal. Or was just the switch 75?

I run 10gig on fiber in my home.

I am not sure how Infiniband feels in windows but fiber feels snappier, quicker, as if it has much lower latency than copper. However that is all in my head because there is no humanly possible way to tell the difference and the difference is non-existant.

I am sure things feel smoother and snappier on your Infiniband setup. Did you get hardware capable of 40gbps or just the 10gbps older stuff?
 
You got all of that for $75 ? Wow thats a good deal. Or was just the switch 75?

I run 10gig on fiber in my home.

I am not sure how Infiniband feels in windows but fiber feels snappier, quicker, as if it has much lower latency than copper. However that is all in my head because there is no humanly possible way to tell the difference and the difference is non-existant.

I am sure things feel smoother and snappier on your Infiniband setup. Did you get hardware capable of 40gbps or just the 10gbps older stuff?

switch $50 best offer on ebay…
$25 box of cards and cables on craigslist...

they are the old cx4 ConnectX-1 InfiniBand/ethernet cards with the old cables but hey it works!!! HAH...
 
switch $50 best offer on ebay…
$25 box of cards and cables on craigslist...

they are the old cx4 ConnectX-1 InfiniBand/ethernet cards with the old cables but hey it works!!! HAH...

Yeah, I have done lots of Fiber installations for businesses over the years. They always do feel faster and snappier. I think it is the enhanced bandwidth that makes things feel smoother. Going from 1 to 10Gb is a huge leap in available bandwidth for machines to communicate across.

I am happy your infiniband is working so good for so low a price.
 
Yeah, I have done lots of Fiber installations for businesses over the years. They always do feel faster and snappier. I think it is the enhanced bandwidth that makes things feel smoother. Going from 1 to 10Gb is a huge leap in available bandwidth for machines to communicate across.

I am happy your InfiniBand ( running in ethernet mode ) is working so good for so low a price.

so I fixed your comment up there..hah

so I believe these are 20gb InfiniBand cards... OR 10gb ethernet cards. I do have an 8 port InfiniBand switch I may play with but for now you are correct...
going 1gb to 10gb is huge...

10gb

upload_2019-2-12_14-50-45.png



1gb - same file

upload_2019-2-12_15-12-23.png
 
The question remains.. How often and do you really need this speed ? I mean 1.1GB/s. Yes it looks very good.
Now many months after I got my 10gb connection, I find myself using it rarely and 200-250MB/s would have been totally sufficient. Backups, OS images.. all realtively rare situations. I mean.. looking at it in perspective, I just wanted to reasonably saturate the big HDDs and then some, which is 250-300MB/s maximum. And I use big HDDs for backups, images, some VMs etc. SSDs for storage is not worth it when I "touch" the backups etc. once in a while.
Of course I did my "part" messing around with some options etc. to see the link saturated at 9.5Gbit/s with iperf3 but then, all those benchmarking stuff etc... is mostly waste of time after a point.
 
The question remains.. How often and do you really need this speed ? I mean 1.1GB/s. Yes it looks very good.
Now many months after I got my 10gb connection, I find myself using it rarely and 200-250MB/s would have been totally sufficient. Backups, OS images.. all realtively rare situations. I mean.. looking at it in perspective, I just wanted to reasonably saturate the big HDDs and then some, which is 250-300MB/s maximum. And I use big HDDs for backups, images, some VMs etc. SSDs for storage is not worth it when I "touch" the backups etc. once in a while.
Of course I did my "part" messing around with some options etc. to see the link saturated at 9.5Gbit/s with iperf3 but then, all those benchmarking stuff etc... is mostly waste of time after a point.

I use 10gb every day ALOT.

In fact I have 2 TB of steam games on my NAS on 10gb and it loads nearly as fast as an SSD. I dont need $1200 2tb SSD in my machine to hold my games I just use an iscsi extent on freenas over 10gb and it's well in excess of 3000 iops off my spinning disk and 10gb has the bandwidth to exceed my spindles combined output in A ZFS raid 10 equivalent.
 
The question remains.. How often and do you really need this speed ? I mean 1.1GB/s. Yes it looks very good.
Now many months after I got my 10gb connection, I find myself using it rarely and 200-250MB/s would have been totally sufficient. Backups, OS images.. all realtively rare situations. I mean.. looking at it in perspective, I just wanted to reasonably saturate the big HDDs and then some, which is 250-300MB/s maximum. And I use big HDDs for backups, images, some VMs etc. SSDs for storage is not worth it when I "touch" the backups etc. once in a while.
Of course I did my "part" messing around with some options etc. to see the link saturated at 9.5Gbit/s with iperf3 but then, all those benchmarking stuff etc... is mostly waste of time after a point.



How often? Not... not really, but when I do... I WANT IT NOW!!!! LOL...

waste of time is a moo point when your screwin around learning.

I have been doing what I do since 2006... Altiris administrator ( SCCM Competitor that Symantec bought).
I am stale so I do stuff like this to learn. IF I had to interview, and I mentioned my test lab running esxi 6.7, vcenter, 10gb network, freenas, etc.. maybe it will help. at least show that at my current age (46) im still willing to learn. that is one of the big turn offs for older people looking for work. I know this as I have 2 friends that got turned down. While Illegal, they were told their commute would of been too far but someone that works in the company that recommended them, were told that their age maybe of question (they are 60... so what 6y then retire?)

so I get ya on this.. how much do I need....

still been a fun journey and hell I got to repurpose my Anten 900!!!! gonna post some pics there...if I can find that old thread



so I fixed your comment up there..hah

so I believe these are 20gb InfiniBand cards... OR 10gb ethernet cards. I do have an 8 port InfiniBand switch I may play with but for now you are correct...
going 1gb to 10gb is huge...

10gb

View attachment 141321


1gb - same file

View attachment 141324
I use 10gb every day ALOT.

In fact I have 2 TB of steam games on my NAS on 10gb and it loads nearly as fast as an SSD. I dont need $1200 2tb SSD in my machine to hold my games I just use an iscsi extent on freenas over 10gb and it's well in excess of 3000 iops off my spinning disk and 10gb has the bandwidth to exceed my spindles combined output in A ZFS raid 10 equivalent.


wow so you are saying map a drive, and install TO THAT MAPPED DRIVE?

I too am having this issue. my machine has 2x 500gb nvme and im out of space due to games.
 
I didn't mean to renounce the benefits of messing around and 'wasting' time on this :) . It was just a point of view, just my point of not being much interested in real life of those speeds. For someone that can make use of it - Ok, I find 1Gbps too slow and old now. But the upcoming 5Gbps would suffice to most, I believe, and then some. 2.5Gbps would be on the edge but 5 is what I wanted. I still think people like tangoseal are minority by a great margin. Sure, there are people who would even benefit 40Gbps at home if they force themselves into making up a scenario... but I learnt to do things effectively. If geting a 10-40gbps setup is cheaper than just equip your local PC with the necessary disks/ssds to run your games, then Ok. Still, most people don't keep terabytes and exabytes of games locally all the time :) . Most don't have the time to game such a lot. Exceptions exist of course.
Sometimes I cannot understand people with tens of 8TB disks NASes at home, which can be seen here in different sections. What do they possibly keep on these.. Movies, music, games.. 40-50 terabytes... come on.
After all, if I need many gigabytes daily for daily work/use, I prefer locally attached drives and NAS/servers at home only for storage/archiving/backup etc.
Bbut everyone is free to spend money according to their liking and using setups as they think they need :) .
 
totally understand on all of that and not complaining at all... were just chatting.. ill be gaming at 8pm when my friends are ready lol..

Tom Clancy's Division...


but I remember now why I wanted faster speeds... I tend to copy family movies/pics (DSLR), ISO around in test lab when I need to update things, and do my own DVR for TV.. don't pay the subscription as I do it myself.

now I recall it all... now having 10gb is nice and fast... yup.. copy files to the server, but my other dilemma was copying FROM the server to my machines... very slow... like 40MB/s …
well as we know.. spinning drives are slow... adding SSD Optimizer to my stablebit drive pool helps on the copying to server but not off of...
thus all the messin with all of this stuff... yeah its insane I agree..
and im always amazed at what people have setup too... 42u rack, full of nas servers, esxi servers doin all kinds of stuff... FOR HOME USE!!!! and agreed.. its their choice to play as they wish... I play lightly.. im a redneck in life, a redneck on pc components... why trash old if it works.
 
How often? Not... not really, but when I do... I WANT IT NOW!!!! LOL...

waste of time is a moo point when your screwin around learning.

I have been doing what I do since 2006... Altiris administrator ( SCCM Competitor that Symantec bought).
I am stale so I do stuff like this to learn. IF I had to interview, and I mentioned my test lab running esxi 6.7, vcenter, 10gb network, freenas, etc.. maybe it will help. at least show that at my current age (46) im still willing to learn. that is one of the big turn offs for older people looking for work. I know this as I have 2 friends that got turned down. While Illegal, they were told their commute would of been too far but someone that works in the company that recommended them, were told that their age maybe of question (they are 60... so what 6y then retire?)

so I get ya on this.. how much do I need....

still been a fun journey and hell I got to repurpose my Anten 900!!!! gonna post some pics there...if I can find that old thread







wow so you are saying map a drive, and install TO THAT MAPPED DRIVE?

I too am having this issue. my machine has 2x 500gb nvme and im out of space due to games.

No not mapped although for steam yhatvwould work perfectly. I recommend iscsi because it emulates a physical disk over the network.
 
Back
Top