Thoughts on building a high-performance NAS Setup for data crunching?

T

TheyDroppedMe

n00b
Joined
May 27, 2011
Messages
30
Hey, I'll get straight to the point. I have been a lurker on HardForum for a long time, but until now haven't really needed to post anything to find the answers I need. I work as the IT man for a small business and have been revamping our IT infrastructure to make expansion easier.

Right now, we use large, high-performance servers for our analytic work (we do econometric modeling and data mining). We want to move towards a more distributed approach, using many cheaper servers to do the work. One of the issues I face is the age-old performance vs development speed, and because of the kind of jobs we do, it pretty much puts a hadoop-style distributed architecture out of the picture.

What I am trying to accomplish is to abstract the storage from the analysis servers, so that every server can have access to the storage pool and jobs can easily be moved between servers. I am trying to figure out the best setup for this structure and am finding the information available is overwhelming. Here's what I'm looking for in a storage system:

-High Performance - the analysis servers will be constantly accessing data from the storage pool and then writing back the finished data sets. 500 MB/s is my goal here. The backend storage arrays are all fast enough to saturate this speed.
-Easily Expandable - adding new servers should be easy and cheap
-Low cost - obviously there are expensive commercial solutions, but being a small business, cost is a huge issue
-Easy to manage and reliable - I'll be the only person on-site to maintain the setup, so it should be fairly simple
-Initially, the network would host two or three storage servers, and 4-6 analysis servers, with the possibility of adding another 10 or so analysis servers in the future. All of the servers would need to be able to access all of the storage (although only one at a time would be working on a particular dataset)

With that fantasy checklist comes a reality check - I know that I won't be able to get everything (high performance, low cost), but perhaps you guys can help me to come up with a good compromise.

Obviously, the cheapest solution would be to set up a NFS share over a dedicated gigabit ethernet network. However, this would limit transfers to around 100 MB/s in my experience. I looked into link aggregation, but my experience with that has been frustrating at best, but that was several years ago. Perhaps there is a reliable method to bond 4 GBe ports together, connect them to a managed GBe switch, and call it a day?

I looked into 10GBe NIC's, and they are finally starting to approach the affordable price range, but switches are still obscenely expensive (unless I am just looking in the wrong place) - several thousand dollars for a 8-12 port switch. Same thing with Fibre Channel, plus I'm not aware of any file-level sharing protocol over FC.

I saw a few things mentioning using InfiniBand for this sort of model, and boy, used infiniband equipment is incredibly cheap, and fast. 4x DDR HCA's can be had for less than $100 each used, and 8-12 port switches in the $1000 range. But, I'm very hesitant to use the tech for this sort of thing, as it seems like no one does (and it's a new technology to me). Have any of you heard of using infiniband for this sort of usage scenario?

Do any of you guys have any recommendations, or links to point me in the right direction? I don't have any experience in designing shared storage beyond just NFS and GBe, so this stuff is all new to me. Also, this would probably be an interim solution, for the next 2-5 years, so it doesn't have to be incredibly expandable or future-proof (although that's always nice to have).

Thanks in advance for the help. I promise, I won't write any more posts this long in the future.

-Greg
 
What kind of data are you wanting to manipulate? Are these like 20 kilobyte files or 20 megabyte? 500MB/s is one single modern SSD on the larger sequential. You would have to RAID several SSD's to get that kind of performance on small writes. But SSD's are your best bet, but that brings us to the next question, what size storage size are you looking at?

Can't help you on the network...I myself was just looking at whats after gigabit yesterday because its very easy to saturate gigabit now-a-days with even the cheapest of harddrives. And ya the price was WAY outside of my budget for home :) I'll deal ~80-90MB/s for a while....
 
These datasets range from 50 MB to 10 GB, so the operations are largely sequential and the 12-spindle RAID-6 arrays we have can easily saturate a 6Gb/s SAS link, so we don't need to move to SSD's just yet (although I am trying to transition all of our storage to ZFS with a nice fat SSD for L2ARC and ZIL - part of the reason for wanting to abstract the storage). As for total storage size, I'd probably be moving about 50TB of storage (data plus onsite backups) to it initially by just moving the external RAID boxes we have to a storage server. But, our storage needs have been growing extremely fast, so I imagine it would grow pretty quickly from there.

-Greg
 
TheyDroppedMe said:
-High Performance - the analysis servers will be constantly accessing data from the storage pool and then writing back the finished data sets. 500 MB/s is my goal here. The backend storage arrays are all fast enough to saturate this speed.
Click to expand...

Are you saying you already have the storage arrays?


TBH, with no clue as to budget, current setup and infrastructure, or current/future storage requirements, all you'll get is abstract discussions about theoretical setups.

With old Infiniband eqpt, support could well be an issue, so do consider what you'd be getting into here. Cabling can also be problematic - and costly.


If you want 500MB/s performance from any one analysis server to any storage server, then TBH, you are probably just going to have to bite the bullet and cough up the cash - this is really beyond gigabit capability. Yes, you could try aggregating multiple links, but this can start to get out of hand with multiple machines.
 
A few questions before I jump in. First, what kind of math are you generally doing on the servers themselves (Floating point, Integer, etc)? Are they highly parallelized workflows? Second, what type of database does the data currently live in and what kind of box is it on? Regarding the "Fantasy Checklist" what is your actual budget range for the storage and number boxes themselves (I will leave out ongoing costs such as service, cooling, offsite backup etc for now)?
 
Well, budget and such is still not known. This is still very early in the planning stage, so I am trying to price out the various options and see if this even makes sense financially, Initially, we would just be moving the existing hardware around, but then would expand it as we require more capacity.

Current setup is as follows: two main Windows servers and one Linux analysis servers, each with its own external RAID array. There are also several onsite backup servers, and a couple off-site backup servers (but those will remain). Each array is around 10 TB of storage, with the Linux server having 20TB. We just underwent a large storage expansion, so we will not be adding any more storage (well, nothing major) in the immediate future (6-12 months). The network would be short-range, with all the servers in adjacent racks.

Yeah, I figured GBe was out of the picture, but I'm reading up on NFS over RDMA on infiniband and it seems that when you can get it work, the speed is phenomenal. The big hurdle I'm running into is the cost of the switches for 10GBe, as I'd like to be able to add machines for a small incremental cost.

Again, this may just be impossible with the current prices, but I figure someone else may have looked into this or even implemented something similar and can share their experiences.

Thanks
 
Without budget numbers it is going to be very hard to make suggestions. If you are interested in moving current hardware around, could you post a more detailed inventory of your current boxes (processors(#,type)disks(#,types)RAM, HBA Controllers (Disk and Comm). Infiniband is great for HPC, we have a lot of Mellanox QDR equipment. 10GB IB equipment can be found used for not a lot of money, but that might not be acceptable for your application (RDMA would be more applicable rather than IP over IB)
Once we get more of an idead of what you have to start with, we can suggest how much of it you would be able to repurpose towards the new infrastructure.
 
mwroobel -- the workload is highly varied, ranging from mostly integer, to heavily floating-point. Some of the workloads are easily parallelized, but many are not. Everything is run through Stata at the moment, but we are trying to move towards using more R and custom applications. Our database is in PostgreSQL on Linux (192GB ram, 2x Xeon E5-2430's).

This is more of a planning exercise for the moment, as our last round of expansion is serving us well, but we anticipate our needs continuing to grow at a near exponential rate, so I am trying to get ahead of the expansion and design our IT infrastructure for easy expansion moving forward. At this point, it's all very theoretical and I'm just trying to get an idea of how I would design all this infrastructure and how much each option would cost - budget is still not decided on. I know it's hard to talk about without all the specifics, but I just don't have them yet as it's all still very early on.

Thanks
 
Depending on how much is fp and your needs for precision types, you might want to look to GPGPU or FPGA based boards to accelerate a lot of what you are doing. Basically you should break down your infrastructure cost expectations as follows:

Physical Plant Costs: Square footage, Racks, Increased cooling and ventilation costs, increased electrical costs and service provisioning, UPS needs
Compute: Costs for Server(s), RAM and local drives.
Storage: Costs for SAN or NAS infrastructure
Comm: Costs for your networking infrastructure
Backup: Costs for onsite and offsite data backup
DR(For this project): What happens when your office burns down, do you have a hotsite/coldsite?

Here is where it gets fun.... You have all these as slices in a pie, which slices do you make bigger and which ones get smaller.
 
Ok well here's as much of the inventory as I can remember off the top of my head:

Windows Servers:
Each with 2x 4c or 6c processors
32 GB ram
RAID 6 arrays with 2tb RE4's in Norco DS-24e boxes
Adaptec 5445 RAID cards

Linux Server:
2x Xeon E5-2430 Cpus, (6c/12t)
192GB Ram
6TB RAID 10 array for PostgreSQL (2TB RE4's)
16TB RAID 6 Array with 2TB RE4's in a Norco DS-24e box
Adaptec 6445 with AFM for External array
Dell PERC h710 for internal RAID 10 Array

Backup Servers (Linux - don't remember the specifics since I didn't build them):
6c CPU
16GB Ram?
Adaptec 6445
Norco DS-24e External boxes w/ 10-20 2TB RE4-GP drives

Future Analysis servers would be something like this:
1x Xeon E5 6c/12t CPU
32GB Ram
Small boot drive
2TB Working/scratch drive

Proposed Storage Servers would have a similar configuration, with multiple RAID cards and external RAID boxes on each machine. Currently, everything is connected via GBe. I imagine moving all the external RAID boxes to a central storage server or two, and then expanding by just adding more Norco Boxes and drives.

The 10g Infiniband stuff is definitely cheap and very interesting as it would result in an easy upgrade path to future needs without having to switch to a different technology - just upgrade the HCA's and Switch(es) to faster units. But, I have heard of lots of trouble with NFS and IP over InfiniBand, especially under Linux. However, those articles/blogs are several years old, so perhaps the software has improved since then...

The cheaper the setup is, the easier it will be to justify to my boss, but we are willing to spend money when necessary.
 
Last edited:
mwroobel -- While I would absolutely LOVE to move to doing stuff in Hadoop, or with some CUDA/OpenCL, honestly, the varied projects we work on don't make sense to code at that low of a level. We are a consulting firm, so rapid development and turnaround is paramount, especially since a lot of what we do is one-off analysis. Additionally, the analysts are not really comfortable with anything outside of something like Stata or R. Have you heard of any statistical packages that allow distributed processing or GPGPU acceleration? That would be interesting.

-Greg
 
IP over IB can be a dance, it all depends on who the partners are as to how well it performs. For our particular application the FDR Mellanox equipment (We have standardized for this application with the ICIB and C-X3 HBAs) is faster than anything out there, and about the same price as the 40GbE stuff out there. We tested RDMA over FDR IB vs RDMA over Converged Ethernet and found significant benefits in the former.
 
Last edited:
mwroobel - Any idea why InfiniBand for storage/high-speed networking has not caught on? It seems like the perfect interconnect - insanely fast, low-latency, inexpensive, and the ability to use for more than just storage/networking. I would probably be looking at used 10g Infiniband equipment, since that is more than enough bandwidth for our usage and the savings from getting used equipment would be worth it. In your experience, is NFS over RDMA infiniband stable/reliable/easy to setup and maintain? Is it recommended to stay with one manufacturer for all IB products, or are there certain models that seem to be extremely reliable/compatible?

If infiniband would work for this scenario, it would definitely be worth it with an equipment cost for infiniband components coming in under $2k (used)... And it sure would be nice to be able to scale up to 40g links or more vs using 10GBe (still more expensive for some reason).

I'm off to bed for the night, but I'll be researching this stuff tomorrow. Thanks again for your input!

-Greg
 
TheyDroppedMe said:
mwroobel -- While I would absolutely LOVE to move to doing stuff in Hadoop, or with some CUDA/OpenCL, honestly, the varied projects we work on don't make sense to code at that low of a level. We are a consulting firm, so rapid development and turnaround is paramount, especially since a lot of what we do is one-off analysis. Additionally, the analysts are not really comfortable with anything outside of something like Stata or R. Have you heard of any statistical packages that allow distributed processing or GPGPU acceleration? That would be interesting.

-Greg
Click to expand...

While R is a very powerful language, I have heard of people not making the jump because of the steep learning curve and just back to what they currently use. MATLAB has a cuda plugin for GPGPU with their statistical toolbox. SPSS is great if you want to buy a Z Series mainframe :) I don't remember exactly, (personally I haven't been exposed to Stata in a few years) but I though Stata was supposed to have added GPU (OpenCL/Cuda) support once suitable double precision math was fully supported. My memory is fuzzy on that so you would certainly know more than I about that.
 
Another quick question - what do y'all use to draw out those nice network maps? I've just been drawing mine in Vizio, but that takes forever and is generally a PITA. Are there any nice web-based tools or freeware to draw/design netoworks/server layouts?

I'd like to draw up some designs to help you guys get a better idea for how this would look.

-mwroobel - Yeah, we've been doing our own initial testing with R, and while it is extremely powerful and produces beautiful graphs, it is definitely a little bit of a change from Stata, but we think it's worth the shift. SPSS is out of the question as it doesn't support some of the things we need (nor do I want to deal with a Z mainframe haha). I have not heard anything about Stata supporting GPGPU, but I will look into it (we're running the latest release, Stata 12 MP).
 
I have no input, but I want just say I am following this with personal interested. I actually have a Degree in Economics, And I have used Sata and Spss, but I have never tried R.

I got a buddy who ask me about how to help build a system to gather all the data, and i told him it was beyond me, so this might be a good read for him or his IT guy. haha.

:)
 
midknight - How did you like Stata compared to SPSS? We kinda ruled it out for not meeting our specific needs, but I love hearing about other people's experience. If you're the kind of person (like me) that likes a little lower-level language, then you'd probably like R. It's more like a general purpose computing language than Stata/SAS. I've got a degree in economics too - do you work in the industry? As for your friend, feel free to have him message me if he wants some input as we just went through a large round of infrastructure upgrades and it's all still fresh in my mind.

--------------------------

Everyone else - does anyone have any experience in link aggregation of Gigabit ethernet and actually getting more than 1Gb/S throughput for a single file transfer? Also, any experiences with infiniband or 10GBe is always welcome...

Thanks
 
TheyDroppedMe said:
..
..

Everyone else - does anyone have any experience in link aggregation of Gigabit ethernet and actually getting more than 1Gb/S throughput for a single file transfer? Also, any experiences with infiniband or 10GBe is always welcome...

Thanks
Click to expand...

Have a little.
Intel S5000SL MB with dual NICs,win2003 OS and Netgear GSM 7248 switch .
Pros - 15-20 % bump in transfer rates .
Cons - accidentally the connection becomes broken - everything seems right but there was no ping.To get connection back -disable/enable of NICs or just reboot.Probably driver related. Terrible. No more NICs teaming/aggregation for me!
 
midnkight said:
I have no input, but I want just say I am following this with personal interested. I actually have a Degree in Economics, And I have used Sata and Spss, but I have never tried R.

I got a buddy who ask me about how to help build a system to gather all the data, and i told him it was beyond me, so this might be a good read for him or his IT guy. haha.

:)
Click to expand...


A neat background read:

What Makes Big Data Different?
Big data refers to large datasets that are challenging to store, search, share, visualize, and analyze.
At first glance, the orders of magnitude outstrip conventional data processing and the largest of
data warehouses. For example, an airline jet collects 10 terabytes of sensor data for every 30
minutes of flying time. Compare that with conventional high performance computing where
New York Stock Exchange collects 1 terabyte of structured trading data per day. Compare again
to a conventional structured corporate data warehouse that is sized in terabytes and petabytes.
Big Data is sized in peta-, exa-, and soon perhaps, zetta-bytes! And, it’s not just about volume,
the approach to analysis contends with data content and structure that cannot be anticipated or
predicted. These analytics and the science behind them filter low value or low-density data to
reveal high value or high-density data. As a result, new and often proprietary analytical
techniques are required. Big Data has a broad array of interesting architecture challenges.
It is often said that data volume, velocity, and variety define Big Data, but the unique
characteristic of Big Data is the manner in which the value is discovered. Big Data is unlike
conventional business intelligence, where the simple summing of a known value reveals a result,
such as order sales becoming year-to-date sales. With Big Data, the value is discovered through a
refining modeling process: make a hypothesis, create statistical, visual, or semantic models,
validate, then make a new hypothesis. It either takes a person interpreting visualizations or
making interactive knowledge-based queries, or by developing ‘machine learning’ adaptive
algorithms that can discover meaning. And in the end, the algorithm may be short-lived.
The growth of big data is a result of the increasing channels and variety of data in today’s world.
Some of the new data sources are user-generated content through social media, web and software
logs, cameras, information-sensing mobile devices, aerial sensory technologies, genomics, and
medical records.
Companies have realized that there is competitive advantage in this information and that now is
the time to put this data to work.
Click to expand...


http://www.oracle.com/technetwork/t...-data-guide-1522052.pdf?ssSourceSiteId=ocomen



More:

http://www.oracle.com/us/technologies/big-data/index.html



inb4 GPL zombie horde: Of course it's an infomercial. But Oracle is the class of this league right now. Jus sayin.;)
 
Last edited:
TheyDroppedMe said:
Another quick question - what do y'all use to draw out those nice network maps? I've just been drawing mine in Vizio, but that takes forever and is generally a PITA. Are there any nice web-based tools or freeware to draw/design netoworks/server layouts?

I'd like to draw up some designs to help you guys get a better idea for how this would look.

-mwroobel - Yeah, we've been doing our own initial testing with R, and while it is extremely powerful and produces beautiful graphs, it is definitely a little bit of a change from Stata, but we think it's worth the shift. SPSS is out of the question as it doesn't support some of the things we need (nor do I want to deal with a Z mainframe haha). I have not heard anything about Stata supporting GPGPU, but I will look into it (we're running the latest release, Stata 12 MP).
Click to expand...

Visio is decent/basic. Gliffy isn't bad. Masshandra is great. Lansurveyor is pretty good too.

I actually remember sometime back some management drone brought an IBM rep to a meeting to explain why a Z-Series mainframe would be better for the financial app they wanted than the client/server build we were doing (And was, conservatively 20x the amount we had budgeted/built the system)
 
I would say infiniband hasn't caught on due to many factors.

The whole infiniband thing is not compatable with anything else, won't talk to disks, or networks, so it's isolated.

It is only good for short distances, I think it maxs out at 30m for 10g and something like 10m for 40g?

People generally look at what new stuff they are buying, and new infiniband stuff being priced where it is, and not being compatable with anything, using fc or ethernet is more attractive for most, then doubling up.
 
Thanks for all the replies - I am in love with MaSSHandra, btw!

Dedobot - I'm interested to hear more about this. Were you actually seeing increased throughput for a single file transfer from one client to one server? Last time I checked (a while ago), it was my understanding that the teaming was only really useful for multiple connections and failover, but maybe they have enabled true bonding (that allows double bandwidth for single connections to a similarly teamed server with a managed switch). It does sound like a bit of a pain, though. Can anyone clarify this?

Paret0 - Thanks for the links! I love big data stuff, and have been researching it for a while. We're kinda in this weird middle ground where our data is not big enough nor standard enough to justify the development for most of the approaches I have seen (it doesn't make sense to write a custom app for a one-off project when the speedup, while great, is the difference between 1 and 4 days of computation time). However, if we keep growing at the current pace, we'll probably get into that territory pretty soon - we're starting to do some stuff with web analytic data that results in MASSIVE files (don't even get me started on the 139 GIGABYTE TEXT FILE someone sent us... had to fire up a linux VM so I could script with my trusty sed/tr/cut/head/etc utils). I would be VERY interested in hearing about anyone's experiences with "big data" analytics.


Here's an update on the stuff I have come across in my researching this setup:
-Windows Server 2008 R2 does not support NFS/RDMA at the moment, but Server 2012 does support SMB Direct (SMB over RDMA) and people are getting insane speeds using InfiniBand (Saturating a 40Gb/s Infiniband connection). That probably puts a halt on the idea of using infiniband for this in the near future, although I will probably test it out next year.
Side Note - Server 2012 has so many features I want, but everyone here HATES the metro interface (we RDP into the analytics servers to do 99% of our work). Why did Microsoft have to force this on Server? We'll NEVER use touch for business purposes and when I set up a VM with the preview, well the reaction was OVERWHELMINGLY negative. I understand it on consumer Windows 8 since they want to unify the desktop/phone/tablet, but who in their right mind thought it would be OK to force it on servers too? It'd probably be ok if they let the start menu hacks continue, but they had to go and remove them. We've got less than 20 employees using the servers, and overseeing the change would be a nightmare for me - I can't imagine what it would be like somewhere with a few thousand people RDP'ing into one...
/rant off

-Running SMB over InfiniBand using plain IPoIB can be tuned to get around 200MB/s transfers, but the CPU usage would be high. Plain SMB over 10GB ethernet is faster, as the cards can offload most of the TCP/IP calculations, but still nowhere near a RDMA solution. If the switches weren't so damn expensive, I might just try that. I wonder if you could set up a simple ring network with 10GBe...? I still may try to get two cards to test between our two high-bandwidth servers.

-The market is sorely lacking a good solution to share network file systems at anything faster than gigabit ethernet. iSCSI and SDP (basically iSCSI over RDMA, but not exactly) work well from Linux-Linux and even Linux-Windows, but anything else is damn near impossible at the moment. I put out a few emails to some vendors, so perhaps there is a working solution, but I'm not holding my breath.

-Linux (debian, ubuntu) is awesome and Windows Server is a pain in my ass :) Windows is great for simplicity and user interaction, but as soon as you start to get into and compute scenarios or need to work with large files, file permissions, or ultra-fast networking, it becomes a huge pain (in my experience). I'm sure it is great for many many people, but in my experience, I spend 10x as much time fixing the windows servers as I do the Linux ones. Hell, we have a few ubuntu machines that I log into only to shutdown before doing infrastructure upgrades/maintenance.

If any of you have any experience with using anything faster than gigabit for shared storage between linux-linux and linux-windows, please share! Everything I am seeing is about iSCSI-type scenarios (and indeed, that stuff is possible *fairly* easily with infiniband now), but that doesn't help when you have many machines that need to access the same data.

I'll keep this tread updated as I continue my research and when I finally get to the experimentation stage (may be several months out though, since it looks like InfiniBand Linux-Windows won't work for NFS/RDMA).

Thanks again. Stay Frosty.
 
mellanox cards have offload too. what MTU are you using when testing and have you run multiple iperf tests?

btw CPU usage is going to be high regardless of offload.
 
TheyDroppedMe said:
Thanks for all the replies - I am in love with MaSSHandra, btw!

Last time I checked (a while ago), it was my understanding that the teaming was only really useful for multiple connections and failover, but maybe they have enabled true bonding (that allows double bandwidth for single connections to a similarly teamed server with a managed switch). It does sound like a bit of a pain, though. Can anyone clarify this?
Click to expand...

LACP is the de facto L2 enet protocol these days. Harder and harder to get around it. You know this.
FreeBSD supports roundrobin ethernet link aggregations. And though it violates RFCs, my Dell and Brocade switches don't mind it at all.
Have I managed to reliably exceed single wire Gbit speed with a roundrobin aggregation between 2 BSD hosts? No, but I haven't even run iperf between those hosts, so I just don't know what's possible.




Paret0 - Thanks for the links! I love big data stuff, and have been researching it for a while. We're kinda in this weird middle ground where our data is not big enough nor standard enough to justify the development for most of the approaches I have seen (it doesn't make sense to write a custom app for a one-off project when the speedup, while great, is the difference between 1 and 4 days of computation time). However, if we keep growing at the current pace, we'll probably get into that territory pretty soon - we're starting to do some stuff with web analytic data that results in MASSIVE files (don't even get me started on the 139 GIGABYTE TEXT FILE someone sent us... had to fire up a linux VM so I could script with my trusty sed/tr/cut/head/etc utils). I would be VERY interested in hearing about anyone's experiences with "big data" analytics.
Click to expand...


You're all over this. ;)

Q:

Y no ZFS? OOTB Dedup and several compression algos can shrink the shit out of datasets. Block level volume manager. Much justice.
 
TheyDroppedMe said:
These datasets range from 50 MB to 10 GB,
Click to expand...

These sizes are relatively trivial. How would working on the model of copying from master storage to processing server, process, then writing back affect your bandwidth requirements?
 
TheyDroppedMe said:
The market is sorely lacking a good solution to share network file systems at anything faster than gigabit ethernet.
Click to expand...

To be fair, what you really mean here is that the market is sorely lacking in a cheap solution to share network file systems at anything faster than gigabit ethernet.

The solution does exist - 10Gb ethernet - it's just that the infrastructure may be too expensive at the moment.

I take it you want faster performance in order to increase production - can you not use that as justification for the initial investment in 10Gb ethernet - if you pitch it as something your company can't afford not to do, that usually makes the money men sit up and listen (they don't like seeing money walk out the door :) )
 
madrebel - I have not actually picked up any equipment yet to test with - everything I have found is just based on tests from other people and articles/forum posts. I do plan on getting some sort of test equipment to play around with in the next month or two, though.

paret0 - Yeah, I don't have much faith that LACP would really make much of a difference in my scenario, especially between Linux-Windows. I am in love with ZFS, and trying to move more of my storage over to it. The data consistency/error-checking is awesome, and snapshots make incremental backups a breeze. However, in my experience, dedupe and the built-in compression have a large performance hit (as madrabel said). I may experiement with it on my backup servers, as performance it not as big of a deal on those machines, but I've found that just manually gzipping the files results in a wonderful 10:1 to 20:1 compression ratio without the downsides of a compressed filesystem. But back to the point, moving to ZFS is one of the goals/benefits of this separated storage model.

Quartz-1 - Yes, the sizes are relatively trivial, but our datasets have been growing in size as we have started doing more web stuff, so it's a future-proofing solution. Additionally, I have found that the analysts working on stuff are very resistant to change (I have been trying to get them used to Linux and trying to enforce good coding practices, but that has been a huge headache in and of itself), and so the IT goals here have been tailored to maximizing performance while still allowing the analysts to maintain productivity at their current rate. There's a bunch that even I do differently, but when dealing with other people and a business, the ideal solutions aren't always they ones that get implemented...

Billy_nnn - While I do mean a cheap solution, I still feel that even if you were to spend the money, you're still left with a lot of holes. 10GB ethernet is not yet settled on a "standard" interface (I know 10gbe is a standardized protocol, but they support many different interfaces) - you have to choose between Cat6 ethernet, fibre, and even a few CX4-based solutions. Not to mention, there is a massive hole in switches with more than 4 10GBe ports (and they're used for uplinks) - the only ones on the market are either massively expensive (I consider it a hole when the cost of a switch is more than my most expensive server), or not really designed to be a 10GBe switch (modular switches with 10GBe "uplink" modules which you can just populate the switch with entirely).

Also, even with 10GBe, there is no way to really saturate the pipe between Windows-Linux with SMB/NFS at the moment. Windows Server 2012 will introduce SMB Direct (RDMA), but when Samba on Linux will support it is unknown. Additionally, having to upgrade to a brand new server OS with massive and unpopular interface changes is not really considered a good solution to me. Perhaps SMB 3.0 and SMB direct will provide this universal protocol, but right now, it's still lacking when you want to share network directories at > 1Gb/s. If you're OK with just sharing bare drives with one computer, you're correct, the market offers many good solutions (even if they aren't cheap), but even 10GBe doesn't solve my problems. Again, there could be a solution out there somewhere, but it's not easy to find, universal, and well-supported, no matter how much it costs.

But I do agree with your idea of posing it as something we can't afford not to do :) Especially since this sort of change in IT infrastructure is always easier to make early on when the number of changes is still manageable. I hate having to justify IT expenses, although my boss now is pretty good about letting me buy the things I need.

Thanks y'all
 
You could also scale out, rather than up (which has been the focus so far).

I have no idea if your workload would make use of it but you could look at parallel file systems (GPFS, Lustre, Gluster etc etc).

We routinely push multiple gigabytes per second of transfer speeds down to storage platforms using GPFS.
 
Last edited:
TheyDroppedMe said:
I saw a few things mentioning using InfiniBand for this sort of model, and boy, used infiniband equipment is incredibly cheap, and fast. 4x DDR HCA's can be had for less than $100 each used, and 8-12 port switches in the $1000 range. But, I'm very hesitant to use the tech for this sort of thing, as it seems like no one does (and it's a new technology to me). Have any of you heard of using infiniband for this sort of usage scenario?
Click to expand...

Actually, IB has really seen a nice uptick in usage recently. 10GigE adoption is going decently well but next generation stuff is still a bit away. Conversely, 40-56gbps IB is readily available.

If you go the Infiniband route, I would strongly suggest spending $500 on a test setup and playing with it. Topspin 120's are CHEAP (<$250). But the whole IB experience for me has always been, good performance, less than super easy to setup. Here is a quick Ubuntu Infiniband guide one of STH's users wrote up for IB and Ubuntu. That was using the $35 dual port DDR adapters.
 
pjkenned said:
Actually, IB has really seen a nice uptick in usage recently. 10GigE adoption is going decently well but next generation stuff is still a bit away. Conversely, 40-56gbps IB is readily available.

If you go the Infiniband route, I would strongly suggest spending $500 on a test setup and playing with it. Topspin 120's are CHEAP (<$250). But the whole IB experience for me has always been, good performance, less than super easy to setup. Here is a quick Ubuntu Infiniband guide one of STH's users wrote up for IB and Ubuntu. That was using the $35 dual port DDR adapters.
Click to expand...

We have now standardized (for where we are using IB) on the Mellanox FDR 56Gb hardware. For what it does nothing else can touch it. That speed and reliability does not come cheap though. I fully agree with pjkenned that if you are looking towards IB, buy yourself some cheap IB10Gb stuff (you can find deals on eBay and a number of the server pull resellers) and setup a few lab machines and see how it works for your particular implementation. If you get to the point you want to move up to the big stuff, call Mellanox (Or whatever company you decide to deal with, we chose Mellanox) and you can negotiate an onsite demo/loan (depending on how comfortable they are based on the financials they will demand) of some cards and a switch.
 
TheyDroppedMe said:
I consider it a hole when the cost of a switch is more than my most expensive server)
Click to expand...

I think you are doing poor bookkeeping.

The cost of a solution to your problem is not just the cost of a server or just the cost of moving the data. The cost is the entire system.

Either the problem you are trying to solve is worth the cost of the system (regardless of the relative cost of the components) or it is not.
 
TheyDroppedMe said:
paret0 - Yeah, I don't have much faith that LACP would really make much of a difference in my scenario, especially between Linux-Windows. I am in love with ZFS, and trying to move more of my storage over to it. The data consistency/error-checking is awesome, and snapshots make incremental backups a breeze. However, in my experience, dedupe and the built-in compression have a large performance hit (as madrabel said). I may experiement with it on my backup servers, as performance it not as big of a deal on those machines, but I've found that just manually gzipping the files results in a wonderful 10:1 to 20:1 compression ratio without the downsides of a compressed filesystem. But back to the point, moving to ZFS is one of the goals/benefits of this separated storage model.
Click to expand...

Yep, it takes ~30GB RAM/TB to dedup and minimize the hit. So one should budget what filesystems one deduplicates according to reward. :D

http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe


But it is quite worth it in some cases...
Given that atrocious 136GB text file, assuming it was in only 3 or 4 alphabets Roman, Kanji, Cyrillc, you could conservatively expect >10:1 dedup ratio. Now you're storing, schlepping, manipulating a 14GB text file, which ain't near as horrible as before.

Real world, English text files: 400++:1 dedup ratios. I'll take a 50% hit on transport speeds for that to the bank. Furthermore, what's left after dedup may well be compressible.
Now let's upload a 1/400-size dataset to a hellacore ZFS cloud infrastructure so we can run our R or S-PLUS logic in .12 seconds processor time, yo? Oh? Already done? Good. We can bill sooner. :D
 
Last edited:
GeorgeHR said:
The cost of a solution to your problem is not just the cost of a server or just the cost of moving the data. The cost is the entire system.

Either the problem you are trying to solve is worth the cost of the system (regardless of the relative cost of the components) or it is not.
Click to expand...


Exactly. That's why vertically integrated solutions sell so well at whatever price.


Much can be, indeed needs to be, learned from how the better engineers solve the same problems.
 
Last edited:
mwroobel said:
If you get to the point you want to move up to the big stuff, call Mellanox (Or whatever company you decide to deal with, we chose Mellanox) and you can negotiate an onsite demo/loan (depending on how comfortable they are based on the financials they will demand) of some cards and a switch.
Click to expand...

Well since Intel bought QLogic's Infiniband IP the list is a bit shorter.
 
mwroobel said:
I actually remember sometime back some management drone brought an IBM rep to a meeting to explain why a Z-Series mainframe would be better for the financial app they wanted than the client/server build we were doing (And was, conservatively 20x the amount we had budgeted/built the system)
Click to expand...
Great. IBM always does that. The lock in effect is enormous with Mainframes. Once mainframe, you are stuck.

On top of the extremely high prices, the Mainframe cpus are very slow. The new z196 at 5.26 GHz is something like half the performance of a high end Intel Xeon cpu. There is a reason you do number crunching on a cluster of cheap x86 pcs, and not buy one/few Mainframes.

As you already have figured it out: avoid legacy Mainframes at all costs.
 
how exactly did you arrive at that performance guestimate? those mainframes are anything but slow. stupid expensive, absolutely. however for what they do they do an extremely good job. not that you can't build something superior for less, you can, however those bigiron boxes do some really impressive shit.
 
You must log in or register to reply here.
Back
Top