stevebaynet
Limp Gawd
- Joined
- Nov 9, 2011
- Messages
- 204
I have decided to start this thread and document my build because i have seen others do it and have gotten a wealth of information from it. I once read an article with a quote "the reason open source doesn't work is because 99% take and only 1% give back". I may not be able to contribute code to ZFS or illumos, but i can at least document my trials and tribulations and put them up somewhere where other people can see, benefit and even debate the merits of it.
The end result will be a Nexenta based HA cluster. I really like what i see with OI+napp-it but i need HA and most importantly for me, i need someone to call at 2am when something goes wrong, lol (this is for work).
Like most shops, we have (and have had) various legacy and proprietary based solutions in our racks. The frustration for me has mainly centered around price VS the usable lifetime of the product.
A colleague suggested nexenta after the last vmworld where nexenta participated in the "hands on lab". This was actually my first exposure to ZFS and i was highly impressed with the features i saw. Namely:
- commodity hardware
- no vendor lock in
- copy on write file system with snapshots that dont drag down your perf
The articles and data that came out of the vmware HOL at vmworld showed me that ZFS could scale and really keep up with the legacy systems. Also, i really liked the hybrid storage model teaming SSD's with spindle disks.
At the time, there were (for me) three standout options:
- OpenSolaris/OpenIndiana + Napp-IT
- TrueNAS (commercial version of FreeNAS)
- Nexenta
I liked the free OI+Napp-IT idea, but for commercial production stuff, that made me a little nervous (we intend to use this for one of our backup SAN's tho)
I quoted out solutions with both TrueNAS and Nexenta based setups to see how things would come out price wise. But mainly, i did my due diligence looking up reviews, builds, feedback, problems... basically, anything i could find.
For TrueNAS i pretty much came up with nothing, this scared me. i did however read alot about FreeNAS and their implementation of ZFS. I assume TrueNAS was new at the time so this is not a knock on them. + i think FreeNAS is a great piece of software. Ultimately, the lack of a track record with vmware based setups lead me elsewhere. For home or SOHO stuff i would definitely give it a second look. I assume with BSD's improving implementation of ZFS they will warrant a second look down the road. (love BSD)
For Nexenta, i liked that they were a certified partner with vmware, i liked that i was able to google out a good amount of information about them and their products. I did have some concerns tho, mainly:
- Nexenta is based on open solaris, which is affectively dead thanks to Oracle
- Although Nexenta based solutions are at release 3, i still saw in reviews and blog postings that some real issues still persisted that concerned me.
- The approved list of hardware seemed limited
- As a company that provides both a free community option along side a commercial one, i looked at the community forums to see what kind of support was there and it looked very limited to me.
Ultimately, after lots of thought as well as replying to nearly every blog or post i could find out there about ZFS/Nexenta, we decided to go the Nexenta route.
My logic, based on the above concerns:
- Nexenta ver 4 will be based on illumos (well, illumian really, but you get it)
It wasn't until i watched a video linked on this forum that i really understood the history of solaris and why illumos is the future for ZFS.
http://www.youtube.com/watch?v=-zRN7XLCRhc
(if you have an hour to spare and your interested in this kinda stuff, i recommend watching it)
- Nexenta stability at ver 3: My main concern was about problems i had seen online while doing my searches. Mainly, poor performance when a drive is starting to die (but hasnt died yet) and not being alerted to this via the Nexenta software (but being able to see the problem in the OS). I found some OS ways around this when it comes to alerting and while ultimately, i dont like it, the only way to see it get fixed is to join in and suggest workarounds or file bug reports to see it fixed in the next update (and pester if its not). I also noticed that alot of community issues had to do with hacking hardware to work with the OS. This concerned me less since i would be going the route of using only hardware tested with this system.
- Nexenta HSL (hardware supported list): is indeed limited, but i saw some flexibility here when comparing my self built quoted solutions against pre-built vendor created ones (that had similar items to the HSL, but not always exact). bottom line, work with a nexenta sales engineer and they can help you work out a solution that will hit the target goal (and in most cases price).
- Community support: the nexentastor.org forum is built on horrid forum software (i hear they are replacing it) and at first i did not see much action there. but over the past few months i have seen it improve, more help coming from community members as well as more nexenta employees starting to show up on the boards. This was encouraging for me. It also led me to joining myself, and while i am certainly no nexenta or ZFS guru, having a second pair of eyes looking at logical problems seems to help regardless. I see less posts going un-answered now.
Holy crap, this is turning into a novel, next up: Why we decided to go the self built route VS a nexenta/ZFS hardware partner.
(and after that, pictures of the build and the install)
I'd like future posts to include the testing phases while we put this thing through its pases. I'll have lots of questions on this forum i assume as i have never used iometer or bonnie. lol. so bear with me.
The end result will be a Nexenta based HA cluster. I really like what i see with OI+napp-it but i need HA and most importantly for me, i need someone to call at 2am when something goes wrong, lol (this is for work).
Like most shops, we have (and have had) various legacy and proprietary based solutions in our racks. The frustration for me has mainly centered around price VS the usable lifetime of the product.
A colleague suggested nexenta after the last vmworld where nexenta participated in the "hands on lab". This was actually my first exposure to ZFS and i was highly impressed with the features i saw. Namely:
- commodity hardware
- no vendor lock in
- copy on write file system with snapshots that dont drag down your perf
The articles and data that came out of the vmware HOL at vmworld showed me that ZFS could scale and really keep up with the legacy systems. Also, i really liked the hybrid storage model teaming SSD's with spindle disks.
At the time, there were (for me) three standout options:
- OpenSolaris/OpenIndiana + Napp-IT
- TrueNAS (commercial version of FreeNAS)
- Nexenta
I liked the free OI+Napp-IT idea, but for commercial production stuff, that made me a little nervous (we intend to use this for one of our backup SAN's tho)
I quoted out solutions with both TrueNAS and Nexenta based setups to see how things would come out price wise. But mainly, i did my due diligence looking up reviews, builds, feedback, problems... basically, anything i could find.
For TrueNAS i pretty much came up with nothing, this scared me. i did however read alot about FreeNAS and their implementation of ZFS. I assume TrueNAS was new at the time so this is not a knock on them. + i think FreeNAS is a great piece of software. Ultimately, the lack of a track record with vmware based setups lead me elsewhere. For home or SOHO stuff i would definitely give it a second look. I assume with BSD's improving implementation of ZFS they will warrant a second look down the road. (love BSD)
For Nexenta, i liked that they were a certified partner with vmware, i liked that i was able to google out a good amount of information about them and their products. I did have some concerns tho, mainly:
- Nexenta is based on open solaris, which is affectively dead thanks to Oracle
- Although Nexenta based solutions are at release 3, i still saw in reviews and blog postings that some real issues still persisted that concerned me.
- The approved list of hardware seemed limited
- As a company that provides both a free community option along side a commercial one, i looked at the community forums to see what kind of support was there and it looked very limited to me.
Ultimately, after lots of thought as well as replying to nearly every blog or post i could find out there about ZFS/Nexenta, we decided to go the Nexenta route.
My logic, based on the above concerns:
- Nexenta ver 4 will be based on illumos (well, illumian really, but you get it)
It wasn't until i watched a video linked on this forum that i really understood the history of solaris and why illumos is the future for ZFS.
http://www.youtube.com/watch?v=-zRN7XLCRhc
(if you have an hour to spare and your interested in this kinda stuff, i recommend watching it)
- Nexenta stability at ver 3: My main concern was about problems i had seen online while doing my searches. Mainly, poor performance when a drive is starting to die (but hasnt died yet) and not being alerted to this via the Nexenta software (but being able to see the problem in the OS). I found some OS ways around this when it comes to alerting and while ultimately, i dont like it, the only way to see it get fixed is to join in and suggest workarounds or file bug reports to see it fixed in the next update (and pester if its not). I also noticed that alot of community issues had to do with hacking hardware to work with the OS. This concerned me less since i would be going the route of using only hardware tested with this system.
- Nexenta HSL (hardware supported list): is indeed limited, but i saw some flexibility here when comparing my self built quoted solutions against pre-built vendor created ones (that had similar items to the HSL, but not always exact). bottom line, work with a nexenta sales engineer and they can help you work out a solution that will hit the target goal (and in most cases price).
- Community support: the nexentastor.org forum is built on horrid forum software (i hear they are replacing it) and at first i did not see much action there. but over the past few months i have seen it improve, more help coming from community members as well as more nexenta employees starting to show up on the boards. This was encouraging for me. It also led me to joining myself, and while i am certainly no nexenta or ZFS guru, having a second pair of eyes looking at logical problems seems to help regardless. I see less posts going un-answered now.
Holy crap, this is turning into a novel, next up: Why we decided to go the self built route VS a nexenta/ZFS hardware partner.
(and after that, pictures of the build and the install)
I'd like future posts to include the testing phases while we put this thing through its pases. I'll have lots of questions on this forum i assume as i have never used iometer or bonnie. lol. so bear with me.
Last edited: