UPDATE: It seems I may be re-evaluating my choice of OpenSolaris vs. Nexenta. I’ve experienced a lot of issues with SMB/CIFS authentication on OpenSolaris, and have not been able to get it to work properly. I’ve also had a reply from a commenter assuring me as to the stability of Nexenta 3. I’ll post again once I’ve re-evaluated my choice of SAN OS.
A few months ago I wrote1 about a new home server I was setting up. I designed the server from ground up to handle VMware ESXi 4.0. When I built it I did not build in data redundancy, as I had two mismatched drives (a 1.5TB and a 1TB). Also, because I was relatively new to ESXi, I created the datastore with default block sizes, limiting me to 256GB virtual disk file sizes. I used Ubuntu Linux to link the virtual disks together with Logical Volume Manager (LVM), and create one big mount for my data storage. Unfortunately, the 2.5TB volume is now full.
With a full server volume, nothing would seem more obvious than to go out and buy more storage. So, I went out and bought two 1.5TB disks (I don’t believe that 2TB drive sizes have hit the optimal price point yet). On my way home however, I realized that I now have the capacity for almost 5.5TB of storage. If one physical drive were to crash, I would not only lose the data that was on that drive, I would also lose my entire dataset. LVM does not handle missing drives, so the entire logical volume, with all my data on it, would be gone. This is far too much risk, and I decided to build in data redundancy.
With that in mind, I began to consider various options. The motherboard’s BIOS supports RAID 1,0, 10, and 5. Of those options, I would prefer a RAID5 configuration, as it offers the best capacity/redundancy ratio. Unfortunately however, I’ve already got ESXi installed on the existing 1.5TB drive, and the data between it and the second drive must remain intact. I don’t know how well ESXi would handle a sudden BIOS change to a RAID configuration. Also, after some reading, I found that it was likely that drivers would be required in the OS install to support the RAID configuration. There are too many unknown variables to risk my data with a BIOS RAID configuration change.
The next option I considered was a software level RAID5 implementation, one where I’d have a virtual machine handle the RAID5 control. Unfortunately however, this approach also has its drawbacks. RAID5 requires 3 drives of the same size to setup. I have 3 1.5TB drives right now, but one of them is full of data, including my ESXi host install. I would have to create a deteriorated RAID5 array with two drives, install another physical drive for the ESXi host install, import my original ESXi host configuration to the new host install, move my data to the new array, then move the actual client OS virtual disk to the new physical drive. After that point I could wipe the original 1.5TB and add it to the RAID5 array. I would be left with the 1TB to use for other purposes. During this whole process praying that something does not mess up the LVM i the Linux install. All in all, a very messy endeavor. Too much risk, both with the data itself as well as with the host/client OS installations.
Since a RAID configuration seemed to be out, I looked for other ‘outside-the-box’ solutions. Obviously it would have to be a disk/file level solution, as LVM with virtual disks wasn’t going to cut it. Then I remembered looking at ZFS2 (a file system format) a couple years ago. ZFS offers great data redundancy for little disk cost, flexibility, compression, good performance, and a host of other things (things most non-technical people wouldn’t care about). The stability of the filesystem has come a long way since I first looked at it (it was more proof of concept at the time), to the point where I would trust my data with it. ZFS seemed to fit my current needs and network conditions perfectly.
Now that I’d decided on ZFS as my new network storage solution, I had to decide how I was going to implement it. Because it was developed by SUN Microsystems, there are licensing quirks that have kept it from being incorporated into the Linux kernel. There is however, an implementation via the FUSE project. I could potentially install it into my Ubuntu media server virtual machine, and have a relatively easy transition. After some investigation however, I felt that ZFS-fuse was still too much of a hack for me to trust my data with.
The only other real ZFS options were FreeBSD, OpenSolaris, and a project called Nexenta3. Nexenta is a Gnome (Ubuntu-like) user land built around the OpenSolaris kernel. This initially attracted me quite a bit, as it seemed to perhaps be the easiest way forward. Two things kept me back however. One, the version of Nexenta that offers deduplication support for ZFS is currently labeled beta. Two, because it was built around the OpenSolaris kernel, there would be a lot more hacking required if I was going to try to replicate my Ubuntu media serving services. At this point I realized that it would be easiest to keep my Ubuntu media server, and just point its data volumes to another VM’s network share, as though the other VM was a SAN. So, I decided that Nexenta was more than I needed, and that it was targeting a different person than I. Add to that the lack of deduplication, and Nexenta was out.
The other two ZFS options were OpenSolaris and FreeBSD. Since I’m a Max OS X (built around BSD Unix) power user, it seemed the most attractive option. On doing some analysis however, it seemed that OpenSolaris had better support and a better-performing ZFS implementation. Consequently, I’ve decided to go the OpenSolaris route for my virtual SAN.
I’ll post Part 2 over the next week or so. Part 2 will cover the actual implementation (still in progress), and some of the challenges encountered.
NCP3 beta is quite stable.. moreso than NCP2 stable. The RC release will be out soon.
There’s also the free NexentaStor CE (nexentastor.org) specifically for Home NAS/SAN, based on Nexenta. It supports dedup as well.
Hi Anil. I’m sure as NCP3 is at beta 3 it is at a fairly stable state of development. However, I don’t like risking terabytes of data with something that hasn’t had development complete yet. Perhaps I’ve been working too long, and getting too cautious.
Regarding NexentaStor CE. I looked at it briefly, but again, it was at the point where I’d realized all I wanted this particular VM to do was SAN. NexentaStor is still a relatively young platform, attempting to bring a lot of new functionality, most of which I don’t actually need. I opted for a more mature platform, both because of its maturity and also because there is far more supporting documentation and community online for it. For example, I hadn’t really heard much of Nexenta until now, but I’d heard of OpenSolaris many times, over many years. When it comes to massive amounts of data I get tender feet.
OpenSolaris is being slowly killed by Oracle… I find it too risky to do anything with this OS at the moment.
Hi Oded. It depends on what your needs are. Personally, I think that for purposes of a media server at home, an OpenSolaris based distro is supported well enough. I don’t think it will be disappearing anytime soon, and even if it does, the only implication for me is that there will be no enhancements added. What I have suits my needs, and is documented, stable, and supported. Even if Oracle drops OpenSolaris tomorrow, it will not affect my current implementation.
Wes, I fully agree with your assessment, and personally I think that the core of OpenSolaris is incredible. It’s just that by knowing how Oracle works (I deal a lot with their products where I work…) I feel that it could be a problem sooner than later.
However, a solution like Nexenta where they pick up with Oracle seems to have left off sounds like a great idea to me. I also recommend to have a look at the Illumos project (http://www.illumos.org/), it is in very early stages and not usable yet, but it has potential.
Interesting reading, thanks.