UPDATE: It seems I may be re-evaluating my choice of OpenSolaris vs. Nexenta. I’ve experienced a lot of issues with SMB/CIFS authentication on OpenSolaris, and have not been able to get it to work properly. I’ve also had a reply from a commenter assuring me as to the stability of Nexenta 3. I’ll post again once I’ve re-evaluated my choice of SAN OS.

A few months ago I wrote¹ about a new home server I was setting up. I designed the server from ground up to handle VMware ESXi 4.0. When I built it I did not build in data redundancy, as I had two mismatched drives (a 1.5TB and a 1TB). Also, because I was relatively new to ESXi, I created the datastore with default block sizes, limiting me to 256GB virtual disk file sizes. I used Ubuntu Linux to link the virtual disks together with Logical Volume Manager (LVM), and create one big mount for my data storage. Unfortunately, the 2.5TB volume is now full.

With a full server volume, nothing would seem more obvious than to go out and buy more storage. So, I went out and bought two 1.5TB disks (I don’t believe that 2TB drive sizes have hit the optimal price point yet). On my way home however, I realized that I now have the capacity for almost 5.5TB of storage. If one physical drive were to crash, I would not only lose the data that was on that drive, I would also lose my entire dataset. LVM does not handle missing drives, so the entire logical volume, with all my data on it, would be gone. This is far too much risk, and I decided to build in data redundancy.

With that in mind, I began to consider various options. The motherboard’s BIOS supports RAID 1,0, 10, and 5. Of those options, I would prefer a RAID5 configuration, as it offers the best capacity/redundancy ratio. Unfortunately however, I’ve already got ESXi installed on the existing 1.5TB drive, and the data between it and the second drive must remain intact. I don’t know how well ESXi would handle a sudden BIOS change to a RAID configuration. Also, after some reading, I found that it was likely that drivers would be required in the OS install to support the RAID configuration. There are too many unknown variables to risk my data with a BIOS RAID configuration change.

The next option I considered was a software level RAID5 implementation, one where I’d have a virtual machine handle the RAID5 control. Unfortunately however, this approach also has its drawbacks. RAID5 requires 3 drives of the same size to setup. I have 3 1.5TB drives right now, but one of them is full of data, including my ESXi host install. I would have to create a deteriorated RAID5 array with two drives, install another physical drive for the ESXi host install, import my original ESXi host configuration to the new host install, move my data to the new array, then move the actual client OS virtual disk to the new physical drive. After that point I could wipe the original 1.5TB and add it to the RAID5 array. I would be left with the 1TB to use for other purposes. During this whole process praying that something does not mess up the LVM i the Linux install. All in all, a very messy endeavor. Too much risk, both with the data itself as well as with the host/client OS installations.

Since a RAID configuration seemed to be out, I looked for other ‘outside-the-box’ solutions. Obviously it would have to be a disk/file level solution, as LVM with virtual disks wasn’t going to cut it. Then I remembered looking at ZFS² (a file system format) a couple years ago. ZFS offers great data redundancy for little disk cost, flexibility, compression, good performance, and a host of other things (things most non-technical people wouldn’t care about). The stability of the filesystem has come a long way since I first looked at it (it was more proof of concept at the time), to the point where I would trust my data with it. ZFS seemed to fit my current needs and network conditions perfectly.

Now that I’d decided on ZFS as my new network storage solution, I had to decide how I was going to implement it. Because it was developed by SUN Microsystems, there are licensing quirks that have kept it from being incorporated into the Linux kernel. There is however, an implementation via the FUSE project. I could potentially install it into my Ubuntu media server virtual machine, and have a relatively easy transition. After some investigation however, I felt that ZFS-fuse was still too much of a hack for me to trust my data with.

The only other real ZFS options were FreeBSD, OpenSolaris, and a project called Nexenta³. Nexenta is a Gnome (Ubuntu-like) user land built around the OpenSolaris kernel. This initially attracted me quite a bit, as it seemed to perhaps be the easiest way forward. Two things kept me back however. One, the version of Nexenta that offers deduplication support for ZFS is currently labeled beta. Two, because it was built around the OpenSolaris kernel, there would be a lot more hacking required if I was going to try to replicate my Ubuntu media serving services. At this point I realized that it would be easiest to keep my Ubuntu media server, and just point its data volumes to another VM’s network share, as though the other VM was a SAN. So, I decided that Nexenta was more than I needed, and that it was targeting a different person than I. Add to that the lack of deduplication, and Nexenta was out.

The other two ZFS options were OpenSolaris and FreeBSD. Since I’m a Max OS X (built around BSD Unix) power user, it seemed the most attractive option. On doing some analysis however, it seemed that OpenSolaris had better support and a better-performing ZFS implementation. Consequently, I’ve decided to go the OpenSolaris route for my virtual SAN.

I’ll post Part 2 over the next week or so. Part 2 will cover the actual implementation (still in progress), and some of the challenges encountered.

Related posts:

1. Update re: ESXi & ZFS Storage
2. VMWare ESXi as a Media Server

Enter ESXi. Virtualized machines brings me the ability to leave a media server up and running while I add, build, or reconfigure other machines, without the expense of all the hardware. It also brings all those machines into a single console, making it really convenient for me to manage. Along with that, I can now easily allocate more storage to the virtual machine as I have need!

Of course, this presented a problem. There is a lot of data that needed to be migrated. 1TB on the iMac, 320GB + 500GB + 160GB + 160GB on the existing media server, 160GB on the other MacBook Pro, and ~100GB on my MacBook Pro. How do you migrate all this data over? I budgeted for 1 new hard drive, but not more than one.

I purchased new server components (see below) on Boxing Day, and spent about ~$600 on what would have been an ~$830 server.

Components:

• Motherboard: Asus P5Q Pro Turbo
• CPU: Intel E7500 Boxed (mid-late 2009 versions have VT-x, a requirement for 64bit host VM’s)
• Ram: 4GB OCZ DDR2 800
• Hard-drive: WD 1.5TB
• Case: Antec P183
• PSU: Antec TruePower 650 New
• Video Card: Asus EAH4350 (Asus motherboard doesn’t have on-board video)
• NIC: Intel EXPI9301CT (Required, as ESXi 4 didn’t support the integrated Atheros-based NIC)

I began the new ESXi build by assembling the new hardware. As I wasn’t installing from USB and instead opted for the lower-hassle CD installation method, I borrowed a CD drive from my spare hardware box (I also borrowed keyboard/mouse/monitor). When I tried to boot ESXi for the first time, it informed me that the integrated NIC was not compatible. I nipped out to Canada Computers and purchased a new one, and the installation proceeded smoothly from that point on. Once installation was completed, I enabled SSH to ESXi by doing the following:

1. alt-f1 (Note: As pointed out below, you will not see your typing on this screen, just trust us, it is there).
2. unsupported
3. root pw
4. vi /etc/inetd.conf
5. delete the “#” from ssh
6. services.sh restart

I rebooted the machine, and went to my Windows box (no Mac ESXi/vSphere client yet) to manage the server. (If you haven’t installed the vSphere client yet, you can download it by going to your ESXi installation’s IP in your web browser) I setup a new VM for Ubuntu Linux 9.10, allocating 16GB to the VM hard drive, and configuring it with thin provisioning (dynamically expands). All my data would be on a second virtual drive. I then proceeded to install Ubuntu and install all the updates. Now it was time to add space for my data.

It was at this point that I hit a snag. VMware ESXi only supports 256GB virtual disks. I have terabytes of data, so how am I supposed to store all this information?

Enter LVM (Logical Volume Manager). LVM is a great technology that has seen a lot of use in Linux, allowing you to create volumes that span multiple hard-drives/partitions. I added 5 256GB thin provisioned disks to the Ubuntu installation, and proceeded to mount them as a single ext3 volume under /mnt/Shares. I set the volume to auto-mount on reboot in my /etc/fstab file, and proceeded to create the directories inside /mnt/Shares that would serve up my data. You can see a list of my shares below.

• Backup (To be used for Time Machine)
• Data (To be used for miscellaneous data, like eBooks)
• Movies
• Music
• Photos (Will contain an rsync-ed folder of my Adobe Lightroom photos, as well as a folder for miscellaneous images (scans, etc))
• Software (Will contain a copy of my software. I have directories created Windows and Mac, and the respective software in each)
• TV

The next interesting bit. How to actually share these shares. Obviously I don’t want to just open everything up to the world, and I also happen to have a multi-platform environment here, with Windows clients, Mac clients, and my landlord’s Linux clients. I elected to share all the shares using AFP (for my Macs), and the Movies, TV, and Software directories via SMB (for my Windows clients and the Linux clients). I created a user account called ‘main’ for my use, ‘kidd’ for my landlord, and ‘visitor’ for any guests that may be over. All three are password protected. ‘main’ has access to all shares, ‘kidd’ has access to Movies, TV, and Software, and ‘visitor’ has access to Movies and TV.

I shared the AFP shares to my Macs using a custom-compiled version of netatalk, following the great tutorial here. SMB configuration was fairly straightforward, and everything went well. I copied over as much data as possible (from the old server), before running out of disk space. At this point, I copied over data from the iMac 1TB into the now-empty old server, and popped the 1TB drive into the server. At this point I ran into the second snag. ESXi did not want to format the drive! I did some digging, and found that it was due to having been formatted as GUID before (in the Mac). I used dd to wipe out the first and last sectors of the drive, and tried to create a new partition via fdisk as per these instructions. Again, no go. After several attempts, I discovered that immediately after creating the partition table with fdisk, I had to reboot before using the vmkfstools command to create a datastore. After rebooting the server, I was able to format the drive appropriately. I logged into the Windows GUI console and deleted the new datastore. I then added the drive to the existing datastore, and fired up Ubuntu. I created several more 256GB thin-provisioned disks, and added them to the LVM.

I’ve installed ‘forked-daapd’ as a service on Ubuntu. This allows me to open Front Row and select the server as a source. I’ve also ‘tightvncserver’ set to run on startup, and have advertised the service in Avahi/Bonjour. This allows me to see the ‘Share Screen…’ button in OS X finder, and control the MediaServer via Screen Sharing, rather than a dedicated VNC client.

It’s been a great experiment, and now a great production home-media server. I’m looking forward to setting up my lab environment in other VM’s on ESXi. Feel free to leave me any questions/comments below that you might have on the process.

Related posts:

1. ESXi, ZFS, and Home Server Data Storage – Part 1
2. Update re: ESXi & ZFS Storage
3. Thoughts on the iPad