I posted an update¹ regarding my home server on ESXi a few months ago, and that I was re-evaluating how my data storage implementation was going to change. I’m happy to report that I’ve finally found an appropriate solution, utilizing the NexentaStor OpenSolaris-based OS². While not without its problems (notably a buggy web-management interface, I suspect due to Python issues), it has proved itself to be quite stable as far as data access and integrity.

In my last post about my home server I mentioned a few issues I would have migrating data, including the scaling down an LVM-based Linux VM to enable RDM (Raw Disk Mapping) to the NexentaStor VM. I mitigated this issue by adding 3 new drives, one ATA based 500GB drive to host the actual virtual machine installations, and two new 1.5TB SATA drives for storage. I mapped these two new drives to the NexentaStor VM using RDM and set them up as a ZFS array. I then migrated data over the network from the old Ubuntu VM to the new storage VM. After a large portion of data was migrated, I diminished the size of the LVM to free up the 1.5TB drive, and disconnected from the old storage VM. I then mapped it to the new storage VM and added it to the ZFS array. I then migrated the rest of the data and deleted the old storage VM.

So, my data storage needs are met for now. Unfortunately, the original 1TB drive has the ESXi installation on it, meaning I can’t format and add to the new ZFS array. I am considering the best approach to this, but it seems I’ll likely move the VM’s from the 500GB drive to the 1TB, disconnect all drives except the 500, and install ESXi to the drive. After this I would import the machines from the 1TB back to the 500GB drive. My concern with this approach is how to ensure that the physical ZFS disks import appropriately to the storage VM.

As always, post any questions in the comment section, or send me an email, and I’ll do my best to respond.

Related posts:

1. ESXi, ZFS, and Home Server Data Storage – Part 1
2. VMWare ESXi as a Media Server
3. Update Re: Print Issues

UPDATE: It seems I may be re-evaluating my choice of OpenSolaris vs. Nexenta. I’ve experienced a lot of issues with SMB/CIFS authentication on OpenSolaris, and have not been able to get it to work properly. I’ve also had a reply from a commenter assuring me as to the stability of Nexenta 3. I’ll post again once I’ve re-evaluated my choice of SAN OS.

A few months ago I wrote¹ about a new home server I was setting up. I designed the server from ground up to handle VMware ESXi 4.0. When I built it I did not build in data redundancy, as I had two mismatched drives (a 1.5TB and a 1TB). Also, because I was relatively new to ESXi, I created the datastore with default block sizes, limiting me to 256GB virtual disk file sizes. I used Ubuntu Linux to link the virtual disks together with Logical Volume Manager (LVM), and create one big mount for my data storage. Unfortunately, the 2.5TB volume is now full.

With a full server volume, nothing would seem more obvious than to go out and buy more storage. So, I went out and bought two 1.5TB disks (I don’t believe that 2TB drive sizes have hit the optimal price point yet). On my way home however, I realized that I now have the capacity for almost 5.5TB of storage. If one physical drive were to crash, I would not only lose the data that was on that drive, I would also lose my entire dataset. LVM does not handle missing drives, so the entire logical volume, with all my data on it, would be gone. This is far too much risk, and I decided to build in data redundancy.

With that in mind, I began to consider various options. The motherboard’s BIOS supports RAID 1,0, 10, and 5. Of those options, I would prefer a RAID5 configuration, as it offers the best capacity/redundancy ratio. Unfortunately however, I’ve already got ESXi installed on the existing 1.5TB drive, and the data between it and the second drive must remain intact. I don’t know how well ESXi would handle a sudden BIOS change to a RAID configuration. Also, after some reading, I found that it was likely that drivers would be required in the OS install to support the RAID configuration. There are too many unknown variables to risk my data with a BIOS RAID configuration change.

The next option I considered was a software level RAID5 implementation, one where I’d have a virtual machine handle the RAID5 control. Unfortunately however, this approach also has its drawbacks. RAID5 requires 3 drives of the same size to setup. I have 3 1.5TB drives right now, but one of them is full of data, including my ESXi host install. I would have to create a deteriorated RAID5 array with two drives, install another physical drive for the ESXi host install, import my original ESXi host configuration to the new host install, move my data to the new array, then move the actual client OS virtual disk to the new physical drive. After that point I could wipe the original 1.5TB and add it to the RAID5 array. I would be left with the 1TB to use for other purposes. During this whole process praying that something does not mess up the LVM i the Linux install. All in all, a very messy endeavor. Too much risk, both with the data itself as well as with the host/client OS installations.

Since a RAID configuration seemed to be out, I looked for other ‘outside-the-box’ solutions. Obviously it would have to be a disk/file level solution, as LVM with virtual disks wasn’t going to cut it. Then I remembered looking at ZFS² (a file system format) a couple years ago. ZFS offers great data redundancy for little disk cost, flexibility, compression, good performance, and a host of other things (things most non-technical people wouldn’t care about). The stability of the filesystem has come a long way since I first looked at it (it was more proof of concept at the time), to the point where I would trust my data with it. ZFS seemed to fit my current needs and network conditions perfectly.

Now that I’d decided on ZFS as my new network storage solution, I had to decide how I was going to implement it. Because it was developed by SUN Microsystems, there are licensing quirks that have kept it from being incorporated into the Linux kernel. There is however, an implementation via the FUSE project. I could potentially install it into my Ubuntu media server virtual machine, and have a relatively easy transition. After some investigation however, I felt that ZFS-fuse was still too much of a hack for me to trust my data with.

The only other real ZFS options were FreeBSD, OpenSolaris, and a project called Nexenta³. Nexenta is a Gnome (Ubuntu-like) user land built around the OpenSolaris kernel. This initially attracted me quite a bit, as it seemed to perhaps be the easiest way forward. Two things kept me back however. One, the version of Nexenta that offers deduplication support for ZFS is currently labeled beta. Two, because it was built around the OpenSolaris kernel, there would be a lot more hacking required if I was going to try to replicate my Ubuntu media serving services. At this point I realized that it would be easiest to keep my Ubuntu media server, and just point its data volumes to another VM’s network share, as though the other VM was a SAN. So, I decided that Nexenta was more than I needed, and that it was targeting a different person than I. Add to that the lack of deduplication, and Nexenta was out.

The other two ZFS options were OpenSolaris and FreeBSD. Since I’m a Max OS X (built around BSD Unix) power user, it seemed the most attractive option. On doing some analysis however, it seemed that OpenSolaris had better support and a better-performing ZFS implementation. Consequently, I’ve decided to go the OpenSolaris route for my virtual SAN.

I’ll post Part 2 over the next week or so. Part 2 will cover the actual implementation (still in progress), and some of the challenges encountered.

Related posts:

1. Update re: ESXi & ZFS Storage
2. VMWare ESXi as a Media Server

Let me start off by laying out what I believe are the three tiers of cloud computing. I believe that cloud computing consists of hosted services (IBM), hosted applications (VMWare, Citrix), and hosted storage/data. This contrasts with internal services (SharePoint, Exchange), internal applications (MS Office, Adobe Acrobat), and internal data (File server). The internal computing may be run on virtualized servers, or on their dedicated boxes. In my opinion, businesses are concerned firstly with security regarding hosted documents/storage layer, secondly web services, and thirdly, most open to hosted web applications.

We are seeing a shift away from applications run on local machines to run on a server and/or virtualized (Citrix, VMWare). These applications tie into the internal services and storage. We are also seeing a shift away from local and/or virtualized applications to web applications. While not all applications can be run as hosted applications (Multimedia applications, lab machines, etc), most of the basic productivity apps are able to be run online or on a VPN server. Web applications (whether hosted or run internally) tie into the company’s services. These services in turn create mountains of data to be stored. Where does the data get stored? Should companies use hosted storage or should they use internal storage? If they want internal storage, they are going to have to create a framework infrastructure to dump this data internally, whether it be from web services or from hosted/web applications. There is no competing with the onslaught of hosted applications. Some are transitioning to web applications, but most (if not all) will be transitioning from local to being hosted.

Here’s a brief synopsis of what companies are offering what (in my model of a three tier system):

• Vmware vCloud – virtualized hosted apps, tie into internal infrastructure, thinapps
• Microsoft Azure- hosted services, storage, web applications
• Cisco VN- virtualized apps, network
• Citrix – hosted apps
• IBM – hosted services

Let’s move to the next tier, the web services layer. This layer provides services such as email or forums (Microsoft Exchange Online, Drupal, Joomla, IBM services). I would argue that it also includes web serving services as well. The problem with this layer is that this is where more and more malicious apps are being targeted. Microsoft recently did a study that showed that malicious apps are targeted more and more at the browser or specific apps. They target the authentication component of the web services. If compromised, an attacker can use the information gathered to gain access to the data contained (emails, documents in the document management system). However, with enough security measures, this risk can be mitigated for the most part. 

The other negative to having hosted web services is that your emails and forums are being hosted on someone else’s server. How much this affects your organization is something you have to decide. 

The benefit of hosted services is that the cost to run them is miniscule. Organizations no longer need to employ internal staff to maintain these, and pay a lot less on implementation costs. It also provides arguably better failover services. The benefits are very attractive for organizations. I would argue that competing with hosted web services is a waste of time for the most part.

The third layer is the data storage layer. This is where all your documents are stored. This would include Microsoft SharePoint Online, Live Mesh, or other web storge services. This is critical. Are you willing to have your documents hosted elsewhere? Do you have thousands of employee SSN’s or financial data stored? This is a weighty matter to consider. What about if somehow an employee at the company hosting your data manages to break in. How big is this risk? Are you willing to take it? 

Here is where I see things going:
Government & Financial institutions will likely graduate to hosted applications or web apps, while moving to virtualized services and internal data. VMWare/Cisco will be the big players. Security is paramount, so services will all be virtualized internally.

Medium to Large organizations will move to hosted services, hosted AND virtualized applications, while maintaining data internally. Big players include Microsoft (Azure) and VMWare. Cisco will also play a fairly large role. The employees will be working from home/remotely more frequently, as well as have intermittent (flights, dialup, etc) internet outages, so offline virtualization is more important.

Small to Medium organizations will graduate to hosted or virtualized applications, hosted services, and hosted data. Big player: Microsoft. The cost outweighs the benefit, so being heavily internet based will be more attractive than maintaining ability to work offline.

Related posts:

1. Merits of Windows Vista
2. Update re: ESXi & ZFS Storage
3. The Cloud as a Load Balancer

Enter ESXi. Virtualized machines brings me the ability to leave a media server up and running while I add, build, or reconfigure other machines, without the expense of all the hardware. It also brings all those machines into a single console, making it really convenient for me to manage. Along with that, I can now easily allocate more storage to the virtual machine as I have need!

Of course, this presented a problem. There is a lot of data that needed to be migrated. 1TB on the iMac, 320GB + 500GB + 160GB + 160GB on the existing media server, 160GB on the other MacBook Pro, and ~100GB on my MacBook Pro. How do you migrate all this data over? I budgeted for 1 new hard drive, but not more than one.

I purchased new server components (see below) on Boxing Day, and spent about ~$600 on what would have been an ~$830 server.

Components:

• Motherboard: Asus P5Q Pro Turbo
• CPU: Intel E7500 Boxed (mid-late 2009 versions have VT-x, a requirement for 64bit host VM’s)
• Ram: 4GB OCZ DDR2 800
• Hard-drive: WD 1.5TB
• Case: Antec P183
• PSU: Antec TruePower 650 New
• Video Card: Asus EAH4350 (Asus motherboard doesn’t have on-board video)
• NIC: Intel EXPI9301CT (Required, as ESXi 4 didn’t support the integrated Atheros-based NIC)

I began the new ESXi build by assembling the new hardware. As I wasn’t installing from USB and instead opted for the lower-hassle CD installation method, I borrowed a CD drive from my spare hardware box (I also borrowed keyboard/mouse/monitor). When I tried to boot ESXi for the first time, it informed me that the integrated NIC was not compatible. I nipped out to Canada Computers and purchased a new one, and the installation proceeded smoothly from that point on. Once installation was completed, I enabled SSH to ESXi by doing the following:

1. alt-f1 (Note: As pointed out below, you will not see your typing on this screen, just trust us, it is there).
2. unsupported
3. root pw
4. vi /etc/inetd.conf
5. delete the “#” from ssh
6. services.sh restart

I rebooted the machine, and went to my Windows box (no Mac ESXi/vSphere client yet) to manage the server. (If you haven’t installed the vSphere client yet, you can download it by going to your ESXi installation’s IP in your web browser) I setup a new VM for Ubuntu Linux 9.10, allocating 16GB to the VM hard drive, and configuring it with thin provisioning (dynamically expands). All my data would be on a second virtual drive. I then proceeded to install Ubuntu and install all the updates. Now it was time to add space for my data.

It was at this point that I hit a snag. VMware ESXi only supports 256GB virtual disks. I have terabytes of data, so how am I supposed to store all this information?

Enter LVM (Logical Volume Manager). LVM is a great technology that has seen a lot of use in Linux, allowing you to create volumes that span multiple hard-drives/partitions. I added 5 256GB thin provisioned disks to the Ubuntu installation, and proceeded to mount them as a single ext3 volume under /mnt/Shares. I set the volume to auto-mount on reboot in my /etc/fstab file, and proceeded to create the directories inside /mnt/Shares that would serve up my data. You can see a list of my shares below.

• Backup (To be used for Time Machine)
• Data (To be used for miscellaneous data, like eBooks)
• Movies
• Music
• Photos (Will contain an rsync-ed folder of my Adobe Lightroom photos, as well as a folder for miscellaneous images (scans, etc))
• Software (Will contain a copy of my software. I have directories created Windows and Mac, and the respective software in each)
• TV

The next interesting bit. How to actually share these shares. Obviously I don’t want to just open everything up to the world, and I also happen to have a multi-platform environment here, with Windows clients, Mac clients, and my landlord’s Linux clients. I elected to share all the shares using AFP (for my Macs), and the Movies, TV, and Software directories via SMB (for my Windows clients and the Linux clients). I created a user account called ‘main’ for my use, ‘kidd’ for my landlord, and ‘visitor’ for any guests that may be over. All three are password protected. ‘main’ has access to all shares, ‘kidd’ has access to Movies, TV, and Software, and ‘visitor’ has access to Movies and TV.

I shared the AFP shares to my Macs using a custom-compiled version of netatalk, following the great tutorial here. SMB configuration was fairly straightforward, and everything went well. I copied over as much data as possible (from the old server), before running out of disk space. At this point, I copied over data from the iMac 1TB into the now-empty old server, and popped the 1TB drive into the server. At this point I ran into the second snag. ESXi did not want to format the drive! I did some digging, and found that it was due to having been formatted as GUID before (in the Mac). I used dd to wipe out the first and last sectors of the drive, and tried to create a new partition via fdisk as per these instructions. Again, no go. After several attempts, I discovered that immediately after creating the partition table with fdisk, I had to reboot before using the vmkfstools command to create a datastore. After rebooting the server, I was able to format the drive appropriately. I logged into the Windows GUI console and deleted the new datastore. I then added the drive to the existing datastore, and fired up Ubuntu. I created several more 256GB thin-provisioned disks, and added them to the LVM.

I’ve installed ‘forked-daapd’ as a service on Ubuntu. This allows me to open Front Row and select the server as a source. I’ve also ‘tightvncserver’ set to run on startup, and have advertised the service in Avahi/Bonjour. This allows me to see the ‘Share Screen…’ button in OS X finder, and control the MediaServer via Screen Sharing, rather than a dedicated VNC client.

It’s been a great experiment, and now a great production home-media server. I’m looking forward to setting up my lab environment in other VM’s on ESXi. Feel free to leave me any questions/comments below that you might have on the process.

Related posts:

1. ESXi, ZFS, and Home Server Data Storage – Part 1
2. Update re: ESXi & ZFS Storage
3. Thoughts on the iPad