I posted an update¹ regarding my home server on ESXi a few months ago, and that I was re-evaluating how my data storage implementation was going to change. I’m happy to report that I’ve finally found an appropriate solution, utilizing the NexentaStor OpenSolaris-based OS². While not without its problems (notably a buggy web-management interface, I suspect due to Python issues), it has proved itself to be quite stable as far as data access and integrity.

In my last post about my home server I mentioned a few issues I would have migrating data, including the scaling down an LVM-based Linux VM to enable RDM (Raw Disk Mapping) to the NexentaStor VM. I mitigated this issue by adding 3 new drives, one ATA based 500GB drive to host the actual virtual machine installations, and two new 1.5TB SATA drives for storage. I mapped these two new drives to the NexentaStor VM using RDM and set them up as a ZFS array. I then migrated data over the network from the old Ubuntu VM to the new storage VM. After a large portion of data was migrated, I diminished the size of the LVM to free up the 1.5TB drive, and disconnected from the old storage VM. I then mapped it to the new storage VM and added it to the ZFS array. I then migrated the rest of the data and deleted the old storage VM.

So, my data storage needs are met for now. Unfortunately, the original 1TB drive has the ESXi installation on it, meaning I can’t format and add to the new ZFS array. I am considering the best approach to this, but it seems I’ll likely move the VM’s from the 500GB drive to the 1TB, disconnect all drives except the 500, and install ESXi to the drive. After this I would import the machines from the 1TB back to the 500GB drive. My concern with this approach is how to ensure that the physical ZFS disks import appropriately to the storage VM.

As always, post any questions in the comment section, or send me an email, and I’ll do my best to respond.

Related posts:

1. ESXi, ZFS, and Home Server Data Storage – Part 1
2. VMWare ESXi as a Media Server
3. Update Re: Print Issues

UPDATE: It seems I may be re-evaluating my choice of OpenSolaris vs. Nexenta. I’ve experienced a lot of issues with SMB/CIFS authentication on OpenSolaris, and have not been able to get it to work properly. I’ve also had a reply from a commenter assuring me as to the stability of Nexenta 3. I’ll post again once I’ve re-evaluated my choice of SAN OS.

A few months ago I wrote¹ about a new home server I was setting up. I designed the server from ground up to handle VMware ESXi 4.0. When I built it I did not build in data redundancy, as I had two mismatched drives (a 1.5TB and a 1TB). Also, because I was relatively new to ESXi, I created the datastore with default block sizes, limiting me to 256GB virtual disk file sizes. I used Ubuntu Linux to link the virtual disks together with Logical Volume Manager (LVM), and create one big mount for my data storage. Unfortunately, the 2.5TB volume is now full.

With a full server volume, nothing would seem more obvious than to go out and buy more storage. So, I went out and bought two 1.5TB disks (I don’t believe that 2TB drive sizes have hit the optimal price point yet). On my way home however, I realized that I now have the capacity for almost 5.5TB of storage. If one physical drive were to crash, I would not only lose the data that was on that drive, I would also lose my entire dataset. LVM does not handle missing drives, so the entire logical volume, with all my data on it, would be gone. This is far too much risk, and I decided to build in data redundancy.

With that in mind, I began to consider various options. The motherboard’s BIOS supports RAID 1,0, 10, and 5. Of those options, I would prefer a RAID5 configuration, as it offers the best capacity/redundancy ratio. Unfortunately however, I’ve already got ESXi installed on the existing 1.5TB drive, and the data between it and the second drive must remain intact. I don’t know how well ESXi would handle a sudden BIOS change to a RAID configuration. Also, after some reading, I found that it was likely that drivers would be required in the OS install to support the RAID configuration. There are too many unknown variables to risk my data with a BIOS RAID configuration change.

The next option I considered was a software level RAID5 implementation, one where I’d have a virtual machine handle the RAID5 control. Unfortunately however, this approach also has its drawbacks. RAID5 requires 3 drives of the same size to setup. I have 3 1.5TB drives right now, but one of them is full of data, including my ESXi host install. I would have to create a deteriorated RAID5 array with two drives, install another physical drive for the ESXi host install, import my original ESXi host configuration to the new host install, move my data to the new array, then move the actual client OS virtual disk to the new physical drive. After that point I could wipe the original 1.5TB and add it to the RAID5 array. I would be left with the 1TB to use for other purposes. During this whole process praying that something does not mess up the LVM i the Linux install. All in all, a very messy endeavor. Too much risk, both with the data itself as well as with the host/client OS installations.

Since a RAID configuration seemed to be out, I looked for other ‘outside-the-box’ solutions. Obviously it would have to be a disk/file level solution, as LVM with virtual disks wasn’t going to cut it. Then I remembered looking at ZFS² (a file system format) a couple years ago. ZFS offers great data redundancy for little disk cost, flexibility, compression, good performance, and a host of other things (things most non-technical people wouldn’t care about). The stability of the filesystem has come a long way since I first looked at it (it was more proof of concept at the time), to the point where I would trust my data with it. ZFS seemed to fit my current needs and network conditions perfectly.

Now that I’d decided on ZFS as my new network storage solution, I had to decide how I was going to implement it. Because it was developed by SUN Microsystems, there are licensing quirks that have kept it from being incorporated into the Linux kernel. There is however, an implementation via the FUSE project. I could potentially install it into my Ubuntu media server virtual machine, and have a relatively easy transition. After some investigation however, I felt that ZFS-fuse was still too much of a hack for me to trust my data with.

The only other real ZFS options were FreeBSD, OpenSolaris, and a project called Nexenta³. Nexenta is a Gnome (Ubuntu-like) user land built around the OpenSolaris kernel. This initially attracted me quite a bit, as it seemed to perhaps be the easiest way forward. Two things kept me back however. One, the version of Nexenta that offers deduplication support for ZFS is currently labeled beta. Two, because it was built around the OpenSolaris kernel, there would be a lot more hacking required if I was going to try to replicate my Ubuntu media serving services. At this point I realized that it would be easiest to keep my Ubuntu media server, and just point its data volumes to another VM’s network share, as though the other VM was a SAN. So, I decided that Nexenta was more than I needed, and that it was targeting a different person than I. Add to that the lack of deduplication, and Nexenta was out.

The other two ZFS options were OpenSolaris and FreeBSD. Since I’m a Max OS X (built around BSD Unix) power user, it seemed the most attractive option. On doing some analysis however, it seemed that OpenSolaris had better support and a better-performing ZFS implementation. Consequently, I’ve decided to go the OpenSolaris route for my virtual SAN.

I’ll post Part 2 over the next week or so. Part 2 will cover the actual implementation (still in progress), and some of the challenges encountered.

Related posts:

1. Update re: ESXi & ZFS Storage
2. VMWare ESXi as a Media Server

Enter ESXi. Virtualized machines brings me the ability to leave a media server up and running while I add, build, or reconfigure other machines, without the expense of all the hardware. It also brings all those machines into a single console, making it really convenient for me to manage. Along with that, I can now easily allocate more storage to the virtual machine as I have need!

Of course, this presented a problem. There is a lot of data that needed to be migrated. 1TB on the iMac, 320GB + 500GB + 160GB + 160GB on the existing media server, 160GB on the other MacBook Pro, and ~100GB on my MacBook Pro. How do you migrate all this data over? I budgeted for 1 new hard drive, but not more than one.

I purchased new server components (see below) on Boxing Day, and spent about ~$600 on what would have been an ~$830 server.

Components:

• Motherboard: Asus P5Q Pro Turbo
• CPU: Intel E7500 Boxed (mid-late 2009 versions have VT-x, a requirement for 64bit host VM’s)
• Ram: 4GB OCZ DDR2 800
• Hard-drive: WD 1.5TB
• Case: Antec P183
• PSU: Antec TruePower 650 New
• Video Card: Asus EAH4350 (Asus motherboard doesn’t have on-board video)
• NIC: Intel EXPI9301CT (Required, as ESXi 4 didn’t support the integrated Atheros-based NIC)

I began the new ESXi build by assembling the new hardware. As I wasn’t installing from USB and instead opted for the lower-hassle CD installation method, I borrowed a CD drive from my spare hardware box (I also borrowed keyboard/mouse/monitor). When I tried to boot ESXi for the first time, it informed me that the integrated NIC was not compatible. I nipped out to Canada Computers and purchased a new one, and the installation proceeded smoothly from that point on. Once installation was completed, I enabled SSH to ESXi by doing the following:

1. alt-f1 (Note: As pointed out below, you will not see your typing on this screen, just trust us, it is there).
2. unsupported
3. root pw
4. vi /etc/inetd.conf
5. delete the “#” from ssh
6. services.sh restart

I rebooted the machine, and went to my Windows box (no Mac ESXi/vSphere client yet) to manage the server. (If you haven’t installed the vSphere client yet, you can download it by going to your ESXi installation’s IP in your web browser) I setup a new VM for Ubuntu Linux 9.10, allocating 16GB to the VM hard drive, and configuring it with thin provisioning (dynamically expands). All my data would be on a second virtual drive. I then proceeded to install Ubuntu and install all the updates. Now it was time to add space for my data.

It was at this point that I hit a snag. VMware ESXi only supports 256GB virtual disks. I have terabytes of data, so how am I supposed to store all this information?

Enter LVM (Logical Volume Manager). LVM is a great technology that has seen a lot of use in Linux, allowing you to create volumes that span multiple hard-drives/partitions. I added 5 256GB thin provisioned disks to the Ubuntu installation, and proceeded to mount them as a single ext3 volume under /mnt/Shares. I set the volume to auto-mount on reboot in my /etc/fstab file, and proceeded to create the directories inside /mnt/Shares that would serve up my data. You can see a list of my shares below.

• Backup (To be used for Time Machine)
• Data (To be used for miscellaneous data, like eBooks)
• Movies
• Music
• Photos (Will contain an rsync-ed folder of my Adobe Lightroom photos, as well as a folder for miscellaneous images (scans, etc))
• Software (Will contain a copy of my software. I have directories created Windows and Mac, and the respective software in each)
• TV

The next interesting bit. How to actually share these shares. Obviously I don’t want to just open everything up to the world, and I also happen to have a multi-platform environment here, with Windows clients, Mac clients, and my landlord’s Linux clients. I elected to share all the shares using AFP (for my Macs), and the Movies, TV, and Software directories via SMB (for my Windows clients and the Linux clients). I created a user account called ‘main’ for my use, ‘kidd’ for my landlord, and ‘visitor’ for any guests that may be over. All three are password protected. ‘main’ has access to all shares, ‘kidd’ has access to Movies, TV, and Software, and ‘visitor’ has access to Movies and TV.

I shared the AFP shares to my Macs using a custom-compiled version of netatalk, following the great tutorial here. SMB configuration was fairly straightforward, and everything went well. I copied over as much data as possible (from the old server), before running out of disk space. At this point, I copied over data from the iMac 1TB into the now-empty old server, and popped the 1TB drive into the server. At this point I ran into the second snag. ESXi did not want to format the drive! I did some digging, and found that it was due to having been formatted as GUID before (in the Mac). I used dd to wipe out the first and last sectors of the drive, and tried to create a new partition via fdisk as per these instructions. Again, no go. After several attempts, I discovered that immediately after creating the partition table with fdisk, I had to reboot before using the vmkfstools command to create a datastore. After rebooting the server, I was able to format the drive appropriately. I logged into the Windows GUI console and deleted the new datastore. I then added the drive to the existing datastore, and fired up Ubuntu. I created several more 256GB thin-provisioned disks, and added them to the LVM.

I’ve installed ‘forked-daapd’ as a service on Ubuntu. This allows me to open Front Row and select the server as a source. I’ve also ‘tightvncserver’ set to run on startup, and have advertised the service in Avahi/Bonjour. This allows me to see the ‘Share Screen…’ button in OS X finder, and control the MediaServer via Screen Sharing, rather than a dedicated VNC client.

It’s been a great experiment, and now a great production home-media server. I’m looking forward to setting up my lab environment in other VM’s on ESXi. Feel free to leave me any questions/comments below that you might have on the process.

Related posts:

1. ESXi, ZFS, and Home Server Data Storage – Part 1
2. Update re: ESXi & ZFS Storage
3. Thoughts on the iPad

Anyway, this started a train of thoughts. Were I planning the technology end of a product launch, how would I prepare to deal with unanticipated demand? The problem with unanticipated demand is that you don’t know how many resources need to be set aside to cope with or load-balance. To me the answer is relatively simple. Leverage cloud computing resources. Build and use your front end for launch day on the cloud, preferably a cloud service that auto-scales to meet demand. Microsoft Azure would seem a great candidate for such a task, as it seems to extend the Windows Server platform nicely. You pay for the extra resources and bandwidth you utilize, but having a working front end on the most important day in your product’s lifecycle far outweighs the cost.

Simply planning ahead with the thought to use the ‘unlimited’ resources of a cloud computing platform for a product launch would be my preferred method of preparing for a major product launch.

Please leave me your thoughts below.

Related posts:

1. Cloud Computing & IT
2. Microsoft vs. iPad
3. Rogers & DOCSIS 3.0

My main concern is that, just like last time, this driver is going to mess up the print spooler service on the server, like the previous incident. We stand a very good chance of losing print services again if the dll is not replaced. The whole operation to fix it would take less than 10 minutes, including time to download the driver from HP.

In other news, the Microsoft Support site has been updated with a new look. Quite nice if you ask me.

Related posts:

1. Printer Issues
2. Update re: ESXi & ZFS Storage
3. Update on OHA Rollout Project

Update: I’ve gotten an insane amount of hits for “you do not have sufficient access to your computer to connect to the selected printer”, “windows cannot connect to the printer 000006be”, and “hp printer You do not have sufficient access”. (start reading at the 3rd paragraph if you want to jump straight to the details of my incident) If you want to jump straight to what the solution was for my issue, please see this Microsoft bulletin. I highly recommend reading this (PDF) HP document for a more detailed technical explanation. Also, if you could do me the courtesy of letting me know if this resolved your issue, whether via comment or email, I would greatly appreciate it.

Today’s post will be an exercise in troubleshooting. It addresses a problem I believe I’ve solved regarding our printer issues at work, as well as what was actually implemented.

So, two weeks ago we rolled out new network printers to our organization (I was not present at the time, I attend school fulltime). All the printers were the same make and model as the ones they were replacing, with the exception of a small P3005 (All our network printers are HP). Since the network printers were replaced, there have been a number of issues. Some people randomly can’t print to printers. Others can’t add printers to their local machine, receiving an error message ‘You do not have sufficient access to your computer to connect to this printer’. Initially we tried adding the users to the local power users and then local administrators group. This did not solve the issue. According to an employee from our finance department, the problems she had seemed to start she said the problems she had seemed to start when the printer near her was replaced. Interesting coincidence.

Our second level support tech told me that it was a problem with the local machines, and that they all were using the same standard image from June 2008 that I’d created. Interestingly enough, NOT ALL computers using that image were having the permissions issue. I then asked him if he’d updated any drivers on the server. He replied that he hadn’t updated any, except the driver for the P3005, the little printer, and that it was impossible for it to affect the other network printers. So I decided to do some more digging.

I found this Microsoft KB, KB954367. Microsoft’s bulletin describes a problem with an older HP driver, and an error message with:

Windows cannot connect to the printer. 
Operation could not be completed (error 0x000006be). 
You do not have sufficient access to your computer to connect to the selected printer.

Microsoft also says that “You may experience one or more of the following symptoms on a Windows Vista-based or a Windows Server 2003-based computer”. I tried to connect to one of the ‘affected’ printers using Windows Vista, and the spooler promptly crashed on the server, another symptom described in the bulletin. Microsoft suggests that the problem is due to a driver using HPBmini.dll of earlier than 1.0.0.19, or HPcdMC32.dll version less than 1.0.2.35. I went and checked the version of the driver on the server. The HPBmini.dll file is 1.0.0.19, but the HPcdMC32.dll file is 1.0.1.17, quite a bit earlier than the max threshold set by the Microsoft KB. Armed with this knowledge, I felt that it was very likely the driver that caused the issue. I then went and checked out the HP bulletin linked to from Microsoft’s KB, and found that among the affected products, the driver shipping with P3005 is one of the affected.

I brought this information to my manager, and he instructed me to talk to the second level tech before I updated the driver on the server. The second level tech refused to allow me to update the driver on the server, saying that it was impossible for the driver to cause the issue, since P3005 is only a small printer and the driver shouldn’t affect any other printer drivers. I tried to explain the fact that it updated files common to all the HP printers, but this was rejected. He ended up installing a new image on the client machines, and the problem seemed to go away. The driver on the server was not updated.

Note: To be perfectly fair, the knowledge bulletin specifies that the issue occurs on a Windows Server 2003 or Windows Vista installation with an HP printer connected. However, in my opinion this does not mean that the same issue won’t occur on Windows XP. In fact, it seems to me even more likely, since Windows Server 2003 and Windows XP use very similar driver mechanisms.

Unfortunately however, it seems the problem might NOT have gone away. Apparently this morning (I’m not in on Mondays) the printspl.exe process on the print server refused to start, and no one on the network was able to print to any printers. Now, (calling any and all people with any IT experience) what’s the common cause of the print spooler not working? A corrupted driver? You guessed right! I did some browsing on HP’s forums, and it seems like their printer drivers or the HP Inject service (which is installed along side the driver frequently) often mess up the print spooler.

Note: There are other causes for a print spooler to fail, but since it’s been functioning fine for the last several years, and the only real change was an updated driver, that the driver is what was causing havoc.

I found out that the way they tried to resolve the issue was by uninstalling every driver till the last modified. Unfortunately however, if the information I’d submitted had been read completely, it would have been understood that the whole reason for the issue in the first place was because the files are timestamped incorrectly. Windows Server 2003 does not look at the file version when updating. Rather, it looks at the time stamp of the file to determine whether it should be updated or not. Uninstalling won’t help. HP says that the only way to rectify this problem is to manually copy over the new dll’s. Uninstalling and rolling back the drivers won’t work. This would have been known had the ‘More information’ been read.

After getting an update from my manager this evening, I found that they’re (IT) is setting up a new print server on one of our other servers. At least it will get the organization printing again. Our main job is to get keep the organization functioning. I’m just disappointed that we had to experience downtime, and that what I suspect was the real cause wasn’t fixed. I’m trying to understand what the big resistance was to following my recommendation to update the driver per HP’s suggestion. I still firmly believe that if we were to fix the ‘updated’ driver on the old print server, things would be up and functional. However, I haven’t been given that liberty, so we’re stuck with our new print server. This means everyone has to re-add their printers, whether they’re having issues or not.

Feel free to leave your comments and/or troubleshooting suggestions. I can’t really implement them now that we have a new print server, but I’d love to hear suggestions for alternative solutions.

Related posts:

1. Update Re: Print Issues
2. Imaging Issues
3. Interesting Outlook 2007 Client Issue