Mitch's Blog

Colleagues who have worked with me in the last ten years know I have some religion about appliances. To a large degree, I believe that most enterprise software that is installed in a customer’s data center should be packaged as an appliance. The embodiment of this could be as a physical appliance or as a virtual software appliance, but either way, it’s an appliance.

My sense is that appliances, especially virtual appliances, are picking up momentum. We’re seeing more and more software companies offering virtual appliances on the market today and more products that could be packaged as software instead packaged as a hardware offering.

There’s a problem though.

Most of these appliances are crap.

What is an Appliance?
It would be cliché to turn to Merriam Webster for a definition, so let’s consider what the Oxford American Dictionary says an appliance is:

ap-pli-ance n. 1 a device or piece of equipment designed to perform a specific task

So if we’re building an enterprise IT appliance, it’s going to be a virtual machine or a physical machine to perform a specific task.

In my mind I’ve always given Dave Hitz, co-founder of NetApp, credit for defining an appliance, but in this interview [PDF], he gives credit to the model Cisco established:

Hitz: […] We were modeling ourselves after Cisco. If you look before Cisco, people used general purpose computers, UNIX workstations, Sun workstations for their routers. Cisco came along and built a box that was just for routing. It did less than a general purpose computer and so the conversation we kept having with the VCs, they would say, “So your box does more than a general purpose computer so it’s better?” And we’d say, “No, no, it does less.” And they’d go, “Well, why would I pay you money to start a company that does less?” And we said, “No, no, it’s focused. It’ll do it better.”

He continues:

The other analogy that we used was a toaster. If you want to make toast, you could go put in on your burner, you can put it in the oven and it’s certainly the case an oven’s a lot more of a general purpose tool than a toaster. A toaster? What good’s a toaster? All it does is make toast. But, man, if you want toast, a toaster really does a good job, you know? Try making toast in your oven and you’ll discover how awesome a toaster is. And that really, for storage over the network, it was that idea of making a device to do just that one thing, focused. Not run programs, not do spreadsheets, not be a tool for programmers, just store the data.

I’ve always stuck by the toaster model when thinking about appliances for customers.

So what does this mean for the user?
Let’s talk about product requirements.

When a vendor ships an appliance to the customer, everything is included: The OS, the management tools, and the application. The beauty of this approach is that the customer no longer has to install software. The customer is no longer responsible for dealing with OS patching and ensuring the right OS patches are installed that match with the requirements of the application.

The user interface should be stripped down and minimal. The user should never see a general purpose bash shell prompt. The appliance should be as self-contained as reasonable. Configuration should be as minimal as possible. IP address and related information, LDAP or user set-up, NTP and timezone, email and SNMP, application-specific coordinates, and that’s it! This can be implemented in a small X11 set-up GUI (Nasuni does an excellent job here) or a simple ncurses-based text wizard (similar to what ESXi has).

The user should never be told to edit files off in /etc or manually tweak key-value pairs to get the basic application up and running. Some application programmers get off on connecting to a customer-provided database. In a particularly large installation, this might make sense. To store 500 rows of data, it does not.

An appliance is likely Linux-based these days, as it should be. Software updates should come from an update server provided by the vendor. From the user’s perspective, there is no apt-get or yum details to worry about; it’s a simple matter of clicking a ‘Upgrade’ button. Anyone shipping a Windows-based appliance should only do so for very specific reasons (e.g., need access to Windows APIs).

The GUI for the appliance should be web-based. These days, it needs to be accessible from a variety of browsers. Product managers need to understand the average IT guy has more things to worry about than a single product, and hints need to be provided on how to find the management UI for the appliance. I love how HP handles this on their switches and I adopted their model at my last company: Every login console tty included the IP address and how to find the GUI; both shell and GUI login screens included the default account name and password, right there. The last time you installed a home router, did you have to fire up Google and search for the default user name and password? Or did you read the documentation? Did you read the documentation for Tetris? Why should the default user name and password be harder than playing Tetris?

DHCP should be the default on a first boot, and active on all interfaces with a link. Anyone shipping an appliance without DHCP in this day and age needs to get with the times.

Any CLI should be succinct and hierarchical. Enterprise IT gets tricky, no matter how we simplify it, and users need a way to drill into the available commands. The CLI needs to balance “learning” with speed of use. The reality is, most IT folks are using a lot of different products throughout the day. Make yours the one that is easiest for the admin to jog his mind on how to see replication status (or whatever).

When it comes to lights-out management, product managers and engineers must understand that the product is not the center of the end-user’s universe. Problems and reports should be available via email. With a generated email, the subject line is critical and should be succinct but descriptive. “2 alerts (product name)” with a From: field of the hostname is a good start at a subject line. Failures and recovery from failures should be as automatic as possible. Customers should never have to login to click something in a GUI that the product could have run itself to recover from an error.

Although not strictly appliance-related, let’s also talk about sockets and ports of the TCP/IP variety. When it comes to networking, products should be designed with WANs, VLANs, routers, closed ports, and different views of networks when communicating with other instances of the same product. Don’t rely on ICMP pings being available. Don’t rely on your connection target matching the listening target coordinates. Have flexibility. Design your network protocols for the simplest firewall configurations. In a spoke/hub topology, all the spokes should reach into the hub–that is, all spokes open sockets on a port the hub is listening on, even if the logical client/server is the other direction.

Shipping a Virtual Appliance
This is where I see the most sin in the market today. So many folks install Ubuntu, install their software, shutdown the VM, zip it up, ftp it over to their web site and call it a “virtual appliance”. This is a VMDK full of crap. /root/.bash_history is still populated, as are DHCP cache files, log files, and other things that should be scrubbed. There is no OVF installer file. Old snapshots of the VM are included. What kind of embarrassing stuff is in there?

Let me state up front that OVF is one of the dumbest things in the VMware world. There is no need for it to exist; everything that it does could have been accomplished much more simply. But VMware doesn’t like to make things easy, so we’re stuck with it. On the bright side, the user experience of installing an OVF appliance into a VMware environment is awesome. Thus, vendors need to use it.

From a mindset perspective, the vendor needs to get into a mental mode of a manufacturing process to scrub the VM prior to OVF packaging. Things need to be set to “factory defaults”.

With both physical and virtual appliances, the size of Linux distribution should be minimal. This is more important with the virtual appliance distribution, since customers will be downloading it. As a vendor, you can pick a few approaches on how you do this, but in general, reduce the number of packages you’re including to the bare minimum. You need to tune your CPUs and RAM to the minimum required for your application and let customers tune up from there. If you’re shipping both virtual and physical appliances, you need to understand these are different beasts. How will you sell them? My bet is that you’re pushing virtual appliances into remote-office/branch-office use cases, so size the requirements for the VM appropriately. Size your VMDKs down to something reasonable or use thin-provisioning. Note that OVFs with thin-provisioning will not work on ESX versions less than 4.1 or 4.5 (I can’t remember exactly which versions are broken for thin-provisioning with OVF).

Shipping a Physical Appliance
Ten years ago, I worked on my first appliance product. One of the problems we had with this physical appliance when we went out in the field was that we needed a screwdriver to get it in the rack. Customers sometimes had to spend 10 minutes searching for tools if they were installing it themselves. We started to include a multi-tip screwdriver with the product in the shipping carton. In 2010, my new company had the opportunity to buy a few systems from the previous company. The screwdriver was still included. If I ever do a physical appliance again, I will include the screwdriver. What’s $10 on a $50,000 system anyway? Don’t forget this is a branding opportunity–put your logo or URL on the handle.

If your system is heavy, ship it on a pallet. If you’re on the border (70-80 lbs), pay attention to the condition that the system packaging is in when it arrives at customer sites and adjust accordingly.

Pick rails that don’t suck and work in 2-post and 4-post racks.

Use fans that are quiet. Spend a bit of time to add software support to control the fans appropriately. If you own a previous product I worked on with loud fans, I apologize. I really do.

There are lots of little details to take care of to finish a solid appliance offering. The application should be monitored by a watchdog (inittab, eventd) so when it goes down, it comes back up. Log file size and rotation should be set up. Cores should be constrained if runaway coring occurs. The application should start when the appliance is booted by default. It sounds obvious, but I’ve seen a few appliances that didn’t do this. Vendors should change /etc/rc.sysinit so that when fsck fails during boot, customers are instructed to call the vendor, or dropped into an emergency shell, rather than prompted for a root password. Vendors need to abstract configuration variables set by the users from their destination /etc files. It needs to be easy for the user to back-up, reset, and re-apply configuration settings.

Where Do We Go From Here?
Lots of applications are going to the cloud, which is fantastic. We’ll continue to see more of that. But we’ll still need to have software in our data centers. And it would be nice to see much of that software shipping as an appliance. I hate installing software applications on an OS for IT workloads. It’s stupid, it’s complex, and it’s largely unnecessary. Anyone who has installed a complex application knows how stupid it can be. Installing a TSM server package on a CentOS base is a quick 200 step process and the fact that even I could make it work means it was probably pretty easy.

Over time, we might find that some of the Cloud APIs are enabling compute resources with a different layer underneath than the traditional OS. A proper API underneath that extends onto physical and service offerings that can help standardize some of these appliance computing characteristics would be wonderful, but I don’t see how anyone can make money developing such a thing, so I am not optimistic that it will come to be.

So, remember, an appliance:

• Does one thing. It is focused and minimal.
• Is stripped down. There’s no bash shell for the user. The UI is focused, concise, web-based.
• Is lights-out. It recovers from failure. It is well behaved. It handles updates easily.
• Reduces burden on the user.

Vendors need to quit shipping software junk that they call an appliance. Customers need to demand more; it’s amazing how much bad software IT has to deal with. Stop adding to the pile.

Ship something good.

Many thanks to my colleagues and friends who reviewed this post before publishing.

Tonight a fellow asked me about who has the “best” rewards program for travel.  I don’t really know, but I do have a few tips and notes on my experiences.  To frame my experience, I’ve traveled about 100,000 miles per year for the last few years.  I am by no means an expert traveler, but I know more about this than I did 10 years ago.

So, in no particular order:

1. I really like the Choice Hotels rewards program.  It’s easy to earn free nights, and this chain includes Quality Inn, Sleep Inn, Comfort Inn.  I am not sure what the difference between the labels is (they seem about the same to me), but there’s thousands of these hotels in the US, they generally are clean and well-run.  Many of them have a slightly better breakfast than a donut + coffee on a folding table, and many of them have a few pieces of exercise equipment.  From a rewards perspective, it’s very easy to become a “Gold” member, which starts acceleration of points.  I’ve been in this reward program for 12 years and had no problem with redeeming dozens of free nights.
2. I also really like hotels.com.  There’s a lot of choice with hotels.com and I’ve had great luck with their rewards system, which is very simple–stay 10 nights, get one free.  There is a small caveat here in that the free room is actually a discount room if you stay at a higher end (priced?) hotel.  But I’ve gotten plenty of rooms for free with hotels.com.  Just beware when you check-in that hotels.com is owned by Expedia and the hotel might ask, “You booked on Expedia?”
3. I love Enterprise for car rentals.  However, they don’t have a rewards program that is useful.  The main reason to join their program is that large airports have a separate “Enterprise Plus” waiting line.
4. The mainstream US airlines have about the same rewards system in many ways.  You should pick an airline that is convenient for where you live (if you live in Charlotte, fly US Airways; Dallas, American, etc.) for the best direct flight scenario.  I personally hate United, but if you fly enough to get into their 1K program, you will be all set.  I generally only fly two airlines:  Virgin America for BOS<->SFO and US Airways for everything else.
5. While I love Virgin America, their rewards program sucks.  I’ve gotten several free tickets with them, but you have to spend a lot of money to pull this off.  Their rewards program is, in my opinion, far worse than American, US Airways, or United.
6. The key with a lot of travel is to pick with certain vendors and stick with them.  I can go into Quality Inn and tell them I spent 60 nights with them this year.  I can tell Enterprise I’ve had their cars 150 days in the last 365.  When things go wrong, this can carry a lot of weight.  Once you’re in elevated levels of frequent travel programs, you will often get a separate phone number for elite travelers for service–put this number in your cell phone and use it when things go wrong.  The manager at the SFO Enterprise knows me by sight/name.  I know many airline and hotel employees personally as well.
7. I’ve never had good luck trying to use points from one vendor with another (e.g, using Choice Hotels points for a discount at Enterprise).  YMMV.
8. Whenever I buy from a new vendor, I open a rewards account with that vendor.  I only have a few thousand points with some vendors, but eventually they will turn into a freebie.
9. I use TripIt to keep up with all my points (along with everything else TripIt does!).
10. I only carry one credit card that is tied to a points system.  It’s tempting to move cards around, but it’s not efficient for me to do so.
11. I try to stay within vendors from 6 or so groups of points for airlines and hotels.  This lets me optimize price without paying a premium for a particular vendor just to get points.
12. A friend of mine has had luck buying bundles with Orbitz for lower prices.

I’m sure others know more about this than I do.  I haven’t read up on travel program tips, these are just observations I’ve picked up on along the way.

On May 21st at around 7:45 am, I locked the keys to my rented Hyundai Sonata 2011 in the trunk of the car. I have never done this before, but I was in a hurry, stressed about some things I needed to take care of that day, and somehow dropped the keys into the trunk before slamming the lid. I am always paranoid about locking my keys on the other side of a barrier from me, and I did indeed check for keys before I closed the lid—unfortunately, I did not check the keys closely enough that they were the right keys.

So I called Enterprise. I am a very loyal Enterprise customer—I almost always rent from Enterprise—mostly due to their stellar customer service. I drive rental cars about 20-25 weeks per year. Even when I get average service at certain Enterprise locations, it’s generally much better than what I get at other rental companies.

Enterprise informed me that AAA would cost me $65 since it was my fault. No problem.

AAA showed up in about 50 minutes and did a quick demo of how easy it is to break into a car. I was relieved that I would be able to get on my way, finally. It was now a little before 9 am. With the car alarm blaring, AAA opened the driver’s side of the vehicle and pressed the trunk release button.

Nothing happened.

It turns out the Sonata is a very smart car. When the alarm is activated, the release button in the passenger compartment for the trunk is disabled.

So if you lock your keys in the trunk, you had better have a second set of keys or a locksmith.

I called a locksmith. “We do not have a way to pick the electronic locks on the Sonata.”

I called Enterprise back and went through several rounds with several people for the next TWO HOURS. Yes, the keys are in the trunk. No, the trunk release doesn’t work. It’s electronic. No, I can’t lower the backseats—the release for the seats is in the trunk.

At this point, Enterprise gave me two options:

1. They could tow the car somewhere. The customer rep was dubious I would ever get the stuff in the trunk back. I had some expensive clothes in the trunk and wasn’t a fan of this option.
2. Enterprise would overnight the 2nd set of keys from their key storage in another state to the airport location and I could pick it up on Saturday, the 22nd. This sounded better, though I wouldn’t be able to pick up the keys until the 23rd due to my schedule.

In the meantime, a co-worker gave me a ride to another Enterprise location where I rented another car. Initially, the agent told me he had a Hyundai Accent to rent me and I said OK. Then after going through the papers, he said we’d have wait while someone drove the car over. It turned out the only car he had was a Jeep. He then wanted to double the rate on the Jeep over the Accent.

At this point, I had been doing nothing for the last 4 hours other than deal with Enterprise and was starting to lose my patience.

I rented the Jeep at a reduced price.

Because most of my clothes were in the trunk, I had to go buy new clothes and shoes.

On Sunday, I drove to the airport location, 90 mile roundtrip, to pick up the keys. The agent told me he didn’t have them, but he had seen an email about it.

I wasn’t too surprised—FedEx doesn’t deliver to all commercial locations on Saturdays. I asked him to call me on Monday when the keys came in.

On Monday morning—72 hours after this all started—I called Enterprise to ask what was going on and ended up leaving voicemail for the woman who had called on Friday (Crystal), since no one at the location was answering the phone.

Around 3pm, I got a voicemail informing me that the keys had come in. I did another 90 mile roundtrip to pick up the keys.

They were the right keys! I could open the doors without the alarm sounding! I could get my stuff out of the trunk!

I called my co-worker to see if he could pick me up at the local Enterprise location so I could get rid of the Jeep. As I was driving away from the Sonata, I realized that I never tried starting the Sonata. I did a U-turn.

The ignition is locked, apparently as an extra layer of security. I have all the keys for the car in my possession and none of them will start the car. I do not have the owner’s manual.

A few observations:

1. Hyundai has significantly over-engineered the security of the trunk. If I am a thief, I don’t care about the cosmetic condition of the car and will use a crowbar or explosives to open the trunk if I am hell-bent on it.
2. If I have the legitimate keys, the ignition should work—I understand it has been disabled by the alarm, but if I had the legitimate keys for starting the car, I would have also used them to unlock the car.
3. Enterprise reps know too little about the cars to support them. There was no reason to send out AAA. Sending the keys to get my stuff was good, but not effective for driving the car itself. It’s been 4 days now and I can’t drive the car.
4. Yes, it is my fault for locking the keys in the trunk, but that should be no more than a 2 hour aggravation. I’ve exceeded that threshold by 50x at this point.
5. The Enterprise folks have tried very hard to make this right, but they have not been effective as of yet. I really like Enterprise, and want to get this resolved, but at this point it is physically dreading to consider that I have to spend more time on this. I rent cars as a tool to enable me to do my job. I have a lot of hours and wasted money on this car at this point.
6. This trip was going to involved a lot of driving, which I knew up front, so I spent a little extra to get the Sonata–comfortable, excellent XM and Bluetooth capabilities, good gas mileage—I have none of those features in the vehicle I am currently in, and yet I am paying for two cars.
7. The absurdity of this story is somewhat magnified by the car rental situation, but I can see the scenario reproduced by a traveler far from home (where the 2nd set of keys would be kept) with the same result, as far as the car itself goes. This is a fundamental problem with this car.
8. With all the RFID and keyless entry stuff, why can’t the car detect the keys are in the trunk and take appropriate action?
9. With all the other software in this car, why not reset the ignition lock after 24 hours of peace from the alarm? Why go into a locked state forever?
10. Until all of this happened, I was going to replace my Accord with a Genesis or Sonata. Now I think I need to look at other options.