Mitch's Blog

I want to tell you about my new project, Bit Lasso Reveal.

For years, I’ve worked on slide decks. Hundreds and hundreds of slide decks–sales decks, strategy decks, roadmap decks, investor decks. I did a few presentations before founding Pancetera, but my real introduction to PowerPoint was building the pitch deck for that company. Building a presentation that would raise millions of dollars took an entire summer of monkeying with PowerPoint, and after a few weeks, I remember telling my cofounders I suddenly understood how to make decent decks. I didn’t, really, but I had gotten a lot better at it.

I ended up with over 100 versions of that original pitch deck on my laptop. What is it about slides that causes so much version sprawl? Every time we present, we make changes–the audience was confused by a picture, something felt out of order, a key element was missing. But we never know for sure: did we make the deck better, or worse? Let me throw some of these slides in the back.

Collaborating also leads to sprawl, with copies of decks floating in email, cloud drives, attached to tickets or Wikis, posted in Slack–it’s a huge mess.

So switch to Office 365 or Google Slides and be done, right?

Well, not really–these just move the problem around, but they don’t change how we work with slides.

In fact, slide search is a recall problem. “I know I made a green competition slide last fall.” How do you search for that in Spotlight or Windows search? Well, you could type in a few key words, but everyone who has done this knows what you get:

A bunch of slide covers. You can see in the screenshot I have lots of versions of a deck I was working on a few years ago. How can I quickly find the specific slide I want? I can’t! I have to open the decks up and look through them manually.

This is madness!

But with Bit Lasso Reveal, I can immediately see all the “How It Works” slides I have for these decks. I don’t have to open the decks up, and the slides are automatically stacked into similar piles. You can see I have a number of variations of these slides, as I evolved the concept over months of thinking about it, and now I can see a complete history of those thoughts–perhaps at some point in the past, a concept was removed from a slide and I’d like to get it back–with Reveal, I can identify those changes over time.

The best part: Reveal doesn’t require any changes to how you work. You can carry on in as tidy or as messy of a fashion as you’d like. Go ahead and put slides in email, on your desktop, in Dropbox, or OneDrive, or all of them–Reveal doesn’t care. Reveal automatically finds your slides and organizes them in its own database without altering your content. Reveal can even associate slides that you have in both Keynote and PowerPoint versions!

Reveal is available now on Mac, and I know a lot of PowerPoint activity happens on Windows, so you could imagine I’m working on that. Comments? Feedback? Suggestions? I’d love to hear from you. Write to mitch@bitlasso.com or tweet at @bitlasso

P.S.: As a bonus, here’s a picture of the slide comparison:

I saw a tweet from Chris Mellor referencing a Gartner conclusion that mid-market IT will go to the cloud:

1) Gartner: Many businesses, especially in the midmarket, will eventually migrate away from running their own data centres…

— Chris_Mellor (@Chris_Mellor) May 19, 2015

Today, Storage Newsletter‘s headline quotes an IDC report that personal and low-end storage sales (less than 12 bays) have declined 6.4% y/y. Some dimensions of the business sank 20% y/y.

What happened in the last year? Do people create less data than they did a year ago? Isn’t data storage growing tremendously?

What is changing is where people create and keep their data. As applications move to the cloud, the data goes with it. From Salesforce to consumer photos, from corporate email to songs, all of this stuff is in someone else’s data center.

I have about 100 TB of disks in my house across six fast hardware RAIDs, but all of my critical working set lives in the cloud. The cloud pricing for large amounts of data (e.g., 1 TB) is so cheap that it’s free or bundled (Office 365, Flickr). Dropbox stands alone as an outlier to a priced service and it’s not that expensive–certainly I cannot buy a 1 TB drive and operate it for 1 year at the price point that Dropbox offers.

Generally, IT vendors fail to deliver on simplicity; it’s not in their vocabulary. I’ve been in those meetings–hundreds of them, actually–where engineers want to offer every option for the customer and for some reason (lack of vision?) the product manager lets it happen. The problem with these meetings is that everyone in them usually forgets that while the product is the most important thing in the lives of the folks creating the products, the customers have other things on their minds.

So we end up with these overly complex IT products that are impossible to use. Quick, how do you set up Postgres database backups with Tivoli? I have no idea but I know it will take a few hours to figure it out (if I am lucky). The brilliance of Amazon’s cloud user interface is that (1) the details are hidden and (2) the user is presented with just the critical options. Do you want to back up this database? Sure! Great, when? Hey, you know, I don’t really care. Just keep the backups for 30 days.

One of the most powerful things about AWS is that significant infrastructure is under a single pane of glass. This has been the Holy Grail of IT but never realized. OpenView, System Center, vCenter, TSM–everyone wants to do it, but few organizations pull it off, likely due to a mix of political, technical, and economical reasons.

The best part of Gmail going down is that it’s not my problem to bring it back online. Remember when you worked at a place that ran Exchange and the guy in charge of Exchange was always on edge? The only reason that guy is on edge now is that he is waiting for a call to see if he got the job at a place that has switched to Gmail.

The data center of the future for most mid-market companies is a single rack consisting of network connectivity, security devices, and WAN acceleration devices. No servers or standalone storage–with applications in the cloud, the only thing needed locally is data caching to augment the WAN overhead and maybe provide short-circuit data movement among local peers. This single rack will fit into a closet.

IT will still exist; these cloud applications will still need support, maintenance, and integration–and the network issues will be as challenging as ever.

But anyone who is building IT products for on-site installation is facing a significant headwind if you’re not enabling the cloud.

So you’re about to graduate from college and while looking for a job, and someone expresses surprise when you confess that you’re not well-versed at Linux. Uh oh.

Everyone who expects to work in computing should know some basics about Linux. So much of the world runs Linux these days–phones, thermostats, TVs, cars.

Here’s a list of tasks that the young computer scientist should be able to do with Linux. The goal isn’t for you to be able to get a job as a “sysadmin” but to have a general familiarity with enough different things that you can solve real world problems with a Linux system. Of course, much of this applies to Mac OS X, too.

1. Install CentOS or Ubuntu into a virtual machine on your Windows or Mac desktop/laptop. Open a Terminal window.
2. You’ll probably get most of your help through Google searching, but on the command line, you can get help with specific commands by using the man command. E.g., man ls
3. Basic file navigation: ls, cd, pwd, pushd, popd, dirs, df, du, mv. Be careful with rm.
4. Basic editing with vim (open a file, save it, close it without saving, edit it, copy/paste with yank, jump to a specific line number, delete a word, delete a line, replace a letter.) (You can use nano while you’re coming up to speed on vim.)
5. Use grep, less, cat, tail, head, diff commands. Use with pipes. Use of tail -f, less +F, tail -10, head -5 (or other numbers) is handy.
6. tar and gzip to create and expand archives of files.
7. Use sed, awk — replace the contents of a file, print the column of a file.
8. Command-line git commands to checkout, edit files, commit, and push back to a remote repository (e.g., Github).
9. Basic process navigation: ps, top, kill, fg, bg, jobs, pstree, Ctrl-Z, Ctrl-C.
10. Unix permissions: chmod, chgrp, useradd, sudo, su; what do 777, a+rw, u+r mean, how to read the left column of ls -l / output.
11. Simple bash scripts: Write a loop to grep a file for certain output, set command aliases
12. Compile a simple C program with gcc. Use gdb to set breakpoints, view variables in a C program being debugged (where, bt, frame, p).
13. Use tcpdump to watch HTTP traffic to a certain host.
14. Understand /etc/rc.d and /etc/init.d scripts
15. A basic understanding of /etc/rc.sysinit
16. Attach a new disk and format it with fdisk or parted and mkfs.ext4. Run fsck. mount it. Check it with df.
17. Know how to disable selinux and iptables for debugging. (service, chkconfig)
18. How to use the route, ifconfig, arp, ping, traceroute, dig, nslookup commands.
19. Write an iptables rule to forward a low number port (e.g, 80) to a high number port (e.g, 5000). Why would someone want to do this?
20. A cursory understanding of the filesystem layout — what’s in /etc, /bin, /usr, /var, etc.
21. A cursory understanding of what’s in /proc.
22. Configure and use SSH keys for automatic login to another host.
23. Forward a GUI window over SSH with X11
24. Reboot and halt the machine safely (shutdown -h now, reboot, halt -p, init, etc commands)
25. yum and apt-* commands (CentOS and Ubuntu, respectively)
26. Modify boot options in grub to boot single user, to boot to a bash shell

For extra credit:

1. The find command is a complicated beast, but simple to get started with.
2. Copy files over SSH with scp.
3. The dd command is useful for dealing with a variety of tasks, such as grabbing an image of a disk, getting random data from /dev/urandom, or wiping out a disk, and so on. Also be aware of the special files /dev/zero and /dev/null.
4. Figure out how to recover a forgotten root password.
5. Disable X11 and be able to do these tasks without the GUI.
6. Do the same tasks above on a FreeBSD machine.
7. Without the GUI, configure the machine to use a static IP address instead of DHCP.
8. Use screen to create multiple sessions. Logout and re-attach to an existing screen session.
9. Write a simple Makefile for a group of C or C++ files.
10. What does chmod +s do? Other special bits.
11. netstat, ncat, ntop.
12. ldd, strings, nm, addr2line, objdump
13. Grep with regular expressions
14. What’s in /etc/fstab?
15. history, !<number>, !!, !$, Ctrl-R

Books to peruse:

1. Unix Power Tools
2. sed & awk
3. bash Cookbook
4. Learning Python Every computing professional should know a simple scripting language that ties to the OS for more complex scripts than are rational than bash; python is an excellent place to start.
5. Advanced Programming in the UNIX Environment, 3rd Edition (be sure to get the latest edition)
6. If you’re interested in networking, be sure to read TCP/IP Illustrated, Volume 1: The Protocols (2nd Edition)
7. You probably took an OS class. While Tanenbaum and Silberschatz write great books, if you want to know Linux internals better, Rubini’s device driver book is an excellent read. There is a 4th edition coming later this year. Linux Device Drivers, 3rd Edition

In the last post, I mentioned Conway’s Law:

organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations.

Dr. Conway was referring to people in his article–but what if we substitute “organization” with your product’s source tree and “the communication structures” to how functions and data structures interact? Let’s talk more about Conway’s Law in the context of source tree layout.

Many products of moderate complexity involve multiple moving parts. Maybe a company has a cloud service (back end) and a Web UI (front end). Or a back end and a mobile front end. Or a daemon, some instrumentation scripts, and a CLI. Or a firmware and a cloud service.

I’ve had my hands in a number of companies at various depths of “hands in.” Those who lay out a source tree that fully acknowledges the complexity of the product as early as possible tend to be the ones who win. Often, a company is started to build a core product–such as an ability to move data–and the user interface, the start-up scripts, the “stuff” that makes the algorithm no longer a student project but a product worth many millions of dollars–is an afterthought. That’s fine until someone creates a source tree that looks like this:

trunk/
	cool_product/
		main.c
		error.c
		util.c
		network.c
	stuff/
		boot.c
		cli.c
		gui.c

What’s wrong here? Presumably, some of the code in util.c could be used in other places. Maybe some of the functions in error.c would be handy to abstract out as well. An arrangement like this in which the cool_product is a large monolithic app likely means it’s going to be difficult to test any of the parts inside of it; likely modules and layering are not respected in a large monolithic app. (Note that I am not saying it’s impossible to get this right, but I am saying it’s unlikely that tired programmers will keep the philosophy in mind, day in and day out.)

A slightly different organization that introduces a library might look as follows:

trunk/
	lib/
		util.c
		error.c
		network.c
		tests/
			Unit tests for the lib/ stuff
	prod/
		cool_product/
			main.c
		gui/
		cli/
	tools/
		Build scripts or related stuff required,
		code generation, etc.
	platform/
		boot.c

As a side effect, we can also improve testing of the core code, thus improving reliability and regression detection. Ideally, the cool_product is a small amount of code outside of libraries that can be unit tested independently.

More than once I’ve heard the excuse, “We don’t have time to do this right with the current schedule.”

“I don’t have time for this” means “This isn’t important to me.” When you say, “I don’t have time to clean up the garage,” we all know what you really mean.

I was incredibly frustrated working with a group who “didn’t have time” to do anything right. Years later, that company continues to ship buggy products that could have been significantly less buggy. A few weeks of investment at the beginning could have avoided millions of dollars of expense and numerous poor reviews from customers due to the shoddy product quality. And it all comes back to how hard (or easy) it is to use the existing code, i.e., the communication structure of the code.

If you don’t have time to get it right now, when will you have time to go back and do it right later?

Getting it right takes some time. But getting it wrong always takes longer.

Teams with poor source tree layout often end up copying and pasting code. Sometimes a LOT of code. Whole files. Dozens of files. And as soon as you do that, you’re done. Someone fixes a bug in one place and forgets to fix it in another–over time, the files diverge.

If you’re taking money from investors and have a screwed up the source tree layout, there are two ethical options:

1. Fix it. A week or two now will be significantly cheaper than months of pain and glaring customer issues when you ship.
2. Give the money back to the investors.

If you’re reading this and shaking your head because you can’t believe people paint themselves into a corner with their source tree layouts, I envy you! But if you’re reading this and trying to pretend you don’t face a similar position with your product, it might be time to stop hacking and start engineering by opening up the communication paths where they should be open and locking down the isolation and encapsulation where they should not. This holds true for any language and for any type of product.

Do you find yourself saying things that are so absurd you’d not believe it if you hadn’t lived through it? I’ve had a few of those experiences.

I once worked at a software company that had a strong unit test culture. There were specifications for everything and tests for everything. Sounds great, right? Unfortunately, despite extensive review meetings, the specifications never survived the few first days of coding. The unit tests were detailed and thorough–fantastic! But no one ever ran the whole product in-house. QA didn’t. Developers didn’t. The company didn’t use its own product because “it is too complicated.” After I left the company, I had a problem that the company’s product was designed to solve. As a shareholder, I was interested in the company having customers, and happy to be one. I downloaded the product and installed it. When I fired it up, the product crashed about 3 lines into main(). No one had ever used the product. It’s great to prove that the pieces work individually, but your customers will be using them when they are all bolted together. You have to make sure the holes line up.

I also once worked at a company with the exact opposite approach. There were no unit tests of any kind. No libraries, no modules, nothing. The only way to test anything was to run the product. This meant there were very few crashes on the third line of main(), but it also made it impossible to know whether or not any of the pieces of the code work independently. If you can determine whether pieces work independently, you can track those results, tie them to check-ins, and suddenly understand a heck of a lot about what changes in your source control impacted what areas of the product. You can track down not only data corruption but also which modules hit performance issues, whether within a unit test context or within an integration context within a customer environment.

The first company had a hard time getting customers to be successful with the product. The second company had difficulty predicting when a release would be ready. The former is easier to fix–start using the product internally and also fix the QA organization to include system tests. The latter likely entails an expensive refactor and reflects deeper cultural issues of a hacking organization rather than an engineering organization. Neither extreme is healthy.

How many times have you seen the UPS landing page below?

Why is it that neither FedEx nor UPS know about geolocation of IP addresses? Even if geolocation was used just as a hint or an obvious way was provided to change it if the geolocation went wrong, far fewer folks would hit these screens. And after you make a selection, the UPS site just sits there and does nothing until you click the small blue botton.

I hate software that makes me deal with pop quizzes and stupid modal dialogs. Hey, where are you? Would you like fast or small? Want to update now? Restart Firefox now? Hey, you will need to reboot after you install this, OK? Downloading this RPM will use 30 KB of disk space, sound good? Really quit? There’s an item on your calendar coming up in 10 minutes, you won’t be able to click anywhere in Gmail until you click this OK button… on all of your computers. I couldn’t copy that to the clipboard. Send a bug report to Apple? This document has ColorSync profile 1, and this other one has ColorSync profile 2. Would you like me to alert when submitting a form to an unencrypted site? There was an error with your request, try again later. All the downloadable content has downloaded. I am going to go ahead and join you to the unsecured network ‘linksys’. Hey, couldn’t backup this computer. Holy shit, there’s new software available, I don’t care if you’re watching a movie fullscreen (iTunes + Notification Center). Psst, mind if I phone home real quick? Oh, by the way, the iTunes terms and conditions have changed and I suddenly can’t remember your credit card number.

Who wouldn’t hate a machine that talks to you this way, all day, every day? I am trying to get some work done here, and you’re telling me all this out-of-band shit and very little of it is useful to me right now.

Please stop asking me questions. I don’t know what the answer is and you’re interrupting my train of thought, which I value far more than pondering any of the above questions.

There are lots of examples of how to use the Ethernet Wiznet chips with Arduino, whether as Ethernet shields or as Ethernet Arduinos on a single board. Unfortunately, most of these examples hard-code the hardware (MAC) address, which can make things painful if you’re building more than one device and running them on the same network.

The code snippet below is a more convenient approach. You can setup a prefix (DEADBEEF in the example below) for the hardware address and the last two bytes are set randomly on first boot. The hardware address is stored in EEPROM (7 bytes are needed, 1 for a flag indicating that the next 6 bytes are properly populated).

The bytes->String conversion below is a bit ugly but I didn’t think I wanted the overhead of sprint in this. It is probably not worth the trade off. (0x30 is ‘0’ and 0x39 is ‘9’. Adding 0x07 skips over some ASCII characters to ‘A’.)

Some serious caveats: There’s only two bytes of randomness here. You might want more. Ideally you would have a manufacturing process, but if you’re just building six devices, who cares? Clearly you would never use this approach in a production environment, but it’s easier than changing the firmware for every device in a hobby environment. You could also use a separate program to write the EEPROM hardware address and keep this “manufacturing junk” out of your main firmware. These issues aside, my main requirement is convenience: I want to be able to burn a single image onto a new board and be up and running immediately without having to remember other steps. Convenience influences repeatability.

#include <Ethernet.h>
#include <EEPROM.h>

// This is a template address; the last two bytes will be randomly
// generated on the first boot and filled in.  On later boots, the
// bytes are pulled from EEPROM.
byte NETWORK_HW_ADDRESS[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0x00, 0x00};
String NETWORK_HW_ADDRESS_STRING = "ERROR_NOT_FILLED_IN";

// These are commented out so that this code will not compile
// without the reader modifying these lines.  If you are using
// EEPROM code in your program already, you need to put the
// network address somewhere that doesn't collide with existing use.
//#define EEPROM_INIT_FLAG_ADDR 0
//#define EEPROM_HWADDR_START_ADDR 1

// Call this from your setup routine (see below)
void
initEthernetHardwareAddress() {
    int eeprom_flag = EEPROM.read(EEPROM_INIT_FLAG_ADDR);  
    int i;
    Serial.print("EEPROM flag is " + String(eeprom_flag));
    
    if (eeprom_flag != 0xCC) {
        NETWORK_HW_ADDRESS[4] = random(255);
        NETWORK_HW_ADDRESS[5] = random(255);  
        
        // write it out.
        Serial.println("Writing generated hwaddr to EEPROM...");
        for (i = 0; i < 6; i++) {
            EEPROM.write(EEPROM_HWADDR_START_ADDR + i + 1,
                         NETWORK_HW_ADDRESS[i]);
        }

        EEPROM.write(EEPROM_INIT_FLAG_ADDR, 0xCC);
    } else {
        Serial.print("Reading network hwaddr from EEPROM...");
        for (i = 0; i < 6; i++) {
            NETWORK_HW_ADDRESS[i] =
                EEPROM.read(EEPROM_HWADDR_START_ADDR + i + 1);
        }        
    }
    
    char hw_string[13];
    hw_string[12] = '\0';
    for (i = 0; i < 6; i++) {
        int j = i * 2;
        
        int the_byte    = NETWORK_HW_ADDRESS[i];
        int first_part  = (the_byte & 0xf0) >> 4;
        int second_part = (the_byte & 0x0f);
        
        first_part  += 0x30;
        second_part += 0x30;
        
        if (first_part > 0x39) {
            first_part += 0x07;
        }
        
        if (second_part > 0x39) {
            second_part += 0x07;
        }
        
        hw_string[j] = first_part;
        hw_string[j + 1] = second_part;
        
    }

    NETWORK_HW_ADDRESS_STRING = String(hw_string);

    Serial.println("NETWORK_ADDR = " + NETWORK_HW_ADDRESS_STRING);
}

void
setup() {
    // first call the usual Serial.begin and so forth...

    // setup the Ethernet hwaddr before you start using networking
    initEthernetHardwareAddress();

    int dhcp_worked = Ethernet.begin(NETWORK_HW_ADDRESS);

    // ...
}

I have been a big fan of xxdiff for many years–so much so that I wrote a Mac GUI clone for it as a mental exercise while on vacation in 2006. Unfortunately, xxdiff with svn has always sucked–it requires awkward scripts to plug it into svn, and those don’t tie into ‘svn diff –diff-cmd ‘ very well. A buddy recommended I switch to tkdiff, and it’s not bad at all. This morning I threw together a wrapper to point my ~/.subversion/config at for 2 file diffs. Since I do a lot of work in both SSH and in X11 terminals, I wanted something that could relieve me of typing ‘–diff-cmd diff’ all day long.

#!/bin/bash
#
# Script to enable 'svn diff' as a GUI diff in X11 or a CLI diff
# on the command line (or if stdout is redirected).
#
# I couldn't get tkdiff to take a -geometry argument. I ended up
# setting my geometry in tkdiff's opts dictionary.
#
# Arguments coming in from svn diff:
#
# -u -L src/hello.cpp (revision 1234) -L src/hello.cpp (working copy) src/.svn/text-base/hello.cpp.svn-base src/hello.cpp
#

dash_u=$1
label1="$2 $3"
label2="$4 $5"
file1="$6"
file2="$7"
# echo "Comparing _$6_ and _$7_"
cmd=diff
if [ -t 1 -a -n "$DISPLAY" ]; then
    cmd=tkdiff
fi

$cmd "$label1" "$label2" "$file1" "$file2" 
RC=$?

exit $RC

Colleagues who have worked with me in the last ten years know I have some religion about appliances. To a large degree, I believe that most enterprise software that is installed in a customer’s data center should be packaged as an appliance. The embodiment of this could be as a physical appliance or as a virtual software appliance, but either way, it’s an appliance.

My sense is that appliances, especially virtual appliances, are picking up momentum. We’re seeing more and more software companies offering virtual appliances on the market today and more products that could be packaged as software instead packaged as a hardware offering.

There’s a problem though.

Most of these appliances are crap.

What is an Appliance?
It would be cliché to turn to Merriam Webster for a definition, so let’s consider what the Oxford American Dictionary says an appliance is:

ap-pli-ance n. 1 a device or piece of equipment designed to perform a specific task

So if we’re building an enterprise IT appliance, it’s going to be a virtual machine or a physical machine to perform a specific task.

In my mind I’ve always given Dave Hitz, co-founder of NetApp, credit for defining an appliance, but in this interview [PDF], he gives credit to the model Cisco established:

Hitz: […] We were modeling ourselves after Cisco. If you look before Cisco, people used general purpose computers, UNIX workstations, Sun workstations for their routers. Cisco came along and built a box that was just for routing. It did less than a general purpose computer and so the conversation we kept having with the VCs, they would say, “So your box does more than a general purpose computer so it’s better?” And we’d say, “No, no, it does less.” And they’d go, “Well, why would I pay you money to start a company that does less?” And we said, “No, no, it’s focused. It’ll do it better.”

He continues:

The other analogy that we used was a toaster. If you want to make toast, you could go put in on your burner, you can put it in the oven and it’s certainly the case an oven’s a lot more of a general purpose tool than a toaster. A toaster? What good’s a toaster? All it does is make toast. But, man, if you want toast, a toaster really does a good job, you know? Try making toast in your oven and you’ll discover how awesome a toaster is. And that really, for storage over the network, it was that idea of making a device to do just that one thing, focused. Not run programs, not do spreadsheets, not be a tool for programmers, just store the data.

I’ve always stuck by the toaster model when thinking about appliances for customers.

So what does this mean for the user?
Let’s talk about product requirements.

When a vendor ships an appliance to the customer, everything is included: The OS, the management tools, and the application. The beauty of this approach is that the customer no longer has to install software. The customer is no longer responsible for dealing with OS patching and ensuring the right OS patches are installed that match with the requirements of the application.

The user interface should be stripped down and minimal. The user should never see a general purpose bash shell prompt. The appliance should be as self-contained as reasonable. Configuration should be as minimal as possible. IP address and related information, LDAP or user set-up, NTP and timezone, email and SNMP, application-specific coordinates, and that’s it! This can be implemented in a small X11 set-up GUI (Nasuni does an excellent job here) or a simple ncurses-based text wizard (similar to what ESXi has).

The user should never be told to edit files off in /etc or manually tweak key-value pairs to get the basic application up and running. Some application programmers get off on connecting to a customer-provided database. In a particularly large installation, this might make sense. To store 500 rows of data, it does not.

An appliance is likely Linux-based these days, as it should be. Software updates should come from an update server provided by the vendor. From the user’s perspective, there is no apt-get or yum details to worry about; it’s a simple matter of clicking a ‘Upgrade’ button. Anyone shipping a Windows-based appliance should only do so for very specific reasons (e.g., need access to Windows APIs).

The GUI for the appliance should be web-based. These days, it needs to be accessible from a variety of browsers. Product managers need to understand the average IT guy has more things to worry about than a single product, and hints need to be provided on how to find the management UI for the appliance. I love how HP handles this on their switches and I adopted their model at my last company: Every login console tty included the IP address and how to find the GUI; both shell and GUI login screens included the default account name and password, right there. The last time you installed a home router, did you have to fire up Google and search for the default user name and password? Or did you read the documentation? Did you read the documentation for Tetris? Why should the default user name and password be harder than playing Tetris?

DHCP should be the default on a first boot, and active on all interfaces with a link. Anyone shipping an appliance without DHCP in this day and age needs to get with the times.

Any CLI should be succinct and hierarchical. Enterprise IT gets tricky, no matter how we simplify it, and users need a way to drill into the available commands. The CLI needs to balance “learning” with speed of use. The reality is, most IT folks are using a lot of different products throughout the day. Make yours the one that is easiest for the admin to jog his mind on how to see replication status (or whatever).

When it comes to lights-out management, product managers and engineers must understand that the product is not the center of the end-user’s universe. Problems and reports should be available via email. With a generated email, the subject line is critical and should be succinct but descriptive. “2 alerts (product name)” with a From: field of the hostname is a good start at a subject line. Failures and recovery from failures should be as automatic as possible. Customers should never have to login to click something in a GUI that the product could have run itself to recover from an error.

Although not strictly appliance-related, let’s also talk about sockets and ports of the TCP/IP variety. When it comes to networking, products should be designed with WANs, VLANs, routers, closed ports, and different views of networks when communicating with other instances of the same product. Don’t rely on ICMP pings being available. Don’t rely on your connection target matching the listening target coordinates. Have flexibility. Design your network protocols for the simplest firewall configurations. In a spoke/hub topology, all the spokes should reach into the hub–that is, all spokes open sockets on a port the hub is listening on, even if the logical client/server is the other direction.

Shipping a Virtual Appliance
This is where I see the most sin in the market today. So many folks install Ubuntu, install their software, shutdown the VM, zip it up, ftp it over to their web site and call it a “virtual appliance”. This is a VMDK full of crap. /root/.bash_history is still populated, as are DHCP cache files, log files, and other things that should be scrubbed. There is no OVF installer file. Old snapshots of the VM are included. What kind of embarrassing stuff is in there?

Let me state up front that OVF is one of the dumbest things in the VMware world. There is no need for it to exist; everything that it does could have been accomplished much more simply. But VMware doesn’t like to make things easy, so we’re stuck with it. On the bright side, the user experience of installing an OVF appliance into a VMware environment is awesome. Thus, vendors need to use it.

From a mindset perspective, the vendor needs to get into a mental mode of a manufacturing process to scrub the VM prior to OVF packaging. Things need to be set to “factory defaults”.

With both physical and virtual appliances, the size of Linux distribution should be minimal. This is more important with the virtual appliance distribution, since customers will be downloading it. As a vendor, you can pick a few approaches on how you do this, but in general, reduce the number of packages you’re including to the bare minimum. You need to tune your CPUs and RAM to the minimum required for your application and let customers tune up from there. If you’re shipping both virtual and physical appliances, you need to understand these are different beasts. How will you sell them? My bet is that you’re pushing virtual appliances into remote-office/branch-office use cases, so size the requirements for the VM appropriately. Size your VMDKs down to something reasonable or use thin-provisioning. Note that OVFs with thin-provisioning will not work on ESX versions less than 4.1 or 4.5 (I can’t remember exactly which versions are broken for thin-provisioning with OVF).

Shipping a Physical Appliance
Ten years ago, I worked on my first appliance product. One of the problems we had with this physical appliance when we went out in the field was that we needed a screwdriver to get it in the rack. Customers sometimes had to spend 10 minutes searching for tools if they were installing it themselves. We started to include a multi-tip screwdriver with the product in the shipping carton. In 2010, my new company had the opportunity to buy a few systems from the previous company. The screwdriver was still included. If I ever do a physical appliance again, I will include the screwdriver. What’s $10 on a $50,000 system anyway? Don’t forget this is a branding opportunity–put your logo or URL on the handle.

If your system is heavy, ship it on a pallet. If you’re on the border (70-80 lbs), pay attention to the condition that the system packaging is in when it arrives at customer sites and adjust accordingly.

Pick rails that don’t suck and work in 2-post and 4-post racks.

Use fans that are quiet. Spend a bit of time to add software support to control the fans appropriately. If you own a previous product I worked on with loud fans, I apologize. I really do.

There are lots of little details to take care of to finish a solid appliance offering. The application should be monitored by a watchdog (inittab, eventd) so when it goes down, it comes back up. Log file size and rotation should be set up. Cores should be constrained if runaway coring occurs. The application should start when the appliance is booted by default. It sounds obvious, but I’ve seen a few appliances that didn’t do this. Vendors should change /etc/rc.sysinit so that when fsck fails during boot, customers are instructed to call the vendor, or dropped into an emergency shell, rather than prompted for a root password. Vendors need to abstract configuration variables set by the users from their destination /etc files. It needs to be easy for the user to back-up, reset, and re-apply configuration settings.

Where Do We Go From Here?
Lots of applications are going to the cloud, which is fantastic. We’ll continue to see more of that. But we’ll still need to have software in our data centers. And it would be nice to see much of that software shipping as an appliance. I hate installing software applications on an OS for IT workloads. It’s stupid, it’s complex, and it’s largely unnecessary. Anyone who has installed a complex application knows how stupid it can be. Installing a TSM server package on a CentOS base is a quick 200 step process and the fact that even I could make it work means it was probably pretty easy.

Over time, we might find that some of the Cloud APIs are enabling compute resources with a different layer underneath than the traditional OS. A proper API underneath that extends onto physical and service offerings that can help standardize some of these appliance computing characteristics would be wonderful, but I don’t see how anyone can make money developing such a thing, so I am not optimistic that it will come to be.

So, remember, an appliance:

• Does one thing. It is focused and minimal.
• Is stripped down. There’s no bash shell for the user. The UI is focused, concise, web-based.
• Is lights-out. It recovers from failure. It is well behaved. It handles updates easily.
• Reduces burden on the user.

Vendors need to quit shipping software junk that they call an appliance. Customers need to demand more; it’s amazing how much bad software IT has to deal with. Stop adding to the pile.

Ship something good.

Many thanks to my colleagues and friends who reviewed this post before publishing.

I resisted writing this post as long as I could. I really did. I’ve always been so excited about Drobo and everything they were doing to enable anyone to have the benefits of redundant blocks in a storage system. I was super excited when Drobo started going up-market with the introduction of the Drobo Pro a few years ago and later the Drobo Elite. It was going to be an exciting ride to see where Drobo took things.

Recently I made the mistake of upgrading to the new Drobo Dashboard, the management tool for monitoring one or more Drobo boxes. I was struggling to get my Mac Pro to see my Drobo after installing an SSD and pulling in a recovered OS configuration.

In any event, I ended up with the new Drobo Dashboard. You can tell it was designed by the marketing department because the Drobo name and logo appear at least 8 times in this screenshot. Do you think this reflects the voice of the customer? “Your UI is great, but I really wish you had the name of your company up on the screen in more places. One or two is just not enough!”

There are other problems with the look of this interface, of course–I have no other apps that look like this or behave like this (fact), it’s ugly (opinion), it’s hard to read (fact: white text on a black background is a bad idea)…. I could go on, but it’s too painful and frankly I feel bad. I’ve pumped out my share of bad user interfaces over the years, but none of them had the vendor name on them in 8 places in a single view.