Mitch's Blog

In the last post, I mentioned Conway’s Law:

organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations.

Dr. Conway was referring to people in his article–but what if we substitute “organization” with your product’s source tree and “the communication structures” to how functions and data structures interact? Let’s talk more about Conway’s Law in the context of source tree layout.

Many products of moderate complexity involve multiple moving parts. Maybe a company has a cloud service (back end) and a Web UI (front end). Or a back end and a mobile front end. Or a daemon, some instrumentation scripts, and a CLI. Or a firmware and a cloud service.

I’ve had my hands in a number of companies at various depths of “hands in.” Those who lay out a source tree that fully acknowledges the complexity of the product as early as possible tend to be the ones who win. Often, a company is started to build a core product–such as an ability to move data–and the user interface, the start-up scripts, the “stuff” that makes the algorithm no longer a student project but a product worth many millions of dollars–is an afterthought. That’s fine until someone creates a source tree that looks like this:

trunk/
	cool_product/
		main.c
		error.c
		util.c
		network.c
	stuff/
		boot.c
		cli.c
		gui.c

What’s wrong here? Presumably, some of the code in util.c could be used in other places. Maybe some of the functions in error.c would be handy to abstract out as well. An arrangement like this in which the cool_product is a large monolithic app likely means it’s going to be difficult to test any of the parts inside of it; likely modules and layering are not respected in a large monolithic app. (Note that I am not saying it’s impossible to get this right, but I am saying it’s unlikely that tired programmers will keep the philosophy in mind, day in and day out.)

A slightly different organization that introduces a library might look as follows:

trunk/
	lib/
		util.c
		error.c
		network.c
		tests/
			Unit tests for the lib/ stuff
	prod/
		cool_product/
			main.c
		gui/
		cli/
	tools/
		Build scripts or related stuff required,
		code generation, etc.
	platform/
		boot.c

As a side effect, we can also improve testing of the core code, thus improving reliability and regression detection. Ideally, the cool_product is a small amount of code outside of libraries that can be unit tested independently.

More than once I’ve heard the excuse, “We don’t have time to do this right with the current schedule.”

“I don’t have time for this” means “This isn’t important to me.” When you say, “I don’t have time to clean up the garage,” we all know what you really mean.

I was incredibly frustrated working with a group who “didn’t have time” to do anything right. Years later, that company continues to ship buggy products that could have been significantly less buggy. A few weeks of investment at the beginning could have avoided millions of dollars of expense and numerous poor reviews from customers due to the shoddy product quality. And it all comes back to how hard (or easy) it is to use the existing code, i.e., the communication structure of the code.

If you don’t have time to get it right now, when will you have time to go back and do it right later?

Getting it right takes some time. But getting it wrong always takes longer.

Teams with poor source tree layout often end up copying and pasting code. Sometimes a LOT of code. Whole files. Dozens of files. And as soon as you do that, you’re done. Someone fixes a bug in one place and forgets to fix it in another–over time, the files diverge.

If you’re taking money from investors and have a screwed up the source tree layout, there are two ethical options:

1. Fix it. A week or two now will be significantly cheaper than months of pain and glaring customer issues when you ship.
2. Give the money back to the investors.

If you’re reading this and shaking your head because you can’t believe people paint themselves into a corner with their source tree layouts, I envy you! But if you’re reading this and trying to pretend you don’t face a similar position with your product, it might be time to stop hacking and start engineering by opening up the communication paths where they should be open and locking down the isolation and encapsulation where they should not. This holds true for any language and for any type of product.

In 1968, Dr. Melvin E. Conway published an article called, “How Do Committees Invent?”

In this paper, buried towards the end, is the following insight:

organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations.

Thinking back on my product experiences, this has been the case every time. The cracks in products show up where teams didn’t talk to each other, where two people didn’t get along, or where someone wasn’t willing to pick up the phone and call someone else. Features or modules that integrated well and worked smoothly reflect where two or more people worked well together. In cases where one person went off by himself and re-invented the wheel, sometimes even large core parts of a product, led to internal difficulties and those internal difficulties turned into product difficulties when the product shipped.

As an engineer, every time you don’t pick up the phone to call a colleague about an integration point, you’re making life harder on your customer. As a manager, every time you don’t deal with someone not communicating, you’re making life harder on your customer. Meanwhile your competition who play well together are building beautiful products that flow.

The communication successes and failures of an organization are independent of the organization size. It’s fashionable to say that small teams work better than large organizations (37signals vs Microsoft), but in fact, a small team can be incredibly dysfunctional, just as a large organization can work well (many start-ups vs Apple).

Of course, the scope of “systems” goes beyond products. IT deployments–if your VPN guy and your Exchange guy don’t like each other, how many times do you have to login to different computers? Marketing strategies–700 folks clicked on an emailed link, but did those people have a good experience on the landing page? Sales operations–much time was invested in segmenting and building custom collateral but were those materials used or ad hoc assembled in the field? Manufacturing–sure, everyone signed off on the Micron chips, but “someone” decided to build half the boards with Hynix and didn’t tell anyone? Support–Is your support experience congruent with the product, or is it outsourced with its own login, and the support folks have their own culture?

A team that doesn’t communicate openly, frequently, and freely is expensive to operate and builds lower quality products, end-to-end.

Colleagues who have worked with me in the last ten years know I have some religion about appliances. To a large degree, I believe that most enterprise software that is installed in a customer’s data center should be packaged as an appliance. The embodiment of this could be as a physical appliance or as a virtual software appliance, but either way, it’s an appliance.

My sense is that appliances, especially virtual appliances, are picking up momentum. We’re seeing more and more software companies offering virtual appliances on the market today and more products that could be packaged as software instead packaged as a hardware offering.

There’s a problem though.

Most of these appliances are crap.

What is an Appliance?
It would be cliché to turn to Merriam Webster for a definition, so let’s consider what the Oxford American Dictionary says an appliance is:

ap-pli-ance n. 1 a device or piece of equipment designed to perform a specific task

So if we’re building an enterprise IT appliance, it’s going to be a virtual machine or a physical machine to perform a specific task.

In my mind I’ve always given Dave Hitz, co-founder of NetApp, credit for defining an appliance, but in this interview [PDF], he gives credit to the model Cisco established:

Hitz: […] We were modeling ourselves after Cisco. If you look before Cisco, people used general purpose computers, UNIX workstations, Sun workstations for their routers. Cisco came along and built a box that was just for routing. It did less than a general purpose computer and so the conversation we kept having with the VCs, they would say, “So your box does more than a general purpose computer so it’s better?” And we’d say, “No, no, it does less.” And they’d go, “Well, why would I pay you money to start a company that does less?” And we said, “No, no, it’s focused. It’ll do it better.”

He continues:

The other analogy that we used was a toaster. If you want to make toast, you could go put in on your burner, you can put it in the oven and it’s certainly the case an oven’s a lot more of a general purpose tool than a toaster. A toaster? What good’s a toaster? All it does is make toast. But, man, if you want toast, a toaster really does a good job, you know? Try making toast in your oven and you’ll discover how awesome a toaster is. And that really, for storage over the network, it was that idea of making a device to do just that one thing, focused. Not run programs, not do spreadsheets, not be a tool for programmers, just store the data.

I’ve always stuck by the toaster model when thinking about appliances for customers.

So what does this mean for the user?
Let’s talk about product requirements.

When a vendor ships an appliance to the customer, everything is included: The OS, the management tools, and the application. The beauty of this approach is that the customer no longer has to install software. The customer is no longer responsible for dealing with OS patching and ensuring the right OS patches are installed that match with the requirements of the application.

The user interface should be stripped down and minimal. The user should never see a general purpose bash shell prompt. The appliance should be as self-contained as reasonable. Configuration should be as minimal as possible. IP address and related information, LDAP or user set-up, NTP and timezone, email and SNMP, application-specific coordinates, and that’s it! This can be implemented in a small X11 set-up GUI (Nasuni does an excellent job here) or a simple ncurses-based text wizard (similar to what ESXi has).

The user should never be told to edit files off in /etc or manually tweak key-value pairs to get the basic application up and running. Some application programmers get off on connecting to a customer-provided database. In a particularly large installation, this might make sense. To store 500 rows of data, it does not.

An appliance is likely Linux-based these days, as it should be. Software updates should come from an update server provided by the vendor. From the user’s perspective, there is no apt-get or yum details to worry about; it’s a simple matter of clicking a ‘Upgrade’ button. Anyone shipping a Windows-based appliance should only do so for very specific reasons (e.g., need access to Windows APIs).

The GUI for the appliance should be web-based. These days, it needs to be accessible from a variety of browsers. Product managers need to understand the average IT guy has more things to worry about than a single product, and hints need to be provided on how to find the management UI for the appliance. I love how HP handles this on their switches and I adopted their model at my last company: Every login console tty included the IP address and how to find the GUI; both shell and GUI login screens included the default account name and password, right there. The last time you installed a home router, did you have to fire up Google and search for the default user name and password? Or did you read the documentation? Did you read the documentation for Tetris? Why should the default user name and password be harder than playing Tetris?

DHCP should be the default on a first boot, and active on all interfaces with a link. Anyone shipping an appliance without DHCP in this day and age needs to get with the times.

Any CLI should be succinct and hierarchical. Enterprise IT gets tricky, no matter how we simplify it, and users need a way to drill into the available commands. The CLI needs to balance “learning” with speed of use. The reality is, most IT folks are using a lot of different products throughout the day. Make yours the one that is easiest for the admin to jog his mind on how to see replication status (or whatever).

When it comes to lights-out management, product managers and engineers must understand that the product is not the center of the end-user’s universe. Problems and reports should be available via email. With a generated email, the subject line is critical and should be succinct but descriptive. “2 alerts (product name)” with a From: field of the hostname is a good start at a subject line. Failures and recovery from failures should be as automatic as possible. Customers should never have to login to click something in a GUI that the product could have run itself to recover from an error.

Although not strictly appliance-related, let’s also talk about sockets and ports of the TCP/IP variety. When it comes to networking, products should be designed with WANs, VLANs, routers, closed ports, and different views of networks when communicating with other instances of the same product. Don’t rely on ICMP pings being available. Don’t rely on your connection target matching the listening target coordinates. Have flexibility. Design your network protocols for the simplest firewall configurations. In a spoke/hub topology, all the spokes should reach into the hub–that is, all spokes open sockets on a port the hub is listening on, even if the logical client/server is the other direction.

Shipping a Virtual Appliance
This is where I see the most sin in the market today. So many folks install Ubuntu, install their software, shutdown the VM, zip it up, ftp it over to their web site and call it a “virtual appliance”. This is a VMDK full of crap. /root/.bash_history is still populated, as are DHCP cache files, log files, and other things that should be scrubbed. There is no OVF installer file. Old snapshots of the VM are included. What kind of embarrassing stuff is in there?

Let me state up front that OVF is one of the dumbest things in the VMware world. There is no need for it to exist; everything that it does could have been accomplished much more simply. But VMware doesn’t like to make things easy, so we’re stuck with it. On the bright side, the user experience of installing an OVF appliance into a VMware environment is awesome. Thus, vendors need to use it.

From a mindset perspective, the vendor needs to get into a mental mode of a manufacturing process to scrub the VM prior to OVF packaging. Things need to be set to “factory defaults”.

With both physical and virtual appliances, the size of Linux distribution should be minimal. This is more important with the virtual appliance distribution, since customers will be downloading it. As a vendor, you can pick a few approaches on how you do this, but in general, reduce the number of packages you’re including to the bare minimum. You need to tune your CPUs and RAM to the minimum required for your application and let customers tune up from there. If you’re shipping both virtual and physical appliances, you need to understand these are different beasts. How will you sell them? My bet is that you’re pushing virtual appliances into remote-office/branch-office use cases, so size the requirements for the VM appropriately. Size your VMDKs down to something reasonable or use thin-provisioning. Note that OVFs with thin-provisioning will not work on ESX versions less than 4.1 or 4.5 (I can’t remember exactly which versions are broken for thin-provisioning with OVF).

Shipping a Physical Appliance
Ten years ago, I worked on my first appliance product. One of the problems we had with this physical appliance when we went out in the field was that we needed a screwdriver to get it in the rack. Customers sometimes had to spend 10 minutes searching for tools if they were installing it themselves. We started to include a multi-tip screwdriver with the product in the shipping carton. In 2010, my new company had the opportunity to buy a few systems from the previous company. The screwdriver was still included. If I ever do a physical appliance again, I will include the screwdriver. What’s $10 on a $50,000 system anyway? Don’t forget this is a branding opportunity–put your logo or URL on the handle.

If your system is heavy, ship it on a pallet. If you’re on the border (70-80 lbs), pay attention to the condition that the system packaging is in when it arrives at customer sites and adjust accordingly.

Pick rails that don’t suck and work in 2-post and 4-post racks.

Use fans that are quiet. Spend a bit of time to add software support to control the fans appropriately. If you own a previous product I worked on with loud fans, I apologize. I really do.

There are lots of little details to take care of to finish a solid appliance offering. The application should be monitored by a watchdog (inittab, eventd) so when it goes down, it comes back up. Log file size and rotation should be set up. Cores should be constrained if runaway coring occurs. The application should start when the appliance is booted by default. It sounds obvious, but I’ve seen a few appliances that didn’t do this. Vendors should change /etc/rc.sysinit so that when fsck fails during boot, customers are instructed to call the vendor, or dropped into an emergency shell, rather than prompted for a root password. Vendors need to abstract configuration variables set by the users from their destination /etc files. It needs to be easy for the user to back-up, reset, and re-apply configuration settings.

Where Do We Go From Here?
Lots of applications are going to the cloud, which is fantastic. We’ll continue to see more of that. But we’ll still need to have software in our data centers. And it would be nice to see much of that software shipping as an appliance. I hate installing software applications on an OS for IT workloads. It’s stupid, it’s complex, and it’s largely unnecessary. Anyone who has installed a complex application knows how stupid it can be. Installing a TSM server package on a CentOS base is a quick 200 step process and the fact that even I could make it work means it was probably pretty easy.

Over time, we might find that some of the Cloud APIs are enabling compute resources with a different layer underneath than the traditional OS. A proper API underneath that extends onto physical and service offerings that can help standardize some of these appliance computing characteristics would be wonderful, but I don’t see how anyone can make money developing such a thing, so I am not optimistic that it will come to be.

So, remember, an appliance:

• Does one thing. It is focused and minimal.
• Is stripped down. There’s no bash shell for the user. The UI is focused, concise, web-based.
• Is lights-out. It recovers from failure. It is well behaved. It handles updates easily.
• Reduces burden on the user.

Vendors need to quit shipping software junk that they call an appliance. Customers need to demand more; it’s amazing how much bad software IT has to deal with. Stop adding to the pile.

Ship something good.

Many thanks to my colleagues and friends who reviewed this post before publishing.

I get this question a lot. I wrote this down as my standard answer.

First, you need to be sure you are building a product people want. This means doing market research and talking to potential customers before you do anything else. It sounds obvious, but building something people want is the hardest part. When you’re evaluating whether or not people want what you’re building, you need to hear people say, “OK, when can I buy this? I need this right now.” Potential customers who say, “Yeah, that sounds like a great idea” are misleading–that kind of response means you haven’t gotten them to the mental finish line such that they want to buy. It is crucial to understand the difference between these two reactions.

If you are serious about starting a company to the point that you or your partners are quitting your jobs, you need to go ahead and legally form the company. You can “do it yourself” but I recommend finding a respected attorney familiar with the law of where you are starting your company. In particular, you should have employment contracts between all partners and the company, and you should have intellectual property assignment agreements between all contributors and the company.

Without these basic agreements in place, your company can suddenly be in a position of being (1) un-fundable by VCs or other capital sources, (2) un-acquirable, (3) sued by a partner who has dropped out or feels he has been wronged. And of course, without some non-compete agreements, a partner can leave and potentially take know-how of your business to start a competing company.

Do not depend on a ‘gentleman’s agreement’ of what will happen when someone leaves–even if everyone trusts everyone and everyone has known everyone else for many years. In every company I have ever started, someone has left before the first product was ready to go to market. In one case, that tanked the company and the remaining partners, including yours truly, lost many thousands of dollars.

Setting up a company and getting these agreements in place is relatively cheap, even with a high-end law firm in an expensive city. The cost is essentially zero compared to other start-up costs and it will save you serious stress and money down the road.

Hire an attorney who specifically deals with business. I would not hire a “front door attorney” who “practices whatever comes in the front door.” Ask for references from businesses that were once starting out like yours. Also, you will need a CPA to keep you in-line with the IRS and other government agencies. The CPA will cost you much less and, in some ways, be a far greater value. I wouldn’t start a business without either of these people on my “team”. I’ve done it the wrong way and the right way–and the right way is MUCH better and less stressful.

When you’re just starting out, it can be very easy to fall into the trap of thinking that what you’re doing isn’t going to be “big” or “maybe it’s not important enough” to be worth protecting. But if you have quit your job to do something, it must be big and important enough to justify taking some basic precautions such as these. Chances are living expenses while you build your product will dwarf the costs of protecting yourself–there’s just no reason not to.

Of course, if you’re building a company alone, you might not need much of the above–I am really referring to scenarios involving 2 or more people. If you’re going at it alone, you might still need protection from contractors you hire, though.

See also: Top Ten Legal Mistakes Made by Entrepreneurs. I also recommend this book which covers the above scenarios and many other issues.

Finally, this is not legal advice.

(This is from my office FAQ; it felt more appropriate here.)