Mitch's Blog

Apple hasn’t updated the Mac Pro for almost 600 days. My 2008 Mac Pro is 1500 days old. I’ve been holding out for a new model; the wait has gotten so long I’ve even pondered whether just buying a second 15″ MBP would be an OK way to go.

Here’s what I’d like in a new Mac Pro offering:

• New form factor; no taller (wider) than 3U. The case should fit into 3U of rack space, even if it doesn’t come with rack ears.
• Eight 2.5″ HD bays for internal storage.
• Nuke the optical drives.
• Redundant power supplies (two would suffice, as long as the computer can run on one).
• RAM ceiling increase; at least 128 GB; 256 GB would be better.
• Four PCIe slots with two double-wide slots.
• Better video cards; support two 4 GB cards; four displays per card.
• Two copper 10gigE ports.
• 16 cores at the high end.

The physical changes are easy: By nuking the optical drives, there is room for 6-8 2.5″ bays (on their side) in front of the power supply. By nuking the 3.5″ carriers, the case can lose about 1.25″, which is enough to drop the height under 19″, which gets rack-friendly. That still leaves the thickness problem–the Mac Pro is currently 8″ wide, but 3U is 5.25″ and 4U is 7″. Perhaps getting to 3U is too aggressive without adding depth to the case–30″ depth would be fine for racks but not as friendly for desktop use.

I really want to see Apple start shipping 10gigE. The price is getting cheaper; switches from NetGear with four 10gigE ports and 20 gigabit ports were around $1,500 last I checked. Apple can help drive the price down rapidly.

Somewhere around $12k nicely equipped (e.g., 8×512 GB SSDs, 96 gb of RAM, 8 cores, one 4 GB video card) would be great. You’ll note I didn’t mention anything about Thunderbolt; I just don’t care about Thunderbolt in a desktop system.

No, I don’t expect Apple to ship this any time soon, but man I’d really like to have this system.

Running low on my 32 GB Mac Pro.

The first laser printer I bought was a Lexmark Optra E312. This was in 2001 or so; the Lexmark was $400, had USB, and worked with Macs, though I mostly used it with a parallel port connection to a Linux machine with lpd. It had real PostScript. The print quality was amazing, but the paper handling was terrible. It held paper “upright” in the rear of the printer, which I suppose was a decision of economics, but it also meant that it printed crooked most of the time. It would also jam when making the turn into the “upright” output tray, so the solution was to open the front and let papers fly around on the desk. Despite these issues, I was content with it (for the price) and used it very heavily. I gave it to a friend in 2005 and I believe he still uses it.

In 2005, I bought a new Lexmark network laser printer. It was very cheap, around $250, and it was amazing. I used it very heavily. When I switched coasts for my permanent residence in 2007, I gave it to another friend who is also still using it.

My Canon MF4690 in its natural habitat.

So after this, I needed a new printer. I ended up buying a Canon multi-function laser for around $400. Copying and faxing were important to me and I didn’t have room for 2 or 3 devices (I previously had a separate fax machine). By now, LCD monitors had gotten very cheap and I had a lot of them. I was actually printing far less than I ever had before because I could see so much more on the screen at the same time. I often printed to see more content concurrently, whether the content was code, documentation, or presentations. In fact, I’ve only printed about 4800 pages on this printer in the last 4.5 years.

The multi-function has been amazing. I can scan documents to PDF that the printer emails to me, with no computer involvement. With the auto-document feeder, I can scan in a stack of documents. When doing complex financial transactions, this has been a huge timesaver. No one should be running any kind of operation without one of these devices if you have to sign a lot of documents. Previously I was running to Kinko’s for big “signage” or messing around with a flatbed scanner or dealing with a crappy fax machine. All of those were a huge waste of time.

The downside of the Canon multi-function has been the print quality. It’s somewhere between “mediocre” and “fair.” Gray halftones are a disaster and the definition around the edges is sloppy. While it’s fine for proofreading or marking up a document, it’s embarrassing to hand someone a document from that printer in a professional setting.

Canon MF4690 text output. Note the jaggy edges. Palatino 10 pt font.

In 2009, I needed to print some confidential color documents, so printing them at Kinko’s was out of the question. I bought a color laser printer to augment the Canon, a Lexmark C543DN. As my review on Amazon says, “I liked it. I printed 60 pages. Then it died.” Cause of death was a “service printhead” error. Unfortunately by the time I realized it was dead, it was out of warranty (I did my initial printing, then did 12 months of traveling, then tried to print again). I ended up having 1-800-Got-Junk throw it away with some other trash.

I was wary to buy another color laser printer, but once again I need to create camera-ready documents without hauling myself over to Kinko’s. So I just picked up a Brother color laser printer for cheap on Amazon (and I later added the 2nd paper tray option). Common complaints about the Brother are paper curling and expensive toner–nothing unusual here. The print quality for “business graphics” and text is amazing. I should have bought a new color laser printer when the Lexmark died.

Brother HL4570CDW text output. Much smoother. Taken with the same lighting, microscope, and focus as the previous picture.

The new color printer is quite a bit slower than the Canon when waking from sleep. I printed the above text pages for comparison at the same time from the same computer. The Canon woke up, printed, and was done before the Brother had finished waking up. The Canon has 32 MB of RAM vs the Brother’s 128 MB (upgradable to 384 MB for $10; DIMM ordered). The Brother is also quite a bit louder than the Canon, but quieter than some other printers. To me, these negatives are worth the improved text, grayscale, and addition of color output. Even printing photos on plain 22 lb paper is shockingly good with the Brother; if memory serves, much better than the old Lexmark. Of course, photos will have the halftone pattern and lose a lot of crispness; if you are shopping for a photo printer, laser is probably not the technology you seek. I would love to have a 8-cartridge wide-format printer for such projects.

Toner is pretty well priced for both of these printers; the 1,500-2,500 page toner cartridges for the Brother are $200 for a set of four and the higher yield 3,500-6,000 are $340. The Canon black cartridge has usually run me about $75. These prices are similar to what I’ve paid for other printers. There are a lot of complaints on Amazon about these prices, but I think they are pretty standard and I really appreciate Brother offering the lower yield cartridges as an option. Plenty of other vendors have cartridges running at $120/each. The reality of a color printer is that the printer needs 4 cartridges.

It is amazing how cheap laser printers have gotten. According to this inflation calculator, the $7,000 LaserWriter IINTX printer my dad bought in 1988 would be over $12,500 today. My dad ran 80,000 pages through that printer over the course of 15 or 16 years. I seem to recall reading an article in Macworld about personal laser printers sometime around 1992; the costs were dropping down to $1,500 or so. Today, a pretty good black-and-white laser is $200 and color isn’t much more than that.

In addition to (or because of) high prices, in the “good ole days” of laser printers, Macworld and other magazines would publish enlarged views of actual print quality. I miss that dearly when researching printer purchases.

I am not sure I will get 15 years of service out of my Canon, but I have gotten 5 years so far and can imagine 8 or 10 are quite probable. Even though prices have gone through the floor, there are still quality products out there to be found. I am hopeful the new Brother printer is one of them. The amount of printing I am doing these days is significantly less than what I used to do, but printing is a key part of certain workflows. I suspect that will continue for some number of years to come.

I keep getting asked the question, “Does the Nest thermostat save you money?”

I can only blush and say, “I don’t know.”

Isn’t that a bit silly? I have a “Thermostat 2.0” and I don’t know the answer to this question.

Sure, it’s been a warm winter. But that’s not really the point.

My guess is that the Nest replaces two very simple types of thermostats: Programmable thermostats that had very simple schedules and thermostats with little to no schedule. What I’d really like is a way to put in my old thermostat program into the Nest web interface and have it show me the delta energy-hours saved (or not) for auto-away, the learned schedule, and remote adjustments. I almost never touched my old thermostat, but the schedule the Nest has learned is extremely non-trivial. Being able to understand the difference of how much the furnace would have run with the old schedule vs the new schedule would be very useful. Dollars saved (or burned) could be modeled by asking for the estimated energy price for the furnace per month (last month my furnace operation cost me about $170).

ROI modeling is complex and it can be hard to work up a model that feels legitimate. But I think for Nest, it could be done easily. If Nest had an API exposed, it would be easy for a third party to build some ROI tools as well.

When ROI can be easily expressed, the vendor can benefit from excited users talking about their ROI. But for now, I just have to say, “it’s a cool thermostat” when people ask me if it’s saving me money.

If you’re providing a tool or service, work with your customers to see if you can improve your ROI models. It’s an iterative process, but if you can nail it, you’re that much closer on your next deal. Would-be customers of Nest who ask me about the ROI story aren’t getting a good answer.

(Yes, the Nest includes some data on saving energy, but I really want to know how it compares to what my old thermostat would have done. And I want to see that energy saving data shown on the device itself on the web site and iPad apps. Those are my primary interfaces to the Nest, not the thing on the wall.)

Colleagues who have worked with me in the last ten years know I have some religion about appliances. To a large degree, I believe that most enterprise software that is installed in a customer’s data center should be packaged as an appliance. The embodiment of this could be as a physical appliance or as a virtual software appliance, but either way, it’s an appliance.

My sense is that appliances, especially virtual appliances, are picking up momentum. We’re seeing more and more software companies offering virtual appliances on the market today and more products that could be packaged as software instead packaged as a hardware offering.

There’s a problem though.

Most of these appliances are crap.

What is an Appliance?
It would be cliché to turn to Merriam Webster for a definition, so let’s consider what the Oxford American Dictionary says an appliance is:

ap-pli-ance n. 1 a device or piece of equipment designed to perform a specific task

So if we’re building an enterprise IT appliance, it’s going to be a virtual machine or a physical machine to perform a specific task.

In my mind I’ve always given Dave Hitz, co-founder of NetApp, credit for defining an appliance, but in this interview [PDF], he gives credit to the model Cisco established:

Hitz: […] We were modeling ourselves after Cisco. If you look before Cisco, people used general purpose computers, UNIX workstations, Sun workstations for their routers. Cisco came along and built a box that was just for routing. It did less than a general purpose computer and so the conversation we kept having with the VCs, they would say, “So your box does more than a general purpose computer so it’s better?” And we’d say, “No, no, it does less.” And they’d go, “Well, why would I pay you money to start a company that does less?” And we said, “No, no, it’s focused. It’ll do it better.”

He continues:

The other analogy that we used was a toaster. If you want to make toast, you could go put in on your burner, you can put it in the oven and it’s certainly the case an oven’s a lot more of a general purpose tool than a toaster. A toaster? What good’s a toaster? All it does is make toast. But, man, if you want toast, a toaster really does a good job, you know? Try making toast in your oven and you’ll discover how awesome a toaster is. And that really, for storage over the network, it was that idea of making a device to do just that one thing, focused. Not run programs, not do spreadsheets, not be a tool for programmers, just store the data.

I’ve always stuck by the toaster model when thinking about appliances for customers.

So what does this mean for the user?
Let’s talk about product requirements.

When a vendor ships an appliance to the customer, everything is included: The OS, the management tools, and the application. The beauty of this approach is that the customer no longer has to install software. The customer is no longer responsible for dealing with OS patching and ensuring the right OS patches are installed that match with the requirements of the application.

The user interface should be stripped down and minimal. The user should never see a general purpose bash shell prompt. The appliance should be as self-contained as reasonable. Configuration should be as minimal as possible. IP address and related information, LDAP or user set-up, NTP and timezone, email and SNMP, application-specific coordinates, and that’s it! This can be implemented in a small X11 set-up GUI (Nasuni does an excellent job here) or a simple ncurses-based text wizard (similar to what ESXi has).

The user should never be told to edit files off in /etc or manually tweak key-value pairs to get the basic application up and running. Some application programmers get off on connecting to a customer-provided database. In a particularly large installation, this might make sense. To store 500 rows of data, it does not.

An appliance is likely Linux-based these days, as it should be. Software updates should come from an update server provided by the vendor. From the user’s perspective, there is no apt-get or yum details to worry about; it’s a simple matter of clicking a ‘Upgrade’ button. Anyone shipping a Windows-based appliance should only do so for very specific reasons (e.g., need access to Windows APIs).

The GUI for the appliance should be web-based. These days, it needs to be accessible from a variety of browsers. Product managers need to understand the average IT guy has more things to worry about than a single product, and hints need to be provided on how to find the management UI for the appliance. I love how HP handles this on their switches and I adopted their model at my last company: Every login console tty included the IP address and how to find the GUI; both shell and GUI login screens included the default account name and password, right there. The last time you installed a home router, did you have to fire up Google and search for the default user name and password? Or did you read the documentation? Did you read the documentation for Tetris? Why should the default user name and password be harder than playing Tetris?

DHCP should be the default on a first boot, and active on all interfaces with a link. Anyone shipping an appliance without DHCP in this day and age needs to get with the times.

Any CLI should be succinct and hierarchical. Enterprise IT gets tricky, no matter how we simplify it, and users need a way to drill into the available commands. The CLI needs to balance “learning” with speed of use. The reality is, most IT folks are using a lot of different products throughout the day. Make yours the one that is easiest for the admin to jog his mind on how to see replication status (or whatever).

When it comes to lights-out management, product managers and engineers must understand that the product is not the center of the end-user’s universe. Problems and reports should be available via email. With a generated email, the subject line is critical and should be succinct but descriptive. “2 alerts (product name)” with a From: field of the hostname is a good start at a subject line. Failures and recovery from failures should be as automatic as possible. Customers should never have to login to click something in a GUI that the product could have run itself to recover from an error.

Although not strictly appliance-related, let’s also talk about sockets and ports of the TCP/IP variety. When it comes to networking, products should be designed with WANs, VLANs, routers, closed ports, and different views of networks when communicating with other instances of the same product. Don’t rely on ICMP pings being available. Don’t rely on your connection target matching the listening target coordinates. Have flexibility. Design your network protocols for the simplest firewall configurations. In a spoke/hub topology, all the spokes should reach into the hub–that is, all spokes open sockets on a port the hub is listening on, even if the logical client/server is the other direction.

Shipping a Virtual Appliance
This is where I see the most sin in the market today. So many folks install Ubuntu, install their software, shutdown the VM, zip it up, ftp it over to their web site and call it a “virtual appliance”. This is a VMDK full of crap. /root/.bash_history is still populated, as are DHCP cache files, log files, and other things that should be scrubbed. There is no OVF installer file. Old snapshots of the VM are included. What kind of embarrassing stuff is in there?

Let me state up front that OVF is one of the dumbest things in the VMware world. There is no need for it to exist; everything that it does could have been accomplished much more simply. But VMware doesn’t like to make things easy, so we’re stuck with it. On the bright side, the user experience of installing an OVF appliance into a VMware environment is awesome. Thus, vendors need to use it.

From a mindset perspective, the vendor needs to get into a mental mode of a manufacturing process to scrub the VM prior to OVF packaging. Things need to be set to “factory defaults”.

With both physical and virtual appliances, the size of Linux distribution should be minimal. This is more important with the virtual appliance distribution, since customers will be downloading it. As a vendor, you can pick a few approaches on how you do this, but in general, reduce the number of packages you’re including to the bare minimum. You need to tune your CPUs and RAM to the minimum required for your application and let customers tune up from there. If you’re shipping both virtual and physical appliances, you need to understand these are different beasts. How will you sell them? My bet is that you’re pushing virtual appliances into remote-office/branch-office use cases, so size the requirements for the VM appropriately. Size your VMDKs down to something reasonable or use thin-provisioning. Note that OVFs with thin-provisioning will not work on ESX versions less than 4.1 or 4.5 (I can’t remember exactly which versions are broken for thin-provisioning with OVF).

Shipping a Physical Appliance
Ten years ago, I worked on my first appliance product. One of the problems we had with this physical appliance when we went out in the field was that we needed a screwdriver to get it in the rack. Customers sometimes had to spend 10 minutes searching for tools if they were installing it themselves. We started to include a multi-tip screwdriver with the product in the shipping carton. In 2010, my new company had the opportunity to buy a few systems from the previous company. The screwdriver was still included. If I ever do a physical appliance again, I will include the screwdriver. What’s $10 on a $50,000 system anyway? Don’t forget this is a branding opportunity–put your logo or URL on the handle.

If your system is heavy, ship it on a pallet. If you’re on the border (70-80 lbs), pay attention to the condition that the system packaging is in when it arrives at customer sites and adjust accordingly.

Pick rails that don’t suck and work in 2-post and 4-post racks.

Use fans that are quiet. Spend a bit of time to add software support to control the fans appropriately. If you own a previous product I worked on with loud fans, I apologize. I really do.

There are lots of little details to take care of to finish a solid appliance offering. The application should be monitored by a watchdog (inittab, eventd) so when it goes down, it comes back up. Log file size and rotation should be set up. Cores should be constrained if runaway coring occurs. The application should start when the appliance is booted by default. It sounds obvious, but I’ve seen a few appliances that didn’t do this. Vendors should change /etc/rc.sysinit so that when fsck fails during boot, customers are instructed to call the vendor, or dropped into an emergency shell, rather than prompted for a root password. Vendors need to abstract configuration variables set by the users from their destination /etc files. It needs to be easy for the user to back-up, reset, and re-apply configuration settings.

Where Do We Go From Here?
Lots of applications are going to the cloud, which is fantastic. We’ll continue to see more of that. But we’ll still need to have software in our data centers. And it would be nice to see much of that software shipping as an appliance. I hate installing software applications on an OS for IT workloads. It’s stupid, it’s complex, and it’s largely unnecessary. Anyone who has installed a complex application knows how stupid it can be. Installing a TSM server package on a CentOS base is a quick 200 step process and the fact that even I could make it work means it was probably pretty easy.

Over time, we might find that some of the Cloud APIs are enabling compute resources with a different layer underneath than the traditional OS. A proper API underneath that extends onto physical and service offerings that can help standardize some of these appliance computing characteristics would be wonderful, but I don’t see how anyone can make money developing such a thing, so I am not optimistic that it will come to be.

So, remember, an appliance:

• Does one thing. It is focused and minimal.
• Is stripped down. There’s no bash shell for the user. The UI is focused, concise, web-based.
• Is lights-out. It recovers from failure. It is well behaved. It handles updates easily.
• Reduces burden on the user.

Vendors need to quit shipping software junk that they call an appliance. Customers need to demand more; it’s amazing how much bad software IT has to deal with. Stop adding to the pile.

Ship something good.

Many thanks to my colleagues and friends who reviewed this post before publishing.

A few weeks ago I wrote a post about USB wall outlets. On January 25th, I stumbled over the Cooper USB Charger that is a Decora-shaped receptacle. This is exactly what I wanted. Some of the other vendors poo-poo the Cooper for only being a 700mA charger vs 1 or 2 amps, but frankly, the smaller mA rating means the Cooper footprint is smaller. I am also not sure I want a 2 amp charger in a wall box.

Since my use case is to plug things in overnight or for long periods of time, I don’t care how long it takes an iPad or iPhone to charge with this (within a reasonable amount of time). I do, however, care a great deal about the cosmetic looks–which is the whole point of a USB wall outlet, right? If you didn’t care about looks and only cared about charging speed, a big mess would suit you fine. The Cooper has the looks right and the standard Decora form factor means that the Cooper can be installed into multi-gang boxes with other outlets, other Coopers, etc. and it will all fit OK.

Today I installed the Cooper next to the side table on my side of the couch. What a difference it makes in the wire clutter. Before I had a “right angle” adapter to fit in the iPad charger, iPhone charger, and a Mac laptop charger. The clutter is vastly reduced, as you can see from the pictures. I also installed another one of these outlets next to my nightstand for iPad & Kindle charging.

Installation is fairly easy and no different than any other device you might install. One thing to note that is there’s one only set of screw terminals, so if you’re replacing an outlet in the middle of a run, you might find yourself adding pigtails–which is another plus for this device having a smaller physical footprint than the competition. If the previous sentence didn’t make sense, you should call your electrician to install this.

I’m very happy with this solution. You “lose” an AC outlet, but if at least one of those outlets was for a USB charger, you are actually losing nothing, and if both outlets had a USB charger, you gain an outlet. The Cooper also looks better than the other options on the market I’ve seen (U-socket, NewerTech, and various wall-plate chargers). Also, despite what some guy on Amazon says, these outlets are UL-listed.

I’ve posted more pictures in this flickr set.

You also might enjoy my post on built-in night lights.

P.S. I had noted in the prior blog post that I had previously bought the U-socket from FastMac–In fact, I spent over $90 on buying four of them, but I never installed them because they are ugly. I think I was so disappointed that I threw them away when I got them. Another big downside was that I ordered them on 6/2/2010 and they were shipped to me on 2/18/2011–that’s 8 months later, with no warning, which if not a violation of credit card rules (30 day pre-order is the limit, right?), certainly was disappointing from a customer service perspective.

For the last four years, I’ve been a happy business Comcast Internet customer. I just have Internet and voice with Comcast; no TV. I have 5 static IP addresses, and until earlier this month, Comcast has been rock solid with only a few minutes of downtime here and there. I am an outlier when I say that I love US Airways and I love Comcast. I don’t know anyone else who goes around saying stuff like that. In some areas, you might get hanged.

In January, I decided I wanted more speed, but I debated–should I upgrade to 105 Mbit down/20 Mbit up (about $380/mo) or 50/10 (about $190/mo)? I talked to the Comcast sales rep on Jan 4th and she suggested trying the 50/10 and see how it goes. So I said let’s do it. My install window was set for 3p-5p on Friday, Jan 6th, since they needed to upgrade my modem.

On Jan 6th, at 4:50pm, a Comcast tech knocked on the door. “If this goes well, I’ll be out of here in 20 minutes.”

The Comcast tech was a great guy, I really liked him. However, he was hanging out in my network closet until 9:30pm that night. My girlfriend asked if she should clean up the guest room so he could spend the night.

Why was he at my house so long? The initial modem configuration was a bit tricky to make sure my static IP addresses got moved to the new modem. This took a few minutes. After that, he wasn’t getting the speed that I was supposed to get–about 1 Mbit/s up. He replaced the modem. He replaced the line to the street. He called tech support several times. He eventually gave up and went home. The next day he called me and said there was some kind of signal interference issue in my block and that it would take a few days but it would get solved. I really liked this guy and I really liked Comcast, so I was easy going about it and ran SpeedTest.net on a regular basis. A few days later, around Jan 10th, the problem was fixed and everything was good. I was a happy Comcast customer.

I didn’t push the SLA issue with Comcast, but one of the reasons that the business Internet is twice as expensive as residential is that there is a shorter SLA. I work at home when I am not on the road and I require working Internet access. I initially wired my closet for both Verizon and Comcast, but never felt I needed to add Verizon–Comcast has been that solid. I am not sure what constitutes an SLA breach in this case, but let’s be honest, I couldn’t upload 500 MB files during this time–pushing large files to my co-lo in Texas would eventually fail. If I had a major deadline that needed this kind of activity, I would have been toast. But I didn’t and I liked Comcast, so I let it slide.

On Feb 2nd at 10:45 am, my Internet connection died. At 10:50 it was still down. I called the business support line and got a tech window of 1pm to 3pm. I noted to the woman on the phone that this was pushing the boundaries of the 4 hr SLA. She said it was all she could do. She might as well have been a United Airlines employee.

Around 2pm, a tech showed up. “Oh man, static IPs… I am not too good with those!” (paraphrased). He was a really nice guy, but not sufficiently trained for business Comcast deployments with static IPs. However, he knew this and so he called a co-worker who also came over. These two guys and a third guy on the phone spent 90 minutes in my house pondering the problem. The guy on the phone realized that my old modem had been recently deployed across town at a cafe with my configuration data in it. Once they realized that, they were embarrassed, apologetic, and noted that this happens constantly. What kind of business is Comcast running? One of the techs mentioned that they often deploy DVRs that are “erased” “refurbs” with old people’s personal data and recordings on them.

At this point I was losing my patience. My connection had been down for 5.5 hours. I had Comcast guys in my house for a total of 6 hours on multiple days with a total of 11 hours of downtime for what should have been a 10 minute operation. Comcast was sending out guys who were unqualified to work on my account and being very liberal with my time, patience, and SLA.

So after the Comcast techs left, I told @comcastcares on Twitter what I thought. @comcastwill was very active and responsive and indicated a “local leader” would get in touch.

Today, five days later on Feb 7th, I asked @comcastwill if there was any update. He indicated that Comcast had been trying to email me at my @comcast.net address. Why not email the address where my bill goes? Who knows? I have never read my @comcast.net address and have no intention of doing so now. @comcastwill fixed this and shortly thereafter I received an email from “Sharon” asking me to call her.

I had a few minutes this afternoon before a conference call and rang Sharon from “The Executive Office of Comcast” (their words). She said that Comcast was very sorry and would like to offer me a credit on my account for all the aggravation for taking my Internet connection down due to their own negligence.

Comcast reached deep into their pockets, did some soul searching, and concluded that this hassle caused by my desire to give them more money was worth about…. $6.

I literally laughed. “Why wouldn’t I switch to RCN right now?” In this area, I can pick between TWO cable companies as well as Verizon. She said, “I can credit you up to $20 but that’s all I can do.”

Sure I had short patience left for Comcast, but this was absolutely infuriating. I asked @comcastwill on Twitter, “Why did you guys waste my time for $6?”

Remember, the worst part about this is that it was self-inflicted. I decided to give Comcast more money, they botched it, and then offer me a credit of $6. If I keep this level of service with Comcast for four years, as I have for the last four years, that is $10,560 to Comcast. And I don’t even buy TV from them. Who made the assessment that a $6 credit is appropriate? Who thought to themselves, “If I were in this situation and the vendor offered me SIX DOLLARS–less than the price of a QP with Cheese meal at McDonald’s–I would be satisfied?” Isn’t Comcast supposed to be better than this? Aren’t they the model for customer service on Twitter?

I know sometimes things go wrong. I make my living designing and building technology products for medium and large businesses. Some of my customers pay a small amount of money ($20k) and some pay a lot more. I’ve been embarrassed when customers call with problems. That’s why I didn’t give the tech or Comcast any crap with the issues on the upgrade. But there is a point at which mistakes are no longer mistakes and instead are pure incompetence. What exactly is the process for wiping modems at Comcast? Apparently there isn’t any. Business, especially big business, requires process to ensure proper execution. Business also requires handling the exceptional cases when the vendor drops the ball and has to make it right. I cannot imagine taking a $6 discount to any of my customers.

I want a working Internet connection and I prefer it to be with Comcast. But I also want to be treated with respect. My consulting rate is a little higher than $1 per hour. But why turn this into a credit game? Get creative. Send me a Nordstrom, Amazon, or an Apple gift card. Send me a “Get Well Soon” bouquet for my Internet connection. Or call me, admit that you completely screwed up my upgrade, and have an actual conservation with me about it–find out the whole story before deciding “$6 and if he pukes on it, $20”. Don’t try to slice and dice what my time or Internet connection is worth. Because to me, it’s worth a lot more than anything that Comcast could reasonably offer.

Although this post focuses on the $6, the inhuman hand-off to an admin authorized to go to $20 but completely unaware of what went on really irks me too.

Update March 14, 2012: Someone from the “Comcast Executive office” called me yesterday and we chatted a bit. He said he would try to do a little bit more for me, but was very clear my contract doesn’t require Comcast to do anything. In any event, Comcast credited me an additional $103 + the bonus $20 + $7.11 (vs the $6 calculated above). I am glad the fellow called, apologized, and treated me like a valued customer who merits some respect. It took a while but they got it done.