Brain Murmurs

I’m in here again, testing the linux distribution I put together yesterday. The latest BSG episode (”The Passage”) is downloading now. Yesterday I caught the Firefly pilot while putting together the installer: man, what a good show. Even having seen it before, it really holds up well to repeat viewings. Actually, it is even better having seen the whole series and the movie, because the writing and the acting are spot on right from the start. You know what’s going to happen, and the actions of the characters is totally consistent with that story arc.

My wife and I were talking while I was making us breakfast this morning about the shopping trip that she had with our sons yesterday. They had gone to a retail store, which shall remain unnamed, and she had a terrible time finding what she needed to get. I looked up from what I was doing and said “Man, that’s funny. You know that we turned down a contract with them, right?”

“Really?” she said, surprised.

“Yeah,” I replied. “They wanted us to fix some problems that they had with their website and make some improvements to the site. After looking at it for a couple of days I turned them down, as the job probably would have ruined us. The system was such a mess that there was no way we could fix it with any reasonable budget and we might have ended up liable for the whole system if we couldn’t make it work.”

“Oh my God,” she said, her eyes alight with interest, “did you know that I went on a total *rant* to one of our friends about their website on Thursday!? It’s completely unusable! I couldn’t actually see what I was ordering and I couldn’t see how many items were in my shopping cart. That’s why I had to take the boys in to one of their stores to get what we needed.”

“Actually, ” she went on, “I mentioned it to one of our friends, so she decided to go online and take a look at it herself, and she couldn’t use it either. The entire site is just incredibly bad. It’s unbelievable, I mean, this company has a national chain of retail outlets, it’s not a small business or anything like that.”

Somehow I wasn’t too surprised. I leaned back against the counter and told her what had happened.

A Simple Little Job…
When the client approached us, they described the work as fairly minor. I drove down to their offices and met the manager in charge of the project, and he described the work as fairly routine. The work he wanted sounded very doable, so I told him that we’d be interested in taking on the project. I told him what our standard hourly rate was and he blanched and told me that this was substantially higher than what they typically paid. I wasn’t too surprised, for routine development there are definitely cheaper alternatives than going with us.

He said that he might be able to pay a figure that was almost half of what I’d quoted him and said that it would be considerably higher than anything that they’d paid before. He told me that most of the work had been for $20/hour, and that they’d paid one contractor $30/hour. He then went on to say that they might be able to pay us a little more because they’d also like to get the code “cleaned up a little bit.”

At this point, a little voice in the back of my head said “Uh oh, this sounds familiar.”

The story he went on to tell was one I have heard far too often. The entire website and purchasing system had been kludged together over the years by several different developers, each offering cheap work and a promise to fix the system. At one point the company had acquired another business and had attempted to merge the two websites, again enlisting the services of the lowest bidder.

Various individuals had been hired on the cheap to try and fix the system. Most of them worked on it for awhile, took some money, and then left the manager with a site somewhat more unstable than it had been before.

I asked if there were any documents describing the architecture. No.

I asked if there was anyone that could provide me with an overview of the architecture. No, but there were a few people that administered the system that might be able to tell me a little more about it than he could. There was also a QWest employee that had been moonlighting on the project (but had given up in failure) that might be able to tell me a little more about the current system.

I asked if they had a testbed that we could use for testing out new code. No, they didn’t have one, hey, that sounded like a pretty good idea though, if they could afford it.

I got the number of the QWest employee, promised to get in touch with him, and headed back to the office with a bleak feeling in the pit of my stomach.

A fairly long phone call with the software engineer from QWest confirmed my worst fears: the system wasn’t just a mess, it was an inoperable mess. Several different developers had gone in and worked on the code base, and the system was a mess of dead code, bad code, and redundant code. A few tentative efforts to make even the simplest changes had bizarre, unpredictable results. And since this work was being done on the production system, any thing that broke the web site (uh, “broke it more”) would become a highly public, expensive PR disaster.

What they had was a broken arm, and they were hoping that a band-aid would be enough to fix their problem. The site needed what it should have had in the first place, a real architecture, an actual testbed, and a team of people that actually knew what they were doing. Realistically speaking, a complete re-write from the ground up was the only sane way to go. Trying to come in cold, get up to speed on the current “design” and refactor it into something that would work and be maintainable would take about as long and have a substantially lower chance of success.

Screwed
They had problems, but if we agreed to take on the work, we’d be screwed. There was no way I could take on this job with a clear conscience. Since the customer was already coming at this from the mindset of “let’s get this done as quickly and cheaply as possible” there was no way that they would accept the idea of paying for a full re-implementation. The very best we could hope for from them was a green light to try and make the broken system work in one or two months (which was hopeless) for about half of our usual pay rate. We would be doomed to fail before we even started the work.

We’d be taking responsibility for making it work, at a losing payrate, with no hope of success. Taking on the project would tie up our guys, limit our income, and embroil us in what one of my friends colorfully refers to as “a complete butt noodle*.”

I was fairly sure of the outcome, but decided that the manager deserved to hear my recommendation: pony up the green and start from scratch. Get a real architecture and a real testbed, and do the job they should have done four years ago instead of trying to get by as cheaply as possible.

Needless to say, he declined, and we went our separate ways.

You really DO get what you pay for
The real irony here was that the decision to do things as cheaply as possible had ended up costing them considerably more than they’d saved in the first place by trying to get the job done as cheaply as possible. Not only did they end up hemmoraging cash trying to fix the mess they had, they were losing business with their god-awful website.

“Quality” is often not cheap.

Unfortunately, not everyone knows enough to call “bullshit” on a proposal that costs a lot less, but doesn’t have the legs to meet your needs. In the case of the client mentioned here, the person in charge was NOT technical (they were actually in marketing).

I know of several instances in which companies have gotten burned by deciding to do things as cheaply as possible. This client simply did a more spectacular job of it than some of the others.

Let’s face it, it takes a certain amount of work to do things right. Most people don’t skip getting a foundation built for their house, skimping on a critical resource like your e-commerce infrastructure makes even less sense.

-Daniel

*Rich has never said exactly what a butt noodle is, but when you see one, you recognize it immediately. Trust me, the term will fly unbidden to your own mind at some point in the future.

Filed under: Dan, Professional Services by Daniel