Tuesday 3 December 2013

The Annals of Soft-Despotism

It's Unique, So It's Gotta Be Good

A government which sets forth to care for everybody becomes an inevitable soft-despotic monster.  But the monster is incompetent, bumbling, and stupid.  Irremediably so.  The only solution the monster can proffer to overcome its bumbling failures is to throw millions more of "good money" after bad.  It's only solution is to double down on folly. 

Now this is not to say that private (non-state) enterprise is always efficient.  Far from it.  But private enterprise has an enforcer, a ruthless, take-no-prisoners punisher of business incompetence and failures.  Mr Competition is relentless in punishing and driving the bumbling incompetent out of business.  But the soft-despot has no competitors.  It is a despot after all, protected by law. 

We offer yet another illustration of the follies and self-destructive pathogens of a bumbling  soft-despotic government: the Obamacare website.

Obamacare: 500M Lines of Code, $500M, 60% Complete, No Surprise

30 Nov 2013, 9:59 PM

The historic IT disaster known as the Obamacare web site has been of intense interest to me, since I've been a Senior Software Engineer for decades, and I have personally participated in, witnessed, and reported on (as a technology reporter) a number of IT disasters. But even so, the size of the Obamacare web site catastrophe on October 1 still takes my breath away.

When I first heard, shortly after October 1, that there was 500 million lines of code in Healthcare.gov, I quickly rejected that figure, because it's impossible.


Going back to the development of IBM's System/360 operation system, as described in Fred Brooks' classic book, The Mythical Man-Month, the average programmer on the project wrote six lines of code per day. Of course, every programmer writes a lot more than that on SOME days, but on other days he writes zero lines of code, since he's doing testing or debugging or rewriting or documenting. So for System/360, it all averaged out to six lines per day per programmer.

So let's say that the Obamacare programmers were much better than that, and wrote 100 lines of code per day average. Let's say that there were 1000 programmers. And let's say that, over the three year period, there were about 660 business days. Then, with those generous assumptions, you get 100*1000*660 = 66,000,000 lines of code. It's simply impossible to reach 500,000,000.

And yet, the 500 million figure is apparently true. I've heard it dozens of times in the last month, and no one is denying it. Healthcare.gov apparently really does have 500 million lines of code. How is that possible?


I get a picture in my mind of 1,000 people sitting a computers typing code, without worrying about whether or not it works. Given the size of the catastrophe, some variation of that must have happened.

More important than that, a code base that size is unsupportable. Health services is a rapidly changing field, and every time there's some kind of process or rule change, it will take an army of programmers to make all the necessary changes in the code base. And that assumes that all the bugs have been fixed, which is far from true. Healthcare.gov will not be fully functional at any time in the foreseeable future, if ever.


On October 1, Healthcare.gov had 500 million lines of code, and could handle a handful of simultaneous users. Facebook.com has 20 million lines of code, and handles millions of simultaneous users.

Then there's the cost. Healthcare.gov should have cost $5-10 million to implement. Take into account government corruption and incompetence, it should have cost $10-25 million. Instead, it cost $300-600 million -- let's say $500 million. How do we get to that figure? Well, assume 1000 programmers are paid an average of $100 per hour ($200,000 per year) for 8 hours per day for 660 business days: $100*1000*8*660 = $528 million. So at least that figure makes sense -- as long as you understand that the Obama administration poured half a billion dollars into the pockets of his cronies and supporters, and got exactly what he deserved with Healthcare.gov.


How could President Obama have been so wrong?

There have been numerous reports that the Obama administration had been informed many times, including by McKinsey & Co. in March, that Healthcare.gov was in serious trouble. And yet, just two weeks ago, on November 19, President Barack Obama said:
I was not informed directly that the Web site would not be working the way it was supposed to. Had I been informed, I wouldn't be going out saying, boy, this is going to be great. You know -- I'm accused of a lot of things, but I don't think I'm stupid enough to go around saying this is going to be like shopping on Amazon or Travelocity a week before the Web site opens if I thought that it wasn't going to work. So, clearly, we and I did not have enough awareness about the problems in the Web site.
And so, since he posed the question of how he couldn't have been that stupid, let's try to find an answer.

Once again, I have a number of personal experiences that relate to this, and the main one I'd like to relate is the most bizarre day of my professional life.

In 1985, I was doing contract programming for Northrop Corp., developing embedded software for munitions guidance systems. The project manager had to be reassigned, and I became acting manager. After being in this position for 2-3 weeks, it was clear to me that the whole project was in trouble, would slip at least three months. I told this to the program manager, and he nearly freaked out.

I was then pulled into one meeting after another, and met high-level managers that I never knew existed. Their conclusion apparently was that I was full of crap, and they decided to fire me, but they weren't sure, so they decided to let me stay on until the release date, and then they would fire me.

About a week before the release date, there was a meeting in the lab, where the lead programmer was to demonstrate the embedded system to the Corporate VP. I attended this demo, but I was ordered just to stand there and keep my mouth shut. So I stood back, leaned against the wall, and just watched the proceedings.

The lead programmer gave his demo, and the VP ooohed and ahhhhed. He then asked, "And this will be ready for release next Monday?" The lead programmer said, "Yes, it will be ready on Monday." The other managers in the room also said, "Monday."

I was holding my breath through all this. I couldn't believe what I was hearing. Either I was crazy or all of them were crazy. That's why I call this the most bizarre day of my life.

Well anyway, to make a long story short, the project was not ready for release the following Monday. It slipped six months. I wasn't fired because one Silent generation manager went around and told everyone that "Xenakis was right, so he shouldn't be fired." So I wasn't fired.

I've had several other experiences like that, except that usually I was fired, even though I was always right. In 1992, I was working as a contract programmer for Fidelity. I did a little unit testing, and wrote a memo to my manager, listing numerous problems why the project was in trouble, saying that it would slip at least six months. The manager was so furious that he had smoke coming out of his ears. He fired me. The project crashed completely a couple of months later. I was right, and he was wrong, though I paid the price.

In a previous article ( "14-Oct-13 World View -- HealthCare.gov IT systems a continuing disaster") I related a 2005 experience where I was fired for telling my management that the $10 million project currently under development would slip 6-12 months. I was fired and out of a salary, while the dozens of incompetent engineers kept collecting salaries for over another year, at which time the disastrous project was canceled completely.

So there's really nothing about the Healthcare.gov disaster that's a surprise to me (except the 500 million lines of code). I've been a Senior Software Engineer for a long time, and I've worked on over 100 projects, so I've seen disasters. The only thing that's different about Healthcare.gov is the breathtaking size of criminality of the Obama administration in wasting so much money on cronies and corruption, and the breathtaking size of the resulting well-deserved disaster.

No comments: