Failures in Software Engineering
And why your organization can’t do large development projects.
These are lessons that I learned from running my first large software development project, the obstacles that I smacked head first into, and what lessons we can learn organizationally. I can write about this in a some what optimistic manner because at the end, we did manage to put out a pretty damn good system, though in hindsight, we could have probably done so in much less time.
The entires are numbered, though not by any significance. This post is the first entry only because it contains lessons that I learned.
- The wrong person is running the show and he wrote code for his PhD.
Having me be the technical lead for this project was probably not the best idea in the world. It was my first time running a project of this complexity (basically one that me working alone could not have finished in a six months time) Despite actually having a computer science degree, the technical term of people like me is coding weenie and not software engineer. The difference is that while I can program, I treat coding as a tool to solve a problem, and I do not take kindly to people telling me what the requirements are, what buttons to put where, and how exactly they want to see the output. If I think SDTM is a terrible standard for storing and querying data thought up by statisticians to make their narrow use case easier, I will fight you every step of the way if you tell me that you want an API that returns variables as sliced and diced SDTM domains. I will fight you every step of the way if you give me a requirement that is ambiguous and doesn’t make sense rather than make it so i.e. I want you to automatically join two variables, but I won’t tell you what the join conditions are. A proper software engineer knows not to fight this now, but to implement what you ask for, then when it fails points to your requirements document saying that it does exactly what you told him to do.
Anyways, coding weenies spend too much time fighting for what’s right rather than realizing that the power of software is iteration and fixing things when they don’t perform to expectation. This is also why software projects end up being late. But sometimes we must realize that fighting for the perfect solution also wastes a whole lot of time, and it’s better to be late and have the customer understand it’s their fault and sharing the responsibility, rather than having it all fall on you. Joel Spolsky has warned about hiring PhD’s and while I disagree with him on it because I have a PhD, the mentality of build it, ship it, fix it later does have some merit.
As an aside, I disagree with him that PhD’s aren’t about getting things done. PhD’s are all about getting things done. It’s just that we don’t want you to tell us how to get it done. And if you do. You better be right because we can see through your bullshit.
The second thing in this category is that anyone who has written code for their PhD has a very high tolerance for bad code. Not so much that they are bad coders, but because we have to build on the work of others, and not everyone who works in computational whatever has formal training in coding. Therefore as part of my PhD, I saw some truly nasty code, and in the spirit of getting things done, doing the minimal work to turn it into a library, or worse piping output and parsing with PERL and piping into the input stream of my code are acceptable ways of solving the problem. Then if there is an error in the original code, I will wade through some god awful code in order to fix it. So when I see terrible code, provided that I can verify that it works properly through unit-tests and what not. I’ll let it slide. Code used to solve a problem for someone’s dissertation is basically spit, snot, twine and a bit of PERL mixed with a few nuggets of really elegant code.
Code quality however is damned important once you start adding people who aren’t PhD’s to the project and don’t have such a tolerance for terrible code. You start adding outside developers, and once the project gets big enough that you can’t fit it inside your head, you’re going to be in trouble. In our first iteration of this project, code quality was pretty terrible because timelines were compressed and well, get it done and out. We treated version two as learning lessons from version one, and ended up rewriting the entire thing from scratch.
The takeaway lesson from this is that if you’re going to run a large software project, make sure the guy running it has some experience running software projects first. And while I know that we all have to get experience somewhere and sink or swim is a pretty effective teacher, if you must get a newbie, don’t get the opinionated guy with the PhD. If you must use the guy with the PhD, stay out of his way. Tell him the problem and let him solve it however he wishes.
When Joel Spolsky thinks that PhDs aren’t going to be successful as software developers he’s right. In most environments the guy with the PhD should be treated as a resource that you give a problem to when everyone else is stuck. However, there are certain environments and companies where dudes with PhDs are phenomenally successful such as that company Google many of you might have heard of. It’s because they defined the problem, solved it their way, with minimal orders (only advice). I would bet that if you made Brin and Page right out of their PhD, write inventory management software for Walmart, they’d be bashing their heads against their desks every day, pissing everyone off, and probably giving some project manager heartburn as they miss deadlines and ignore requirement documents.
In the end I feel blessed because the organization that I worked for didn’t fire my butt at the end of release one, but allowed release 1 to limp along and gave me a second shot at the problem, this time with minimal interference and at the end I think we did something pretty neat.