Operations

Lessons learned in the discipline of operations, applicable both to management of mission critical systems and of information technology.

No crisis for me

I was chatting with Atul earlier today when he expressed his dismay that Slashdot was off the air and asked me if I could get there? No. Meanwhile, I realized I couldn’t get to SourceForge. This was proving problematic as I had just done a release of an Open Source project I’m the maintainer of, and was trying to update the project website. Guess announcing the release will have to wait a bit :(.

Oh well. “The internet must be broken somewhere”.

We then recalled that these are both services of OSDN. I recall a few years back talking with the system administrators there about crisis management in IT environments and using procedures to manage change. They said they didn’t need any help with their operations. “We’re all set”. Uh huh. People always say that when things aren’t going wrong.

–

Of course, it is now 3am in California. It actually doesn’t matter where your servers are — it’s always 3am when things like this go wrong. Personally, I take it as conclusive proof about the underlying nature of the universe that this sort of thing only happens when it is the middle of the night. You can’t possibly find a less pleasant time to force sysadmins to get out of bed and to go try and fix things. Frankly, I think that’s just Someone trying to tell sysadmins that they made a poor career choice, but you know, He (or She, or It, or They, take your pick) needs to have a good laugh too.

AfC

Get used to thinking for yourself

Bernard Golden writes:

Fifth, open source. This doesn’t mean occasionally considering it. And it definitely doesn’t mean evaluating it by the standards of how you’ve done things with proprietary software … People criticize open source because it doesn’t “deliver business value.” What they typically mean is that they’re used to letting the vendor do their job of deciding what their infrastructure should look like, then providing them a roadmap of their infrastructure development plans, and then pre-integrating the solution with the vendor’s favored software partners. So, naturally, when you look at open source, it fails to do that. No open source vendor is going to do a dog-and-pony show and then build your proof-of-concept [for you] to get you committed to their solution. Instead of asserting that open source doesn’t deliver business value, run an experiment. Find out for yourself what the costs of doing open source are. And besides, as open source economics eats away at the margins of proprietary vendors … they’ll do less of the legwork for you. So get used to thinking for yourself.

I really like this. For all the FOSS cheerleading I do, I don’t normally get embroiled in proprietary vs open source debates, and I hadn’t really thought of the point that Bernard makes here: one of the biggest reasons that people will resist change is because there are many who want their answers handed to them on a silver platter.

AfC

Deterministic thinking

We get used to thinking there is a “right” answer.

Most of business is built on that premise: if you add 2 and 3 you get 5. Too easy, right?

Much of programming is like that too. Indeed, the whole discipline of unit testing is built around the idea that given a constrained environment and controlled inputs we can say whether or not the output is correct.

But that’s not really the way things work in science and engineering.

In measuring the real world, it’s (2.0 ± .5) and (3.0 ± 15%, we hope). What does that add up to? Even figuring out what the error bars are is an exercise in judgement and approximation. Much of experimental science revolves around the evaluation of uncertainty in measurements.

It’s tough when you stare at something and start doubting whether you can believe what you’re looking at. Davyd Madeley, who works with seismic data, said something interesting the other day which got me thinking about these issues:

“Hopefully I will work out how to design a better debugging framework for datasets that are too big to printf … I can’t trust any layer to be giving me the correct results”

The vogue CS way of looking at this would be to scream “testing”. Certainly, unit and functional testing are an essential part of ensuring the different pieces of your application do what they are supposed to do. That’s a given. But in dealing with large sets of real world data and in applying algorithms to try and find hidden meaning in that data, there is often no “right” answer. How do you test that then? And how do you differentiate between miscalculation, and the noise inherent in the quality of the data?

Little of what we do in modern day information technology conveys error in addition to scalar quantities, so it’s difficult to even know where to start. We tend to think very deterministically in IT.

What does this inflexibility mean for the broader issue of reliability in our systems?

It means we’re not prepared for things going wrong. We’re not prepared for uncertainty. We say to ourselves “oh, it’ll be fine once things get back to normal,” but we’re not owning up to the reality that there is no normal. Change, interruption, and crisis are the only equilibrium.

This question of flexibility comes up in the procedures work I do for clients who run business critical systems. On the one hand you try to create precise instructions that explicitly capture the necessary sequence of events. On the other hand, not only do you need flexibility of mind, you need flexibility of practice in order to adapt to the changing circumstances and uncertainty that defines high pressure operations environments. It’s error bars all over again. Where most people run into trouble is that their procedures don’t encompass the fact that they may run into trouble. The techniques to deal with this are not that hard to learn, but what makes it elusive is that this is about more than IT. It means shifting the way the organization solves problems. That’s a bigger challenge.

Trapped in our usual we-already-know-all-there-is-to-know mindset, we can both tell just how it’s going to work out, right? After all, we’re all thinking deterministically about it. And my oh my, won’t we be surprised when tomorrow doesn’t turn out to be a normal day like we were counting on. But if we find the courage to face up to the fact that there might not be a right answer, then perhaps we’ve started down the road to making systems, companies, and communities that can stand the test of time. Or at least the test of next Sunday.

AfC

In Memoriam

An ex-cadet was killed in Afghanistan a few weeks ago. 22458 Captain Nichola Goddard, RMC class of ’02, was calling indirect fire in support of her combat team when her vehicle was hit.

In reading some of the tributes sent in by her comrades-in-arms and classmates, I came across this:

“Nichola was one of the few people who truly understood what she was doing in the army. It wasn’t just a job, nor was she playing soldier or in for a free degree. She had thought through the responsibilities and ramifications of taking that oath we all took. Too few cadets are truly prepared – intellectually, morally – for the real job of soldiering – but she was. She was someone who gave her whole self to everything she did, and could always be counted on to give a hundred percent, even when we were all out of gas. She was intelligent, compassionate, generous and energetic. Nichola was one of the few people who would do the right thing, always. And make sure it was done right.

We need more people like her in the world, which makes it all the sadder that the world has lost her.”

— 21982 Adrian Rawle

Would that we all have this said of us. People like this are ones to look up to. Even when they’re gone.

And that’s the whole point.

AfC
19809
RMC ’95

The leaders and the led

The other day I was discussing the subtleties of leadership with a client. I was mentioning an opinion I formed during my early days at RMC that there were two kinds of people: those who did what needed to be done, and those who waited to be told what to do.

This in turn reminded me of a passage from one of my favourite books, C.S. Forester’s Lieutenant Hornblower (that’s pronounced ‘leftenant’ for all you Americans reading this). As I looked it over, I was reminded of how complex the human condition really is.

The scene is the quarterdeck of HMS Renown. Thinking over Hornblower’s adroit handling of his superiors, Lieutenant William Bush (also senior to Hornblower and from whose point of view the story is told) looks over at Hornblower and starts to reflect:

“Bush was learning something about personalities. He would never be able to reduce the results of his observations to a tabular system, and it would never occur to him to do so, but he could learn without doing so; his experience and observations would blend with his native wit to govern his judgements, even if he were too self-conscious to philosophise over them.

He was aware that naval officers (he knew almost nothing of mankind on land) could be divided into active individuals and passive individuals, into those eager for responsibility and action, and into those content to wait until action was forced on them. Before that he had learned the simpler lesson that officers could be divided in to the efficient and the blunderers, and also into the intelligent and the stupid — this last division was nearly the same as the one immediately preceding, but not quite. There were officers who could be counted on to act quickly and correctly in an emergency. And those who could not — again the dividing line did not quite coincide with the preceding. And there were officers with discretion and officers with none, patient officers and impatient ones, officers with strong nerves and officers with weak nerves. In certain cases Bush’s estimates had to contend with his prejudices — he was liable to be suspicious of brains and of originality of thought and of eagerness for activity, especially because in the absence of some of the other desirable qualities these things might be actual nuisances.

The final and most striking difference Bush had observed during ten years of continuous warfare was that between the leaders and the led, but that again was a difference of which Bush was conscious without being able to express it in words, and especially not in words as succinct or as definite as these; but he was acutely aware of the difference even though he was not able to bring himself to define it.”

— C.S. Forester, Lieutenant Hornblower (Little, Brown and Company, 1951; republished by Back Bay Books, 1998) page 75.

So which are you? To be sure we don’t live in the harsh starkness of the 18th Century Royal Navy. But the amazing thing about leadership is that it is not static. It can be learnt. When we’re fortunate to have good examples to follow, people we can look up to and situations which force us to go beyond what we thought we could do, that’s when we look within ourselves and become the people we most want to be.

AfC

Paul Kedrosky, Venture Capitalist.

For those of you who are addicted to reading blogs, I recommend Paul Kedrosky. He is a once upon a time techie who is now a fairly prominent venture capitalist (!). He blogs compulsively (frequently quoting other very with-it people), is highly opinionated (on just about everything), and has a pretty good take on trends in the technology marketplace.

A recent favorite was post titled “Just Say No to Marketing“: first he quoted an awesome comment from Bob Feld:

I despise the word “marketing” — it’s often the weakest link in a startup company. “Marketing” is vague and non-specific, often poorly executed and measured, and usually a huge waste of money relative to the output.

Several years ago, I suggested to my portfolio companies that they fire their VP of Marketing and hire a VP of Demand Generation (it could be the same person if the VP of Marketing was willing to accept a quota and meaningful, measurable variable compensation.) Hopefully, this VP of Demand Generation understands the incredible power of having your customers so happy with your product that they’ll talk about it online.

then Kedrosky wrote:

I often tell companies I’m working with that too often marketing (and business development, for that matter) is a joy-ride, essentially sales without a quota. One test when hiring a marketing veep is to tell him/her that 20/30/50% of their comp is variable on revenue — say it even if you don’t mean it — and then see how they react. It’s usually instructive as heck.

The deer-in-the-headlights interview question. Gotta love it.

AfC