Tedious Egghead Thoughts
Autonomous process management. That will get you ten results on Google. “Process management” will get you sixty million results. But I’m into something more specific.
All I know about process management is what I know about computer programming. Here is my logic (or rationalization). Software is a process; in fact, that’s another word for a program while it’s running. Writing software is a kind of management—that of a computer system. Computer programs are autonomous, because once they’re out in the world, the programmer can’t monitor them. So I came up with that term.
It’s not exactly the kind of thing to make people sit up and pay attention. But it’s important, if you think that computer software is important, which most people seem to do, these days, what with the Internet and all that. And I wanted to focus on general characteristics of software, so more common phrases like “software development” and “software engineering” just didn’t fit the bill.
The idea of software is that, to put it simply, you write a bunch of instructions and tell a computer to follow them. The instructions just tell the computer how to push some numbers around, although all it is actually doing is turning lots of little switches on and off very quickly. The switches represent numerical values. The numbers, in turn, represent everything else, like letters of the alphabet, bank account balances, e-mail messages, songs, movies and all the other stuff that you find on computers today.
All that stuff gets into computers because of software, and somebody has to write software, and hopefully they write it well. The rest of this article will only interest you if you think that the quality of software is important.
There are probably three major areas of concern devoted to software quality: software engineering, software architecture, and software correctness. Then there’s the various software construction methodologies, many of which don’t fit into those three theoretical groups, the most important being probably Agile, and it’s test-first philosophy. However, they annoy me, because they simply avoid the question of quality by pushing it off into the testing. They seem to ignore the fact that if you based your quality on testing, then suddenly you have to have a way to evaluate the quality of your tests. Then again, I’m behind in my theoretical reading. (I’m behind because there’s too much and it’s totally disorganized and highly politicized, and I want to have in my head a good set of questions before I go off and read people’s answers to their own questions, which might not be the same as mine.)
A bad question, for example, is “What is the best way to do … ?” where “…” is anything you want to try to do with a computer (or any other kind of machine, for that matter). The best way is anyone’s guess. What we want, in fact, is a sufficiently good way, which comes down to how well our method satisfies the constraints of our task. Specifically: time, energy, and reliability. Perfection is not attainable. We just want to be conservative.
If your task is simple enough, you can probably just implement whatever comes to mind and see how it works out. If the first try doesn’t work, try another one, until you have something working. If you get close, you can usually refine your method, finding the weak parts and modifying them or optimizing them, until you have something feasible.
This might not work with extremely complex processes. In any case, it seems wasteful. There should be some way to determine some good strategies for specifying a task by considering the nature of the task itself. There is always room for innovation, but figuring out how to apply tried and true techniques to new processes is more reliable. Probably ninety percent of the work of any task—the sub-processes—will involve common techniques. Ten percent might benefit from innovation, and that will likely be the same ten percent where most of the actual important and unique work is done.
A major problem in specifying the method to perform a task is that, often, the nature of the task itself is not well known, since perhaps it has never been performed before, but has only been considered a possibility. Nevertheless, if the majority of the work is not novel, is based on well-known kinds of work, then at least we can try to distinguish that from the unique elements.
The essence of automated process management is figuring out how to break a task down into smaller, sub-tasks, and then breaking them down, too, until you have a bunch of jobs which should mostly be obvious. The ones that aren’t obvious are the ones which distinguish this task from anything that went before it. In terms of software, that’s your unique ability which hopefully makes your software useful in a way that nothing else out there is.
Software isn’t the only thing that performs complex tasks, however. Televisions perform tasks: displaying video. Assembly lines perform tasks: assembling widgets. Construction teams perform tasks: building buildings. Software, like these other things, is only partly autonomous, otherwise requiring the input of human beings, which act like overseers. They are, in a sense, like movie directors or orchestra conductors or other kinds of supervisors, although humans are, if not more reliable, then at least more adaptable than software (which isn’t adaptable at all). The idea of autonomous, in fact, at least in the context in which I’m writing, is that of complete and total lack of creativity. An autonomous process is a fully mechanized process, designed to operate in a completely known environment, or at least within well-known parameters. The parameters define the starting point for the task and the end point, and the process defines the way to get there by following instructions provided in advance.
One realization I have recently had is that a common metaphor in software development is, like many metaphors, badly chosen. Often, software is compared to a building, and the process of writing software to the act of construction. However, it would be better to consider the software itself as a construction team, and the building as the data which is transformed from raw materials into something useful. Software developers are thus not only designing the end product, but the process for creating it, and it is in fact the construction process which is the most difficult to understand, although the process and the product are of course closely related.
From that perspective, software is much more like project management. What makes that awkward is that you need project management to help you organize the development of the software itself. But that’s the nature of software development: defining a process for defining a process for creating useful data out of raw input data. It’s the second process that’s the most important. The product may ship as code, but that’s deceptive. The product is really the behaviour that the software engenders in the computer on which it runs. It’s dynamic, unlike a building, which is static. Data is static. Programs are not, except when they are not doing anything. But a program that is doing nothing is worth nothing. How it does what it does, and how well, is what matters. And the same is true for any process.
Given a starting point and an ending point, the first job of a process designer is to identify the sub-processses, as I mentioned. Sub-processes fit into two categories: inter-dependent, or linear, and independent, or orthogonal. Linear sub-processes must occur in a specific, defined order. Orthogonal processes can happen in any order. Most complex tasks involve a combination of linear and orthogonal processes. In product assembly, you can produce the various component parts in parallel, but then there is usually an order to how they are assembled, which depends on the structure of the finished product.
In fact, if you can successfully, and correctly, distinguish the orthogonal from the linear elements in a complex process, you have gotten a significant head start in defining your process.
Why do I care about this abstract, headache-inducing theory? Because all the coolest stuff on earth is insanely complex and hard. All of my favourite stuff depends on successfully managing the process underlying it. I’m an engineering fan boy. I like space craft, gigantic bridges, computer networks, and all the science fiction wonders of technology. Have you ever seen pictures of a super-collider? It’s hard enough to design those things in the first place. Imagine the effort building them! Imagine trying to design factories that would manufacture them! Have you ever seen a microchip fabrication plant?
Well, software is more like a little, virtual factory than anything else. Data goes in, information comes out, usually with some help from a human, although you might also consider the user interaction itself as simply more raw data. My goal is to try to understand how that works, and to come up with some good insights into how to make sure that it works well, and that we know exactly when and how it fails. And that’s where process management turns into software engineering.