Recently I’ve been reading the The Innovator’s Solution by Clayton M. Christensen and Michael E. Raynor, which seeks to analyze the process of creating and sustaining growth through innovation. As part of it, the authors explain how predictable end-results of a process can be obtained by building a good theory. This also applies to the software development process, where you must understand the process itself before you can make any meaningful recommendations for improvement and predictability.
An excerpt from the book which lays out the process for building good theory is below:
The process of building solid theory has been researched in several disciplines, and scholars seem to agree that it proceeds in three stages. It begins by describing the phenomenon that we wish to understand. In physics, the phenomenon might be the behavior of high-energy particles. In the building of new businesses, the phenomena of interest are the things that innovators do in their efforts to succeed, and what the results of those actions are. Bad management theory results when researchers impatiently observe one or two success stories and then assume that they have seen enough.
After the phenomenon has been thoroughly characterized, researchers can then begin the second stage, which is to classify the phenomenon into categories. Juvenile-onset versus adult-onset diabetes is an example from medicine. Vertical and horizontal integration are categories of corporate diversification. Researchers need to categorize in order to highlight the most meaningful differences in the complex array of phenomena.
In the third stage, researchers articulate a theory that asserts what causes the phenomenon to occur, and why. The theory must also show whether and why the same causal mechanism might result in different outcomes, depending on the category or situation. The process of theory building is iterative, as researchers and managers keep cycling through these three steps, refining their ability to predict what actions will cause what results, under what circumstances.