Peter Naur

Lehman on Software, Models and Change

/ Adam / Leave a comment

The modeled and evolving quality of software comes to the surface when thinking about software maintenance. A classic paper on this is Lehman’s 1980 paper Programs, life cycles, and laws of software evolution, which lays out the territory with great clarity, then confidently strides off in the completely wrong direction.

Model

Lehman introduces a distinction between 

  • S-programs, that implement a formal mathematical (s)pecification, such as the travelling salesman problem
  • P-programs, that solve some messy (p)roblem arising in the world, such as actually scheduling real salespeople and their ambiguities and partly-known preferences, and 
  • E-programs, those (e)mbedded in and changing the world they directly model, such as in air-traffic control.

For P-programs and E-programs, “the acceptability of a solution is determined by the environment in which it is embedded.” The distinction is in the programs relationship with its origin story: between top-down and bottom-up; transcendence and immanence.

Lehman goes on to note P-programs are also in a feedback loop arising from their use in the world. Their execution is observed, even lived, by their users, and this results in demand for change. 

This is a cybernetic view, though Lehman doesn’t use the terminology. The paper sketches some more complicated loops, particularly where a specification intermediates between the P-program and the world. It is that intermediation, rather than feedback, that is foregrounded in the difficult and famous statement on code:world relations:

Any program is a model of a model within a theory of a model of an abstraction of some portion of the world or of some universe of discourse.

Lehman drops this on page two, before defining S-, P- or E-programs, and never gets around to defining theory or model, or otherwise directly elaborating, disconnected sagelike pronouncements being an expected feature of software engineering papers of the time. Cook (and a team including Lehman) later link this to the social process of Kuhn’s paradigm shifts – renaming P-programs to (p)aradigm-programs and E-programs to (e)volving-programs.

Weisberg’s work on the crucial role of models in science could also help. For Weisberg, a theory maps a model to the world through (mostly explicit) construals. This plays a similar role to “abstraction” in Lehman’s definition. (Bit more on Weisberg here.)

It’s also worth throwing Naur’s “Programming as Theory Building” into the mix, though his paper does not make much distinction between model-as-code and theory-as-code.

Lehman also introduces “laws” of software evolution, which did have some empirical basis, but appear hard to reproduce. They might be compared to more recent work on meaningful descriptive code metrics, or properties of software as a material.

The Rivers and The Lakes That You’re Used To

After accelerating through insight after insight into the fluid and evolving nature of software, Lehman closes off the theory section by casually inventing microservices (in 1980), then taking a screaming left turn at the Process Street T-Junction, crashing through a railing and tumbling over a cliff. For over that cliff flows a process waterfall, and in the structured programming era, there’s nothing more attractive.

Like the rest of the structured programming crowd, he has factory envy: “An assembly line manufacturing process is possible when a system can be partitioned into subsystems that are simply coupled and without invisible links … Unfortunately, present day programming is not like this.” Lehman goes on to emphasize the care and structure needed when writing separate elaborate requirement and technical specifications. You get the idea. The remaining process recommendations I’m just going to skip.

It is easy to be wise after the fact in 2019. Agile practices and literal miniature software assembly lines (continuous build infra) now exist, and have made us much more sensitive to the damage done by scope size and delivery delay in large software systems. Trying to solve complex problems with more upfront planning was a high modernist worldview going out of fashion, but still very much in the intellectual water in 1980: Lehman gave a lecture in 1974 referencing both city planning and the Club of Rome report Limits to Growth. Perhaps it would be fairer to point out that thinkers who advocated short simple changes as a response to complex systems – like Jane Jacobs, or John Boyd and his OODA loop – were always tacking into the intellectual wind.

—

References

Cook, S., Harrison, R., Lehman, M.M. and Wernick, P.: ‘Evolution in software systems: foundations of the SPE classification scheme’, Software Maintenance and Evolution Research and Practice, 2006, 18, (1), pp. 1-35  
Lehman, M.M. , “Programs, cities, students – Limits to growth?” Inaugural Lecture, May 14,  1974, ICST Inaugral Lecture Series, Ed., VOI. 2, pp. 147-163, 1979. vol. 9, pp. 211-229,  1970-1974; and in Programming Methodology D. Gries, Ed. New York: Springer-Verlag, 1979,  pp. 42-69.
Lehman, M. M. (1980). Programs, life cycles, and laws of software evolution. Proceedings of the IEEE, 68(9), 1060-1076.
Naur, P. (1985). Programming as theory building. Microprocessing and microprogramming, 15(5), 253-261.
Weisberg – Simulation and Similarity.

Metis and General Intellect

/ Adam / 1 Comment

“The general intellect” can be interpreted as tacit craft knowledge embedded in individual cunning and social relations. This definition sets general intellect in contrast and opposition to formal information systems. Framing it this way may not be completely true to historic usage, but has some revealing consequences. It applies to either abstract information machines like traditional bureaucracies, or concretized ones, like specific computer programs. Craft and process arguments in software development are also a lens on this transformation of skills in cognitive capitalism, information valorization and their relationship with bureaucracy and flows of stupidity. To expand the general intellect is to accelerate.

The term the general intellect (sometimes general social knowledge) originates from Marx and was used by operaismo thinkers to grapple with the emergence of cognitive capitalism. Virno states:

Whereas money, the “universal equivalent” itself, incarnates in its independent existence the commensurability of products, jobs, and subjects, the general intellect instead stabilizes the analytic premises of every type of practice. Models of social knowledge do not equate the various activities of labor, but rather present themselves as the “immediate forces of production.”

They are not units of measure, but they constitute the immeasurability presupposed by heterogeneous operative possibilities.

They are not “species” existing outside of the “individuals” who belong to them, but axiomatic rules whose validity does not depend on what they represent. Measuring and representing nothing, these technico-scientific codes and paradigms manifest themselves as constructive principles.

From here I suggest general intellect as a form of tacit and social knowledge, of metis, defined in contrast to formal knowledge, numeric or systematic knowledge. The general intellect is cognition but it is not data. It is highly contextualized, by experience, locality and specific social relations. James C. Scott offers the knowledge embodied by a soccer team or a ship’s pilot as good examples of metis. He also uses it in contrast to systematic forms of knowledge used in high modernist, often Fordist projects of top-down political control, state or corporate.

At the economy scale, the general intellect also seems to have an intersection with the idea of “institutions” in economic development. Institutions established ongoing government policies but also more abstract things like property rights and the rule of law; they are not primarily buildings, but persistent social relations, not commoditized or readily transferable between nations. Acemoglu and colleagues found significant correlation between historic settler mortality and modern economic success, putting forward the type of colonial institutions as the causative link.

Though this is oppositional with computation as a form of knowledge, the two are complementary in production. Operators draw on the general intellect to make machines work and produce things. This is true of concrete machines, like a coffee maker, or abstract machines in the Simondon / Deleuze / Guattari sense. And machines, especially large abstract machines, make use of operator black boxes to be effective. Traditional bureaucracy before the advent of modern computers and networks is an example of an abstract information machine which uses human operators as black box components – for instance to persist information to longer term storage, by writing on paper.

If we look at the skilled cognition involved in designing complex machinery, such as computer programming, we find the general intellect in the sense of individual and organizational craft knowledge. The rise of agile software development techniques, emphasising teamwork and craft skills over Fordist or high modernist planning, is one example of this over the last fifteen years. Yet the act of programming depends very much on an individual mental model, as pointed out by Naur. Programming is not typing; the main productive activity in programming is building a coherent mental model, the actual executable code produced is a side-effect. “Programming in this sense must be the programmers’ building up knowledge of a certain kind, knowledge taken to be basically the programmers’ immediate possession” (Naur). The spread of algorithms and software throughout society would then suggest a shattering of the general intellect into millions of shards of specific intellects. The general intellect – the entirety of system relations – could decay even as systemic shards expand in sophistication.

None of this is deny a certain translatability from metis to formalized knowledge. It can all be boiled down to bits in the end. Translation for functional use is costly and lossy, though. The mechanics of deep learning parallel this metic transformation from formal data structures into occluded knowledge. To understand how a deep learning system internally navigates a problem space requires a separate systematic analysis alien to the learning mechanism of the algorithm. Deep learning is a localized preview of machinic metis.

A countertrend to the fragmentation of general intellect might be the success of open source, but the point of open source is precisely to make the executable details of machines more readily available through social processes. It is a common platform rather than a general intellect, where evolution of the platform happens through patches (explicit formal communication) rather than primarily through evolved social understanding, though those dynamics still exist. It is striking that open source communities are organized primarily around a specific machine or platform rather than user products. This is true from the GNU C compiler through to the Apache web server and git source control. They echo Simondon’s critique of objects made in capitalism not evolving but merely accumulating features. Simondon’s comments on technical culture also parallel general social knowledge:

Now that he is a technical being no longer, man is forced to find for himself a position in the technical ensemble that is something other than the position of individual.

The trend for computer programming to promote skilling up for designers but sometimes exporting deskilling elsewhere was noted by Mackenzie in 1984; it’s because capitalism is a valorizing process rather than a deskilling process per se. Likewise there are deskilling trends in the software industry around outsourcing highly specific work to remote or offshore teams, so long as it promotes valorization (increases shareholder value).

The frustrations of working in or with a bureaucracy are often those of being a black box cog in a larger abstract machine, either through alienation from the meaning of the work, or because the work actually causes an undesirable effect which conflicts with personal goals, or even the stated goal of the organization itself. That is a form of stupidity but relates to all bureaucracy. Deleuze and Guattari say in capitalism:

The apparatus of antiproduction is no longer a transcendent instance that opposes production, limits it, or checks it; on the contrary, it insinuates itself everywhere in the productive machine and becomes firmly wedded to it in order to regulate its productivity and realize surplus value which explains, for example, the difference between the despotic bureaucracy and the capitalist bureaucracy.

eg in a despotic state the army may come and confiscate food and labour from a subsistence farmer when fighting a war, but in capitalism this military anti-production is in the form of a military-industrial complex, production interleaved with anti-production. Yet those critiques could apply to Soviet socialism too; only capitalism manages to create demand and ensure lack in the midst of overabundance.

Deleuze takes stupidity as an inability to dissociate from presuppositions, sense rather than common sense. Contemplating flows of stupidity, I am reminded of the slogan of engineer Jesse Robbins for making useful things in corporate bureaucracies: “Don’t fight stupid, make more awesome”. This could also serve as an accelerationist slogan, and can be critiqued the same way. Are you pushing forward as an elite ignoring politically hard problems, or are you building a concrete alternative solution that will attract change? Are you wasting time trying to change a system accelerated beyond human comprehension, or are you using accelerated human components to build a new system?

References

Acemoglu et al – The Colonial Origins of Comparative Development
Beck et al – Agile Software Development Manifesto
Deleuze and Guattari – The Anti-Oedipus
Garton – Excavating the Origins of Accelerationism
Land – A Quick and Dirty Introduction to Accelerationism
Mackenzie – Marx and the Machine
Naur – Programming As Theory Building
Scott – Seeing Like A State
Simondon – On The Mode of Existence of Technical Objects
Virno – Virtuosity and Revolution: The Political Theory of Exodus, in Radical Thought In Italy
Williams / Srnicek – #accelerate Manifesto for an Accelerationist Politics

Symbiotic Design

/ Adam / Leave a comment

Do we build code, or grow it? I was fortunate enough to attend a Michael Feathers workshop called Symbiotic Design earlier in the year, organized by the good people at Agile Singapore, where he is ploughing biological and psychological sources for ideas on how to manage codebases. There’s also some links to Naur and Simondon’s ideas on technical objects and programming that weren’t in the workshop but are meanderings of mine.

Ernst Haeckel - Trachomedusae, 1904 (wiki commons)

Ernst Haeckel – Trachomedusae, 1904 (wiki commons)

Feathers literally wrote the book on legacy code, and he’s worked extensively on techniques for improving the design of code at the line of code level. Other days in the week focused on those How techniques; this session was about why codebases change the way they do (ie often decaying), and techniques for managing the structures of a large codebase. He was pretty clear these ideas were still a work in progress for him, but they are already pretty rich sources of inspiration.

I found the workshop flowed from two organizing concepts: that a codebase is an organic-like system that needs conscious gardening, and Melvin Conway’s observation that the communication structure of an organization determines the shape of the systems its people design and maintain (Conway’s Law). The codebase and the organization are the symbionts of the workshop title. Some slides from an earlier session give the general flavour.

Feathers has used biological metaphors before, like in the intro to Working Effectively With Legacy Code:

You can start to grow areas of very good high-quality code in legacy code bases, but don’t be surprised if some of the steps you take to make changes involve making some code slightly uglier. This work is like surgery. We have to make incisions, and we have to move through the guts and suspend some aesthetic judgment. Could this patient’s major organs and viscera be better than they are? Yes. So do we just forget about his immediate problem, sew him up again, and tell him to eat right and train for a marathon? We could, but what we really need to do is take the patient as he is, fix what’s wrong, and move him to a healthier state.

The symbiotic design perspective sheds light on arguments like feature teams versus component teams. Feature teams are a new best practice, and for good reasons – they eliminate queuing between teams, work against narrow specialization, and promote a user or whole-product view over a component view. They do this by establishing the codebase as a commons shared by many feature teams. So one great question raised is “how large can the commons of a healthy codebase be?” Eg there is a well known economic effect of the tragedy of the commons, and a complex history of the enclosure movement behind it. I found it easy to think of examples from my work where I had seen changes made in an essentially extractive or short-term way that degraded a common codebase. How might this relate to human social dynamics, effects like Dunbar’s number? Presumably it takes a village to raise a codebase.

Feathers didn’t pretend to have precise answers when as a profession we are just starting to ask these questions, but he did say he thought it could vary wildly based on the context of a particular project. In fact that particularity and skepticism of top down solutions kept coming up as part of his approach in general, and it definitely appeals to my own anti-high-modernist tendency. I think of it in terms of the developers’ theory of the codebase, because as Naur said, programming is theory building. How large a codebase can you have a deep understanding of? Beyond that point is where risks of hacks are high, and people need help to navigate and design in a healthy way.

((You could, perhaps, view Conway’s Law through the lens of Michel Foucault, also writing around 1968: the communication lines of the organization become a historical a priori for the system it produces, so developers promulgate that structure without discussing it explicitly. That discussion deserves space of its own.))

Coming back to feature teams, not because the whole workshop was about that, but because it’s a great example, if you accept an organizational limit on codebase size, this makes feature/component teams a spectrum, not good vs evil. You might even, Feathers suggests, strategically create a component team, to help create an architectural boundary. After all, you are inevitably going to impact systems with your organizational design. You may as well do it consciously.

A discussion point was a recent reaction to all of these dynamics, the microservices approach, of radically shrinking the size of system codebases, multiplying their number and decentralizing their governance. If one component needs changes, the cost of understanding it is not large, and you can, according to proponents, just rewrite it. The organizational complement of this is Fred George’s programmer anarchy (video). At first listen, it sounds like a senior manager read the old Politics Oriented Software Development article and then went nuts with an organizational machete. I suspect that where that can work, it probably works pretty well, and where it can’t, you get mediocre programmers rewriting stuff for kicks while the business paying them drowns in a pool of its own blood.

Another architectural approach discussed was following an explicitly evolutionary approach of progressively splitting a codebase as it grew. This is a technique Feathers has used in anger, with the obvious biological metaphors being cell meiosis and mitosis, or jellyfish reproduction.

The focus on codebases and the teams who work on them brings me back to Gilbert Simondon’s idea of the “theatre of reciprocal causality”. Simondon notes that technical objects’ design improvements have to be imagined as if from the future. They don’t evolve in a pure Darwinian sense of random mutations being winnowed down by environmental survival. Instead, when they improve, especially when they improve by simplification and improved interaction of their components, they do so by steps towards a potential future design, which after the steps are executed, they become. This is described in the somewhat mindbending terms of the potential shape of the future design exerting a reverse causal influence on the present: hence the components interact in a “theatre of reciprocal causality”.

This is exactly what programmers do when they refactor legacy code. Maybe you see some copy-pasted boilerplate in four or five places in a class. So you extract it as a method, add some unit tests, and clean up the original callers. You delete some commented out code. Then you notice that the new method is dealing with a concept you’ve seen somewhere else in the codebase. There’s a shared concept there, maybe an existing class, or maybe a new class that needs to exist, that will tie the different parts of the system together better.

That’s the theatre of reciprocal causality. The future class calling itself into existence.

So, given the symbiosis between organization and codebase, the question is, who and what is in the theatre? Which components and which people? Those are the components that have the chance to evolve together into improved forms. If it gets too large, it’s a stadium where no one knows what’s going on, one team is filming a reality TV show about teddy bears and another one is trying to stage a production of The Monkey King Wreaks Havoc In Heaven. One of the things I’ve seen with making the theatre very small, like some sort of Edinburgh Fringe Festival production with an audience of two in the back of an old Ford Cortina, is you keep that component understandable, but cut off its chance at technical evolution and improvement and consolidation. I’m not sure how that works with microservices. Perhaps the evolution happens through other channels: feature teams working on both sides of a service API, or on opportunistically shared libraries. Or perhaps teams in developer anarchy rewrite so fast they can discard technical evolution. Breeding is such a drag when drivethru immaculate conception is available at bargain basement prices.

Ink and Rubber

/ Adam / 1 Comment

To a developer, an unfamiliar system can be like a picture drawn with an Etch-A-Sketch. You can see what is there, and some relationships between the pieces, by location, style or theme. When you have to alter the picture, though, difficulties arise. The Etch-A-Sketch eraser is quite coarse. It is easy to destroy existing parts of the picture. This is if you can find the eraser at all, and it hasn’t fallen behind the couch, or been chewed by the dog. If you really have to remove something, you may as well just pick up the whole thing and shake it back to a blank slate.

If you are very attached to the picture, it is still possible to add detail on existing empty ground. When large areas of white space are still free, you can choose a space to draw something new fairly easily. Once the picture is detailed, or you need to make a change related in a specific way to the existing picture, this is basically reduced to coloring in between the lines.

We code like we are painting with ink.

Our instinct is not to reuse and refactor, but to tweak and rewrite. Why? Fear and pride, perhaps. We aren’t born coding, though, so these emotional reactions are learnt because of some aspect of working with software. I’d suggest both behaviours stem from incomplete or untrusted knowledge of the existing code.

Peter Naur described programming as theory building.

Programming in this sense primarily must be the programmers’ building up knowledge of a certain kind, knowledge taken to be basically the programmers’ immediate possession, any documentation being an auxiliary, secondary product.

Naur is talking of a mental model, not the formal and external symbolic expression of an equation or a Java class. The model may need to be sophisticated, but it is also defined by its working immediacy. Your model of the code today may easily be different to a year ago, even if the code has not changed. Naur doesn’t relate the two, but to me it is also reminiscent of what sociologist Patricia Hill Collins calls everyday theory. Hill Collins is referring to theories of society, and she contrasts everyday theory with High Theory constructed in and for an academic setting, like, say, Marx’s dialectical materialism. Everyday theory is collective and social, may lack scholarly depth at times, but it also has the workaday vigour of something used every day to navigate a system.

If we follow Naur in considering the main activity in programming to be building mental models, even if the deliverables are software artefacts, one implication is the cost of building a theory – the cost of learning – dominates everyday coding. Writing a new class is then easier because the mental model is built by the writer while they are writing. At a larger scale, this suggests why the greenfield myth is so beguiling. Architecture suffers the same costs of understanding as programs. There’s a lot of interest in microservices at the moment, which is basically the rewrite elevated to an architectural ideal. I guess it works very well in certain domains and organisations, but I haven’t used auto-balkanisation of this sort myself, and at any rate there are many running systems not written that way.

Colouring-in behaviour is a bit harder to describe. It’s when a developer goes to painstaking effort to minimise changes to lines of code, especially structural elements like methods and classes. It’s similar to what Michael Feathers recently described as shoving, except he is using a different spatial metaphor.

Colouring-in is mentally cheaper because of scope. The scope of your working theory doesn’t have to include the interaction of class structures and broader parts of the codebase. From the perspective of the developer and that specific fix, it can even seem neater than more structurally impactful solutions, such as factoring out new methods or consolidating emerging redundancy. Poor unit tests exacerbate this, increasing the risk of errors after changes are made, discouraging refactoring.

Unit tests are also interesting for mental models because they are effectively worked examples of the theory the developer was using at the time. Good tests illustrate an idea as well as describing edge cases, and this decreases the cost of learning, and provides quicker feedback on the suitability of the programmer’s working theory.

The computational material of software is often brittle for a user in the way it runs, but the material itself is strikingly plastic at development time. It’s less like ink painted on paper and more like lego fused with rubber bands. Refactoring recognises this plasticity, and works with it. Compared to reinforced concrete, software is easy to change. Sometimes our minds are not.