Sinister Normative Compulsions

Michael Eby’s piece in New Left Review on Agile software development is interesting, and a little frustrating. I think it’s generally quite useful to get a left-wing, Marx-y class analysis on the structural of the software workplace. Eby argues Agile is a renegotiation within capitalism that by itself doesn’t challenge, and may indeed clarify, the executive power of the capitalist or manager, in the form of the Product Owner. This certainly seems true – it’s right there in the name.

I like the piece a bit more in concept than execution, though. There’s a few too many mistakes in the detail, getting the timeline of methods wrong, treating Scrum as separate to Agile when it’s the most popular method, talking about story points, and not lining up the jargon quite right in various ways. If I had to guess, I’d venture its a generalization from discussion with a few developers rather than a more careful reading of the key books and the c2 wiki. The full pedantic catalogue I will leave to the hacker news comments, while noting that process is more complicated to analyze than it looks, because software processes in practice rarely strictly follow a book. This is a point driven home by Alistair Cockburn’s description of developers always wanting to add stuff they don’t do to their description of their process – in Agile Software Development, as it happens.

Despite mixing up some of the details, Eby still has some sharp moments where he spots things you would never see in the usual software literature. Agile absolutely does have mechanisms of management discipline for workers, for instance:

It is clear that Agile dissolves many of the more visible features of hierarchical managerial control. But it does so only to recontain them in subtle and nuanced ways. For one, the self-organizing strategies of teams allow for certain workplace disciplinary mechanisms to take the form of normative compulsions rather than explicit instructions. 

The example that follows this quote isn’t great (again those details). I’d say instead that the most important disciplinary measure is frequent planning discussion and delivery – iterations and standups. It’s both a bug and a feature. Because they are the leverage points in this structure, iterations and standups are also where Dark Agile happens, if it happens.

But it’s still better. 

Agile work is more satisfying because a software worker has more control over the detail of what they produce. The owners and managers of the firm get more software delivery, through a change to internal communication and decision structures. To the degree society as a whole is helped by the software, it gets those helpful things sooner. It is too easy to forget how spectacularly wasteful waterfall development was. Enormous specification documents that were irrelevant by the time any code was cut. Months of kabuki theatre adversarial ping pong between development and testing teams. Years spent coding features and products never used. All of this was completely known and routine. Flows of waste and stupidity are hardly alien to software today, but at least in Agile a third of your projects aren’t thrown into the sea.

Agile techniques are more effective at delivering software, precisely by taking more technical decisions out of a bureaucratic org chart. Methodologies like Large Scale Scrum even emphasize that it is not a manager’s job to parcel out work, but to remove friction from the flow of money and production. This is surely fruit hanging low and ripe for a socialist critic to take.

Eby goes on to state:

[A] silent bargain between capital and wage-labour has occurred, with capital steadily shedding impediments to accumulation, and wage-earners forfeiting hard-won security in exchange for putative freedom.

This isn’t a terrible description of the new status quo, but again, the historical sequence seems off. The scythes of neoliberal deregulation had already been slicing into corporate and union bureaucracies alike all through the eighties and nineties. Downsizing, private equity buyouts and rightshoring were already well-rehearsed corporate practice before the Agile manifesto was signed in 2001. Agile has now shown itself so successful that it is propagated by management as corporate process. But you could also see it as skilled labour reacting to a disrupted expectation of long-term employment by taking control of low-level details of production, selling the change using increased labour productivity, and succeeding despite a lack of formal legal support. The lack of any explicit political theory (most of signatories of the manifesto are far from leftists) was probably helpful as well, in slipping under the radar, and avoiding the bureaucratic modes of 20th century unionism.

Various leftist thinkers, such as Phillips and Rozworski, have recently been pointing out that Amazon has so much size, computational power and control over its logistics that it is effectively a planned economy. This is true up to a point, but Jeff Bezos also popularized the “two pizza team” guidelines for agile team size. Internally Amazon has strong central control, platform planning, and for skilled software workers, devolved control of details to empowered teams that perform a high degree of horizontal co-ordination. And they build a lot of things. More Marx-inflected analysis of why that succeeds, while high modernist and Soviet central planning failed, would be welcome.

References

Cockburn – Agile Software Development

Phillips and Rozworski – The People’s Republic of Walmart

Lehman on Software, Models and Change

The modeled and evolving quality of software comes to the surface when thinking about software maintenance. A classic paper on this is Lehman’s 1980 paper Programs, life cycles, and laws of software evolution, which lays out the territory with great clarity, then confidently strides off in the completely wrong direction.

Model

Lehman introduces a distinction between 

  • S-programs, that implement a formal mathematical (s)pecification, such as the travelling salesman problem
  • P-programs, that solve some messy (p)roblem arising in the world, such as actually scheduling real salespeople and their ambiguities and partly-known preferences, and 
  • E-programs, those (e)mbedded in and changing the world they directly model, such as in air-traffic control.

For P-programs and E-programs, “the acceptability of a solution is determined by the environment in which it is embedded.” The distinction is in the programs relationship with its origin story: between top-down and bottom-up; transcendence and immanence.

Lehman goes on to note P-programs are also in a feedback loop arising from their use in the world. Their execution is observed, even lived, by their users, and this results in demand for change. 

This is a cybernetic view, though Lehman doesn’t use the terminology. The paper sketches some more complicated loops, particularly where a specification intermediates between the P-program and the world. It is that intermediation, rather than feedback, that is foregrounded in the difficult and famous statement on code:world relations:

Any program is a model of a model within a theory of a model of an abstraction of some portion of the world or of some universe of discourse.

Lehman drops this on page two, before defining S-, P- or E-programs, and never gets around to defining theory or model, or otherwise directly elaborating, disconnected sagelike pronouncements being an expected feature of software engineering papers of the time. Cook (and a team including Lehman) later link this to the social process of Kuhn’s paradigm shifts – renaming P-programs to (p)aradigm-programs and E-programs to (e)volving-programs.

Weisberg’s work on the crucial role of models in science could also help. For Weisberg, a theory maps a model to the world through (mostly explicit) construals. This plays a similar role to “abstraction” in Lehman’s definition. (Bit more on Weisberg here.)

It’s also worth throwing Naur’s “Programming as Theory Building” into the mix, though his paper does not make much distinction between model-as-code and theory-as-code.

Lehman also introduces “laws” of software evolution, which did have some empirical basis, but appear hard to reproduce. They might be compared to more recent work on meaningful descriptive code metrics, or properties of software as a material.

The Rivers and The Lakes That You’re Used To

After accelerating through insight after insight into the fluid and evolving nature of software, Lehman closes off the theory section by casually inventing microservices (in 1980), then taking a screaming left turn at the Process Street T-Junction, crashing through a railing and tumbling over a cliff. For over that cliff flows a process waterfall, and in the structured programming era, there’s nothing more attractive.

Like the rest of the structured programming crowd, he has factory envy: “An assembly line manufacturing process is possible when a system can be partitioned into subsystems that are simply coupled and without invisible links … Unfortunately, present day programming is not like this.” Lehman goes on to emphasize the care and structure needed when writing separate elaborate requirement and technical specifications. You get the idea. The remaining process recommendations I’m just going to skip.

It is easy to be wise after the fact in 2019. Agile practices and literal miniature software assembly lines (continuous build infra) now exist, and have made us much more sensitive to the damage done by scope size and delivery delay in large software systems. Trying to solve complex problems with more upfront planning was a high modernist worldview going out of fashion, but still very much in the intellectual water in 1980: Lehman gave a lecture in 1974 referencing both city planning and the Club of Rome report Limits to Growth. Perhaps it would be fairer to point out that thinkers who advocated short simple changes as a response to complex systems – like Jane Jacobs, or John Boyd and his OODA loop – were always tacking into the intellectual wind.

References

Cook, S., Harrison, R., Lehman, M.M. and Wernick, P.: ‘Evolution in software systems: foundations of the SPE classification scheme’, Software Maintenance and Evolution Research and Practice, 2006, 18, (1), pp. 1-35  
Lehman, M.M. , “Programs, cities, students – Limits to growth?” Inaugural Lecture, May 14,  1974, ICST Inaugral Lecture Series, Ed., VOI. 2, pp. 147-163, 1979. vol. 9, pp. 211-229,  1970-1974; and in Programming Methodology D. Gries, Ed. New York: Springer-Verlag, 1979,  pp. 42-69.
Lehman, M. M. (1980). Programs, life cycles, and laws of software evolution. Proceedings of the IEEE, 68(9), 1060-1076.
Naur, P. (1985). Programming as theory building. Microprocessing and microprogramming, 15(5), 253-261.
Weisberg – Simulation and Similarity.

Industrializing The Noosphere

Control Environment

We are not practicing Continuous Delivery. I know this because Jez Humble has given a very clear test. If you are not checking your code into trunk at least once every day, says Jez, you are not doing Continuous Integration, and that is a prerequisite for Continuous Delivery. I don’t do this; no one on my team does it; no one is likely to do it any time soon, except by an accident of fortuitous timing. Nevertheless, his book is one of the most useful books on software development I have read. We have used it as a playbook for improvement, with the result being highly effective software delivery. Our experience is one small example of an interesting historical process, which I’d like to sketch in somewhat theoretical terms. Software is a psychologically intimate technology. Much as described by Gilbert Simondon’s work on technical objects, building software has evolved from a distractingly abstract high modernist endeavour to something more dynamic, concrete and useful.

The term software was co-opted, in a computing context, around 1953, and had time to grow only into an awkward adolescence before being declared, fifteen years later, as in crisis. Barely had we become aware of software’s existence before we found it to be a frightening, unmanageable thing, upsetting our expected future of faster rockets and neatly ordered suburbs. Many have noted that informational artifacts have storage and manufacturing costs approaching zero. I remember David Carrington, in one of my first university classes on computing, noting that as a consequence of this, software maintenance is fundamentally a misnomer. What we speak of as maintenance in physical artifacts, the replacement of entropy-afflicted parts with equivalents within the same design, is a nonsense in software. The closest analogue might be system administrative activities like bouncing processes and archiving logfiles. What we call (or once called) maintenance is revising our understanding of the problem space.

Software has an elastic fragility to it, pliable, yet prone to the discontinuities and unsympathetic precision of logic. Lacking an intuition of digital inertia, we want to change programs about as often as we change our minds, and get frustrated when we cannot. In their book Continuous Delivery, Humble and Farley say we can change programs like that, or at least be changing live software very many times a day, such that software development is not the bottleneck in product development.

With this approach, we see a rotation and miniaturisation of mid-twentieth century models of software development. The waterfall is turned on its side.

Continue reading