Information Relativity

Key Takeaways

1. Software development is driven by learning experiences that generate knowledge
2. Knowledge is contextual to one’s perspective
3. Perspective is impacted by the relativity of information
4. Relativity means observations are inconsistent across locations
5. To build great software, we need to account for relativity in our systems of work

An important part of software development is that it is driven by experiences—we learn about our systems through hands-on work—spiking and prototyping, experimenting and testing.

We try to have rich experiences using a variety of tools, practices, events, and concepts. We use those experiences to generate ideas and data, confirm or deny assumptions, and solve problems.

This is our process of creation: write something, build something, see if it works, evaluate. The code we write, the way we write it, the tests we use, the packages we import, the systems that we deploy it onto—we are going to bring all these all together through our creativity, using experiments to learn our way into an overall structure and architecture. 

So we cut some cards on the Kanban, get a few people together, pull some work. Soon enough we have something, we demo it to ourselves, we demo it to a customer, and then hey-whaddya-know! someone is actually getting value from our creation.

Now I ask you—at what point did the software exist?

When did we move from nothing to something?

—

We all come to the table with a different view of the world. We all understand what it means to “exist” differently. For one person it’s when the wireframes are built. For another, it’s not until the first tests are passing. And for the finance team? Way back when the budget got approved. You get the idea.

We can call this “perspective”—things have different meaning depending on where you observe them from.

It is interesting to think that something as fundamental as ‘existence’ can differ depending on what “hat” you are wearing. Compared to objects in the real world, this is an awkward thing to wrap your head around. The furniture in my house exists whether I’m an engineering manager working from my living room or a traveling salesman trying to find the shortest path between cities.

But for “processes,” this is actually quite normal (which is what we have been describing so far —a process). They are abstract n-dimensional graphs, materially different depending on who is examining them, and in ways that one might not expect if one is not used to working with abstractions.

This sounds cool, but it is not novel, because it is true about other processes in business. If I am building a skyscraper, I work with many different disciplines to do the construction, and all of them will look at different criteria to assess their overall state of existence. (Though, again, this is much less obvious because the focus of the work is on real objects and not abstractions).

Where perspective gets really interesting is when you bring in the concept of relativity.

Relativity

Relativity was introduced to the world at the beginning of the last century when Einstein proved that reality is fundamentally different depending on your frame of reference. A distortion of the spacetime continuum occurs that is only noticeable at very large and very small scales. The concept has led to the discovery of black holes, gravitational lenses, time dilation, and all kinds of other fantastic things.

Relativity is not at all what one would expect based on our regular day-to-day lives that operate according to classic laws of physics. It changes what it means to observe and to be an observer—it means that how we experience the world differs not just in how we interpret it. There are circumstances where the world I experience is inconsistent with yours.

It turns out that communication has these same characteristics and works in this same peculiar way. Information is distorted depending on the location of the observer.

Mark Burgess calls this “information relativity”: messages can take multiple paths and interfere with one another, information can be reversed in its order as it travels along one path, the speed of communication can be different from the speed of communication on another path. 

All this extends the problem of communication beyond the classic problems of how meaning differs, and into issues of consistency.

We are well aware of consistency issues in building distributed systems. We have the CAP theorem to describe some of the trade-offs we need to make against availability and partitioning. We have the idea of eventual consistency, which is now a household word (in my house). This is a tremendous problem to solve on a day-to-day basis in the dynamics of a distributed system, but it also plays an important role in our organizational dynamics, too.

Unlike building a skyscraper, in a software system, we are only ever working with representations, never “real things.” Our ability to obtain representations depends on the fidelity of our information networks—systems of people, tools, processes, and artifacts. The impact of information relativity on their dynamics is a key impediment to value that we all need to think about when we are building systems. 

Take the problem of calculating capacity for a service across a multitude of microservice owners. Perhaps they are working together to build a capacity plan for customer forecasts. Depending on which information sources they access, the way they access them, the queries they run, and even the time of day they run them, each group is liable to arrive at different results. This is true whether we are talking about the infrastructure or the sales data. Arriving at a consistent representation of capacity becomes difficult and time consuming. 

Or take the problem of managing an incident, where multiple teams need to work together to determine what is failing and how to fix it, then return later to understand why it failed and what can be done to improve the system. Depending on what team you sit on, you will use different tools to look at a phenomenon on different scales. We observe traces of the same phenomena in different ways, arriving at different understandings and perspectives on causality. Incident analysis is made difficult if we don’t take action to mitigate the relativity problems. 

These are difficult problems, and we have not even started talking about the classic issues of whether we understand our measurements to mean the same thing or whether we are measuring the same things in the same way.

So to build great software, we start with those classic problems of perspective, but we also need to grapple with questions of relativity, which does not operate according to the classic deterministic principles that we all know and love, the principles that operate when you eat a sandwich or bike to work. 

We depend on unreliable materials—stories and data that build mental models about what is happening, and which are only ever in a partially-correct, never-stable state. 

Though our systems are based on 1s and 0s, the answers to our questions are not true or false. Quite the opposite, our knowledge exists somewhere on a spectrum of strong and weak inferences. 

And this is the heart of our synthetic systems. We need to design systems of work that take into account the relativistic nature of our knowledge.

To be successful we need to think in terms of probability, not determinism. The unsuccessful will assume that what they hear (and think, and see) has been determined by the past or determines something in the future. But in our world of information dynamics, determinism is a harmful paradigm to try to work inside.

When designing our systems, we need to take into account this challenge to our ability to learn and generate knowledge using practices that,

1. Assume variability: the impact of relativity is difficult to predict, so we need to focus on strategies that emphasize eventual convergence rather than trying to get things right upfront.
2. Use systems thinking: we won’t factor in relativity when we are only thinking about a subset of the system, or ignoring human factors.
3. Provide fast feedback: we cannot detect and respond to problems of inconsistency if we are not getting feedback on our decisions.

It is interesting to consider that many practices that are emerging from software, like Agile and DevOps, are implicitly designed to deal handle these challenges. They allow us to correct the drift in our mental models and understanding.

But this thinking is not self-evident. The same way Einstein discovered that the world works differently underneath the atom, we are still discovering the way things work differently in the world of information. What are the black holes and gravitational lenses in the physics of synthetic systems? What is still waiting to be discovered in the world of software development? History says, a lot more.

This article was originally published on InfoQ on April 12, 2021

Images are clipped from the Wolfram Physics Project