Societal Interstructures and Bigness of Thought

Our company has been working lately at partnering with several different other "ISV" companies who want to use our product as a platform for their product. After being basically screwed around by one of the larger of those ISVs in the last few days (nothing really serious; just super annoying), I came to this conclusion:

Trust people in your company, but don't expect to trust people in other companies.

And therein lies a very interesting lesson, which I will attempt to relate back to software via a mild digression in the opposite direction.

One of my pet theories of capitalism (or at least, the non-insane variant practiced in Canada) is that, unlike idealistic theories like communism or libertarianism, capitalism tries to combine the best of both worlds:

• It is impossible to centrally control an immensely complex system, like an entire country's economy, so we don't try. We implement a complex, mostly self-organizing system instead, by carefully controlling the rules of the game.

• Simpler systems, like small groups of people, are much more efficient when organized as cooperative, not competitive, groups with a centrally organized set of goals. That's why employees of any particular successful company aren't generally set in cutthroat competition with one another.

Why does it work? Because up to a certain size, a single mind can hold and optimize the entire structure. I don't mean just the person at the top of the pyramid; I mean that, in a proper organization, anyone can see and understand the structure they're working in. That means they can understand other people's goals and how those goals fit in with the big picture. When you can do that, you can resolve your differences of opinion by finding the one "right" non-compromise answer. In other words, you can work efficiently.

In groups that are too large, this is impossible. You really can't understand why the person you disagree with is doing what he's doing; or worse, perhaps he's doing that because his goals are actually different from yours and the system is too complicated to find a non-compromise solution. That's why we have multiple companies competing with each other, and it works better than if they just tried to all get along.

But what exactly is too big?

This is the fun part. The last few decades have seen a huge increase in the number of smaller companies, while simultaneously bigger companies have merged and gotten even bigger (and fewer). This is because of two completely separate effects.

First, big companies get bigger because of technology: with better communication and management technologies, it's possible to centrally organize larger and larger groups. What used to be impossible with pencil-and-abacus accounting systems is now possible with computers. This trend should continue, and big companies will be able to get bigger.

The second effect is weirder. It's also because of technology, but causes exactly the opposite effect. With technology, smaller groups can do more and more complex things. Complex things are difficult to manage centrally.

So which companies get bigger? The ones doing simple, parallelizable things. Kraft mass produces food products in giant vats. Various companies massively drill for oil. GM mass-produces cars. Sony mass-produces electronics.

And IBM and Sun mass-produce and recombine Java objects that go on to mass-produce billable consulting hours. Meanwhile tiny little companies start up, produce and recombine open source tools, and (sometimes profitably) produce complex, not always open source, products in small quantities, and almost never in Java. Aha. And here we are back in the world of software.

Linus's rule for functions: The maximum length of a function is inversely proportional to the complexity and indentation level of that function. Take that to the world of OO, and the maximum size of a class is inversely proportional to its complexity. My claim is that it's also proportional to Bigness of Thought (BoT), that is, the amount of complexity of this type that can be held in a human brain - the particular brains working on your project - at once.

Java encourages teeny tiny objects that do only one tiny thing, hopefully well. That's because many Java programmers are people like financial analysts, who never really wanted to be programmers, and so their BoT for programming concepts is teeny tiny. These people can program in Java. You might need a horde of them to get anything done, but they can get it done. Pretty neat, really.

Some people have a very high programming BoT. I think I fall into that category. The danger for those people is that they can write amazing programs nobody else can understand, which in the end, doesn't do anyone any good, because it's impossible for someone to maintain it after they move on. Worse, even people with very high programming BoT can't understand the programs, because everyone with high BoT has a different mental structure. I'm great at remembering concepts, but can't remember my postal code; other people can remember dozens of interest rates and formulas with no problem, but they need to draw a picture to see any difficult concept. Both groups have a high BoT, but for totally different things. Both groups can even make very good programmers, but for totally different programs.

What's the point of all this? Well, I could write for hours about the results, but my main point is: the BoT of your group defines the maximum implementation complexity of your objects. When the overall project exceeds your group's BoT (which is virtually always), you need to subdivide your objects into sub-objects with understandable interfaces that reduce the BoT needed to build the object that combines them. And, as technology (eg. programming languages and libraries) improves, the amount of organized complexity you can squeeze into the same BoT increases.

It's just like capitalism. If you can't centrally manage it all, you split it into two companies to make it achievable. If you have two companies but you could centrally manage it, you merge the two companies to make it more efficient.

And if your partner company ("library programmer") has some idiots ("bugs") you're going to have to ask them nicely to get your problems solved, because your poor brain doesn't have the capacity to hold all the details of the whole picture all at once.

Interesting Side Notes

Compromise (solving each person's problems poorly, instead of solving them all perfectly at once) should be necessary only when the problem exceeds the BoT of all affected parties.

Concepts near the limit of your BoT are very difficult for you to explain to anyone else unless they have an exceedingly high BoT. People with a low BoT, but who manage to understand a particular concept anyway, will be good at explaining it.

It is possible to reexplain difficult solutions from one BoT domain such that they make sense to people in another BoT domain. For example, some concepts can be explained using diagrams, even if the person who solved the problem didn't need a diagram to do so. This is sort of like translating from French to English; something is bound to get lost, but it's better than nothing.

By extension, UML is about as useful as, say, Babelfish.