100 years of whatever this will be

What if all these weird tech trends actually add up to something?

Last time, we explored why various bits of trendy technology are, in my opinion, simply never going to be able to achieve their goals. But we ended on a hopeful(?) note: maybe that doesn't matter. Maybe the fact that people really, really, really want it, is enough.

Since writing that, I've been thinking about it more.

I think we are all gradually becoming more aware of patterns, of major things wrong with our society. They echo some patterns we've been seeing for decades now. The patterns go far beyond tech, extending into economics and politics and culture. There's a growing feeling of malaise many of us feel:

• Rich, powerful, greedy people and corporations just get richer, more powerful, and more greedy.
• Everyone seems to increasingly be in it for themselves, not for society.
• Or, people who are in it for society tend to lose or to get screwed until they give up.
• Artists really don't get enough of a reward for all the benefit they provide.
• Big banks and big governments really do nonspecifically just suck a lot.
• The gap between the haves and have-nots keeps widening.
• You can't hope to run an Internet service unless you pay out a fraction to one of the Big Cloud Providers, just like you couldn't run software without paying IBM and then Microsoft, back in those days.
• Bloody egress fees, man. What a racket.
• Your phone can run mapreduce jobs 10x-100x faster than your timeshared cloud instance that costs more. Plus it has a GPU.
• One SSD in a Macbook is ~1000x faster than the default disk in an EC2 instance.
• Software stacks, governments, and financial systems: they all keep getting more and more bloated and complex while somehow delivering less per dollar, gigahertz, gigabyte, or watt.
• Computers are so hard to run now, that we are supposed to give up and pay a subscription to someone - well, actually to every software microvendor - to do it for us.
• We even pay 30% margins to App Stores mainly so they can not let us download apps that are "too dangerous."
• IT security has become literally impossible: if you install all the patches, you get SolarWinds-style supply chain malware delivered to you automatically. If you don't install the patches, well, that's worse. Either way, enjoy your ransomware.
• Software intercompatibility is trending toward zero. Text chat apps are literally the easiest thing in the world to imagine making compatible - they just send very short strings, very rarely, to very small networks of people! But I use at least 7 separate ones because every vendor wants their own stupid castle and won't share. Don't even get me started about books or video.
• The most reasonable daycare and public transit in the Bay Area is available only with your Big Tech Employee ID card.
• Everything about modern business is designed to funnel money, faster and faster, to a few people who have demonstrated they can be productive. This totally works, up to a point. But we've now reached the extreme corner cases of capitalism. Winning money is surely a motivator, but that motivation goes down the more you have. Eventually it simply stops mattering at all. Capitalism has become a "success disaster."

Writing all this down, you know what? I'm kind of mad about it too. Not so mad that I'll go chasing obviously-ill-fated scurrilous rainbow financial instruments. But there's something here that needs solving. If I'm not solving it, or part of it, or at least trying, then I'm... wasting my time. Who cares about money? This is a systemic train wreck, well underway.

We have, in Western society, managed to simultaneously botch the dreams of democracy, capitalism, social coherence, and techno-utopianism, all at once. It's embarrassing actually. I am embarrassed. You should be embarrassed.

"Decentralization"

I'm a networking person and a systems person, so please forgive me if I talk about all this through my favourite lens. Societies, governments, economies, social networks, and scalable computing all have something in common: they are all distributed systems.

And.

And everyone.

Everyone seems to have an increasingly horrifically misguided idea of how distributed systems work.

There is of course the most obvious horrifically misguided recently-popular "decentralized" system, whose name shall not be spoken in this essay. Instead let's back up to something older and better understood: markets. The fundamental mechanism of the capitalist model.

Markets are great! They work! Centrally planning a whole society clearly does not work (demonstrated, bloodily, several times). Centrally planning corporations seems to work, up to a certain size. Connecting those corporations together using markets is the most efficient option we've found so far.

But there's a catch. People like to use the term free market to describe the optimal market system, but that's pretty lousy terminology. The truth is, functioning markets are not "free" at all. They are regulated. Unregulated markets rapidly devolve into monopolies, oligopolies, monopsonies, and, if things get really bad, libertarianism. Once you arrive there, every thread ends up with people posting about "a monopoly on the use of force" and "paying taxes at gunpoint" and "I'll run my own fire department" and things that "end at the tip of the other person's nose," and all useful discourse terminates forevermore.

The job of market regulation - fundamentally a restriction on your freedom - is to prevent all that bad stuff. Markets work well as long as they're in, as we call it in engineering, the "continuous control region," that is, the part far away from any weird outliers. You need no participant in the market to have too much power. You need downside protection (bankruptcy, social safety net, insurance). You need fair enforcement of contracts (which is different from literal enforcement of contracts).

And yet: markets are distributed systems.

Even though there are, in fact, very strict regulators and regulations, I can still enter into a contract with you without ever telling anyone. I can buy something from you, in cash, and nobody needs to know. (Tax authorities merely want to know, and anyway, notifying them is asynchronous and lossy.) Prices are set through peer-to-peer negotiation and supply and demand, almost automatically, through what some call an "invisible hand." It's really neat.

As long as we're in the continuous control region.

As long as the regulators are doing their job.

Here's what everyone peddling the new trendy systems is so desperately trying to forget, that makes all of them absurdly expensive and destined to fail, even if the things we want from them are beautiful and desirable and well worth working on. Here is the very bad news:

Regulation is a centralized function.

The job of regulation is to stop distributed systems from going awry.

Because distributed systems always go awry.

If you design a distributed control system to stop a distributed system from going awry, it might even work. It'll be unnecessarily expensive and complex, but it might work... until the control system itself, inevitably, goes awry.

I find myself linking to this article way too much lately, but here it is again: The Tyranny of Structurelessness by Jo Freeman. You should read it. The summary is that in any system, if you don't have an explicit hierarchy, then you have an implicit one.

Despite my ongoing best efforts, I have never seen any exception to this rule.

Even the fanciest pantsed distributed databases, with all the Rafts and Paxoses and red/greens and active/passives and Byzantine generals and dining philosophers and CAP theorems, are subject to this. You can do a bunch of math to absolutely prove beyond a shadow of a doubt that your database is completely distributed and has no single points of failure. There are papers that do this. You can do it too. Go ahead. I'll wait.

<several PhDs later>

Okay, great. Now skip paying your AWS bill for a few months.

Whoops, there's a hierarchy after all!

You can stay in denial, or you can get serious.

Western society, economics, capitalism, finance, government, the tech sector, the cloud. They are all distributed systems already. They are all in severe distress. Things are going very bad very quickly. It will get worse. Major rework is needed. We all feel it.

We are not doing the rework.

We are chasing rainbows.

We don't need deregulation. We need better designed regulation.

The major rework we need isn't some math theory, some kind of Paxos for Capitalism, or Paxos for Government. The sad, boring fact is that no fundamental advances in math or computer science are needed to solve these problems.

All we need is to build distributed systems that work. That means decentralized bulk activity, hierarchical regulation.

As a society, we are so much richer, so much luckier, than we have ever been.

It's all so much easier, and harder, than they've been telling you.

Let's build what we already know is right.

The Gift of It's Your Problem Now

Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also needed to be updated in a hurry, but won't be until they facilitate some ransomware, which is pretty normal for these situations.)

I have a 20+ year history of poking fun at Java in this space, and it pains me to stop now. But the truth is: this could have happened to anyone.

What happened was:

• Someone wrote a library they thought was neat
• They decided to share it with the world for free
• Millions of people liked it and used it everywhere
• Some contributors contributed some good ideas and, in this case, at least one bad idea
• Out of a sense of stewardship, they decided to maintain backward compatibility with the bad idea
• The bad idea turned out to have one or more security flaws that affected all the users
• The stewards responded quickly with a fix

From this, if you believe the Internet Consensus, we can conclude that open source doesn't work, people don't get paid enough, capitalism is a sham, billionaires are built on the backs of the proletariat, your $50 Patreon donation makes a real difference, and Blorkchain Would Have Solved This.

(Miraculously the Internet Consensus is always the same both before and after these kinds of events. In engineering we call this a "non-causal system" because the outputs are produced before the inputs.)

Nevertheless, I have been dared to take my take on the issue. It, too, was the same before and after, but the difference is I didn't write it down until now, which makes this journal a causal system. You could probably write an interesting philosophical paper about observations of a non-causal system nevertheless being causal, but mercifully, I will not.

Free Software is Communism

So anyway, meandering gently toward the point, let's go back in time to the original Free Software movement. Long ago, before the average reader of this sentence was born, a person whose name is now unpopular was at a university, where they had a printer, and the printer firmware was buggy. This person firmly believed they could quickly fix the printer firmware if only they had the source code. (In the spirit of every "I could do this better in a weekend" story, I'm not sure whether we ever confirmed if this was true. In any case printer firmware is still buggy.)

As a result, they started a nonprofit organization to rewrite all of Unix, which the printer did not run and which therefore would not solve any of the original problem, but was a pretty cool project nonetheless and was much more fun than the original problem, and the rest was history.

This story archetype is the Hero's Journey that inspires all software development:

• I have a problem
• I do not know how to solve that problem
• But I strongly believe, without evidence, that I can solve a generalized version of that problem if I form a large enough team and work at it for 35 years(*)
• We are now accepting donations

(*) initial estimate is often less than 35 years

Now, you have probably heard all this before, and if you're a software developer you have probably lived it. This part is not really in question. The burning question for us today, as we enjoy the (hopefully) peak of late-stage capitalism, is: ...but where will the donations come from?

Authoritarianism

Before we get back onto communism, let me draw an important distinction. Most communist governments in history ended up being authoritarian systems, which is to say, top-down control. Ironically, the people at the top seem to have more power than the people at the bottom, which at first seems like the antithesis of communism. This is not the place to claim an understanding of why that always seems to happen. But one has to acknowledge a pattern when one sees it.

On the other hand, it's easy to find examples of authoritarianism outside communism. Our world is filled with top-down control systems. Many corporations are in many ways, top-down controlled. The US system of government is increasingly top-down controlled (ie. authoritarian), despite the many safety measures introduced early to try to prevent that.

When politicians rail against communism it is because they don't want you to notice the ever-growing non-communist authoritarianism.

Authoritarianism is self-reinforcing. Once some people or groups start having more power, they tend to use that power to adjust or capture the rules of the system so they can accumulate more power, and so on. Sometimes this is peacefully reversible, and sometimes it eventually leads to uprisings and revolutions.

People like to write about facism and communism as if they are opposite ends of some spectrum, but that's not really true in the most important sense. Fascism blatantly, and communism accidentally but consistently, leads to authoritarianism. And authoritarianism is the problem.

Authoritarianism is about taking things from me. Communism, in its noncorporeal theoretical form, is about giving things away.

I read a book once which argued that the problem with modern political discourse is it pits the "I don't want things taken from me" (liberty!) people against the "XYZ is a human right" (entitlement!) people. And that a better way to frame the cultural argument is "XYZ is my responsibility to society."

As a simple example, "Internet access is a human right," is just a sneaky way of saying "someone should give people free Internet." Who is someone? It's left unspecified, which is skipping over the entire mechanism by which we deliver the Internet. It's much more revealing to write, "To live in a healthy society, it's our responsibility to make sure every person has Internet access." Suddenly, oh, crap. The someone is me!

Healthy society is created through constant effort, by all of us, as a gift to our fellow members. It's not extracted from us as a mandatory payment to our overlords who will do all the work.

If there's one thing we know for sure about overlords, it's that they never do all the work.

Free software is a gift.

I would like to inquire about the return policy

Here's the thing about gifts: the sender chooses them, not the recipient. We can have norms around what gifts are appropriate, and agreements to not over-spend, and wishlists, and so on. But I won't always get the exact gift I want. Sometimes I didn't even want a gift. Sometimes the gift interprets JNDI strings in my log messages and executes random code from my LDAP server. This is the nature of gifts.

On the other hand, the best gifts are the things I never would have bought for myself, because they seemed too expensive or I didn't even realize I would like them or they were too much work to obtain, or because someone hand-made them just for me. These feel like luxuries of the sort capitalism cannot produce, because deciding, going out, and buying something for myself isn't luxury, it's everyday. It's lonely. It's a negotiation. It's limited by my own lack of creativity.

The best part of free software is it sometimes produces stuff you never would have been willing to pay to develop (Linux), and sometimes at quality levels too high to be rational for the market to provide (sqlite).

The worst part of free software is you get what you get, and the developers don't have to listen to you. (And as a developer, the gift recipients aren't always so grateful either.)

Paying for gifts

...does not work.

You don't say to someone, "here's $100, maybe this time get me a gift worth $100 more than you'd regularly spend." It's kind of insulting. It still probably won't get you exactly the thing you wanted. Actually, the other person might just pocket the $100 and run off with it.

We already have a way for you to spend $100 to get the thing you want. It's a market. A market works fine for that. It's not very inspiring, but most of the time it's quite efficient. Even gift-givers will often buy things on the same market, but with a different selection criteria, thus adding value of their own.

When you try to pay for gifts, it turns the whole gift process into a transaction. It stops being a gift. It becomes an inefficient, misdesigned, awkward market.

There's research showing that, for example, financial compensation in a job is more likely a demotivator than a motivator (ie. if you pay me too little, I'll work less hard or quit, but if you double my pay, it won't double my output). If you tie cash compensation to specific metrics, people will game the metrics and usually do an overall worse job. If you pay someone for doing you a favour, they are less likely to repeat the favour. Gifts are inherently socially and emotionally meaningful. Ruin the giftiness, and you ruin the intangible rewards.

So it is with free software. You literally cannot pay for it. If you do, it becomes something else.

This is why we have things like the Linux Foundation, where the idea is you can give a gift because you appreciate and want to support Linux (and ideally you are a rich megacorporation so your gift is very big), but it dilutes the influence of that money through an organization that supposedly will not try to influence the gift of Linux that was already happening. You end up with multiple gift flows in different directions. Money goes here, code goes there. They are interdependent - maybe if one flow slows down the other flow will also slow down - but not directly tied. It's a delicate balance. People who keep receiving Christmas gifts but never give any might eventually stop receiving them. But might not.

Anyway, gifts will not get you 24-hour guaranteed response times to security incidents.

Gifts won't get you guaranteed high quality code reviews.

Gifts will not, for heaven's sake, prevent developers from implementing bad ideas occasionally that turn into security holes. Nothing will. Have you met developers?

Open source

I've avoided the term "open source" so far because it means something different from the original idea of Free Software.

Open source was, as I understand it, coined to explain what happened when Netscape originally opened their Mozilla source code, back at the end of the 1990s. That was not a gift. That was a transaction. Or at least, it was intended to be.

The promise of open source was:

• You, the company, can still mostly control your project
• Customers will still pay you to add new features
• Actually customers might pay other people to add new features, but you can still capitalize on it because you get their code too
• Linux distributions only package open source code so you'll onboard more customers more easily this way
• You can distance yourself from this anti-capitalist gift-giving philosophical stuff that makes investors nervous
• Plus a bunch of people will look at the code and find bugs for you for free!

Maybe this sounds cynical, but capitalists are cynical, and you know what? It worked! Okay, not for Netscape Corporation (sorry), but for a lot of other people since then.

It also failed a lot of people. Many developers and companies have been disappointed to learn that just uploading your code to github doesn't make a community of developers appear. (It does make it more likely that AWS will fork your product and make more money from it than you do.) Code reviews are famously rare even in security-critical projects. Supply chain issues are rampant.

In fact, we've now gotten to the point where some people hesitate to give away their source code, mainly because of this confusion of gifts and customers. If I spend some spare time hacking something together on a weekend and give it away, that's a gift. If you yell at me for making it, that makes giving less fun, and I will spend fewer weekends making gifts.

Whereas when a company has a product and open sources it and you complain, that's customers giving valuable feedback and it's worth money to learn from them and service them, because you eventually earn money in exchange (through whatever business model they've established). No gift necessary.

Call it cynical or call it a win/win relationship. But it's not a gift.

The startup ecosystem

Since the creation of the open source designation 20+ years ago, software startups have taken off more than ever. I attribute this to a combination of factors:

• Cloud computing has made it vastly cheaper to get started
• Incubators like YCombinator have industrialized the process of assembling and running a small software company
• Megacorps have become exponentially richer but no more creative, so they need to acquire or acqui-hire those startups faster and faster in order to grow.

Although a lot of startups open source their code, and they all depend heavily on open source ecosystems, the startup world's motivations are amazingly different from the free software and open source worlds.

Gifts exist in the startup world. They are things like "we were both in YCombinator so I will intro you to this investor I like" or "I got extremely rich so let me invest in your startup and incidentally I get a lottery ticket for becoming even more rich." These absolutely are still gifts. They each strengthen social ties. The startup world is a society, and the society is built up from these gifts. It's a society that largely ignores the trials and tribulations of anyone who isn't a rich software engineer insider, but history has hosted many past societies of that sort and it takes a long time to build and deploy enough guillotines, and anyway they are having fun and producing a lot and surely that counts for something.

If free software gifts are communism and open source is cynically capitalist exploitation, then startups may be, weirdly, the democratization of capitalism.

Hear me out. Big companies don't care what you think; you can't pay them enough to care. Gift givers care only a little what you think; if they gave you what you wanted, it wouldn't be a gift. But startups, well, there are a lot of them and their mantras are "do things that don't scale" and "focus on the customer" and "build rapid feedback loops." What that spells for you is a whole bunch of people who want to give you what you want, in exchange for money, and who are excited to amortize the costs of that over all the other customers who want the same thing.

It's kind of exciting, conceptually, and more self-optimizing than untuned gift giving, and so it's not super surprising to me that it has started to eclipse the earlier concepts of free software and open source. More and more "open" projects are backed by small companies, who have financial incentives to make their users happy because some of the users turn into paying customers. They'll even provide the uptime SLAs and security fix turnaround guarantees you wanted so much. Our company, Tailscale, is unabashedly one of those. Nothing to be ashamed of there. The system works.

What doesn't work is assuming those startup mechanics apply to everyone out there who gives you a software gift. Not every project on github is the same.

Not everyone has the same motivations.

Giving them money won't change their motivations.

Trying to pay them or regulate them taints the gift.

If you wanted to pay someone to fix some software, you didn't want a gift. You wanted a company.

But if there is no company and someone gave you something anyway? Say thanks.

Epilogue

This isn't where evolution stops. There's a lot more to say about how SaaS taints the unwritten agreement of open source (because you don't have to give back your changes to the code), and how startups tend to go bankrupt and their tech dies with them, and how the best developers are not good at starting companies (no matter how much easier it has become), and how acquiring a startup usually destroys all the stuff they innovated, and how open source is often used as a way to exfiltrate past those kinds of disasters, and how simultaneously, whole promising branches of the "gift economy" structure have never been explored. But that's enough for today. Maybe another time.