Why the best technology usually loses

There is a comfortable intuition about technology: that the best solution ultimately wins. Sooner or later, the market recognizes technical superiority, users migrate, the inferior standard dies. It's a story with a just ending, and it's how most companies reason when deciding where to invest: improve the product and the rest resolves itself.

The problem is that this story poorly describes a considerable part of what actually happens. Inventing a new product or process is not sufficient for economic progress: innovation still needs to be exploited and diffused—through licensing, imitation, or simple adoption. These stages are separate, have their own logics, and fail for reasons that have nothing to do with the technical specifications.

The puzzle is historically significant. One of the most persistent questions in economics is why demonstrably less efficient economies persist—why competitive pressure does not eliminate what works worse. The quick answer, that competition would sweep away inferior solutions, was tested against the historical record and failed. It remains, for many, the standard explanation. And it continues to fail to explain what is observed.

The value that depends on others

The first mechanism appears when the value of a technology to each user depends on how many others already use it. These are called positive network externalities, and they come in two forms. The direct one is obvious: a telephone is only useful if there are other connected phones; a file is only useful if the software on the other end can open it. The indirect one is more subtle and perhaps more powerful: the larger the network, the more complementary products are offered, and the cheaper they are—more programs written for a popular computer, more repair shops for a widespread automobile, more tapes compatible with the dominant video system.

When this is the case, the user's choice ceases to be an evaluation of the product and becomes a bet on the behavior of others. The user must anticipate which technology will be widely used. This creates a coordination problem, worsened by the fact that different users may prefer to coordinate around different technologies.

From this coordination problem come two symmetric failures, both formally described in industrial organization theory. The first is excess inertia: everyone would be better off with the new technology, everyone knows this, and no one moves—because each fears migrating alone and being stranded. The second is excess momentum: the old technology is better, but users rush to the new one for fear of being left with an abandoned standard. Note what the two have in common. In neither case was relative quality the decisive criterion.

There is an important limit here, and it comes from the very literature that describes the phenomenon. Both failures depend on the decision moment being simultaneous. If a user can adopt first and attract others, or react quickly to changes by others, the trap loosens. Excess inertia and momentum become serious problems in two situations: when information and reaction lags are long, and when users have conflicting preferences about the standard. That is: lock-in is not automatic. It has conditions, and it is worth knowing what they are.

Not by accident, the most common practical answer to this is institutional, not technical. Where network externalities exist, standards are usually established by collective decision—by governments or sectoral committees. Light bulbs, electrical outlets, and railway gauges were standardized this way. The choice of which technology everyone will use is often not made by the market, but by a conference room.

The weight of what has already been built

The second mechanism is slower and deeper. Consider two competing technologies, both subject to increasing returns: the more they are used, the better and cheaper they become. Agents learn by doing, refine each one—and yet it's impossible to predict which will prove most efficient in the end. Increasing return rates need not be equal; a later advance, unknown at the start, may tip everything to one side; or simply a small event may give advantage to one. Because increasing returns imply a single winner over time, one technology ends up dominating—and may reveal itself later to be inferior or a dead end compared to the abandoned alternative.

The classical argument identifies four self-reinforcing mechanisms: high fixed installation costs, which make unit cost fall with scale; learning effects, which improve the product as it diffuses; coordination effects, which reward those who do what others do; and adaptive expectations, where current prevalence feeds belief in future prevalence. From these four derive four uncomfortable properties: multiple equilibria—several solutions are possible and the outcome is indeterminate; possible inefficiencies—a technology inherently better than another can lose by chance in gaining adherents; lock-in, because once the solution is reached it is hard to abandon; and path dependency, where small events and fortuitous circumstances determine the path which, once prevailing, becomes binding.

It is worth noting the second of these properties, because it is what dismantles the initial intuition. Within this argument, losing does not require being worse: it requires only arriving late to the adoption of others.

The decisive analytical leap was realizing that the same reasoning applies to institutions—the rules, contracts, and organizations surrounding any technology. Creating institutions has high initial costs. Organizations are born and evolve to take advantage of the opportunities that framework defines, and the skills they acquire do not necessarily result in greater social efficiency. Formal rules generate a web of informal constraints that modify and extend them. The result is that the institutional matrix produces massive increasing returns—and comes to shape the long-term trajectory of economies.

Here a caveat applies that the argument itself makes a point of registering, and which tends to be lost in hasty popularization of the idea. Trajectories do not reverse from small events or isolated errors: specific changes in formal or informal rules can alter history, but generally do not reverse its direction. And the reversals that do occur—from stagnation to growth, or the reverse—tend to pass through changes in the political sphere, not from spontaneous market correction. Path dependency, in this reading, is not fatalism; it is the affirmation that the past narrows the range of future choices, and that escaping a trajectory is costly.

Who selects is not who evaluates

There is a third layer, within organizations. The evolutionary approach to economic change describes firms as bearers of routines—established procedures that govern much of decision-making—and treats the search for new techniques as something analogous to biological mutation: the firm's search policy determines the probability distribution of what it will find, and that policy is itself a routine. In other words, what a firm is capable of discovering depends on what it already does.

The survival criterion, in this framework, is profitability: profitable firms grow, unprofitable ones shrink, and the operational characteristics of the most profitable increasingly account for a growing share of the sector's activity. It looks like natural selection in favor of the better. But there are two analytical problems that the theory's own authors highlight. First: the routines of existing firms determine, at least in part, the environment that selects routines—the market prices that decide who survives depend on the firms already there. Second: to play a role in a future equilibrium, a routine must first survive the present disequilibrium.

This second point has immediate practical consequence. Imagine a fall in demand or an increase in input prices. Firms that would have survived in equilibrium may fail before equilibrium arrives—for example, if they are small and have limited access to credit. The superior technique they carried disappears with them. Saving the selection argument requires assuming efficient capital markets or search processes generous enough to rediscover what was lost—and it is precisely the critical nature of these assumptions that is at issue.

It is worth noting that this is a live controversy, not a consensus. The critique that evolutionary theory directs at orthodoxy is that it leaves open the speed and efficacy of forces pushing the system toward equilibrium, and ignores the formative role of differential firm growth in determining what that equilibrium will be. Those adopting the orthodox view answer differently. The point for the non-academic reader is that the question 'Does the market select the best technique?' has no settled answer.

Losers have an address

Up to here, no villain: coordination, increasing returns, and routines are impersonal mechanisms. The fourth mechanism is different, because it has identifiable beneficiaries and victims.

The classical formulation is Joseph Schumpeter's, and the expression he coined—creative destruction—carries a specific argument, not merely an image: the process by which the new ceaselessly destroys the old is, for him, the essential fact of capitalism, what capitalism consists of and within which every firm must live. From this follows a methodological consequence he made a point of underscoring: a system that at every instant fully utilizes its possibilities can, in the long run, be inferior to one that never does this—because precisely that slack may be the condition for long-run performance.

The decisive economic point is that destruction has owners. New sectors attract resources from old ones; new firms take business from established ones; new technologies make existing machines and skills obsolete. There is a structural reason why radical innovation comes from outside: an already-monopolistic firm has less incentive to innovate than an entrant, because innovation would replace its own existing profits, while the entrant has no profit to replace—it would replace the incumbent. From this it follows that entrants typically have stronger incentives than incumbents to invest in research and development of processes.

The costs on the other side are also concrete. Creative destruction can displace employed workers, who go through unemployment before finding new jobs, and it destroys firm-specific human capital—which in turn can make workers and firms less willing to make investments that only pay off within a lasting relationship. These are not anecdotal side effects. They are the reason resistance to diffusion is rational for those resisting.

When the obstacle is political

There is a historical reading that radicalizes this point: that fear of creative destruction is the main reason there was no sustained increase in living standards between the Neolithic and Industrial revolutions. The argument distinguishes two types of losers. Workers displaced by a machine resist, but their resistance is usually circumventable. Elites are a more formidable barrier—especially when what is threatened is political power, not merely income.

The case supporting this argument is William Lee's stocking frame, whose patent was refused by the English crown. According to this reading, what moved the refusal was not concern about the unemployment of manual knitters, but rulers' fear of becoming political losers—that those displaced by the invention would generate instability and threaten their own power. On the eve of the Industrial Revolution, European aristocracies lived on land rents and commercial privileges guaranteed by monopolies; the spread of factories and cities drained resources from land, reduced land rents, raised wages, and eroded privileges. They were the obvious economic losers from industrialization—and, with urbanization and the emergence of new classes, also potential political losers.

The same line holds that in absolutist regimes such as Austria-Hungary, Russia, the Ottoman Empire, and China, rulers not only failed to encourage economic progress: they took explicit measures to block industrial diffusion and the introduction of new technologies. This is an interpretive and disputed thesis, not a neutral fact. But it has an analytical virtue: it transforms 'the technology didn't catch on' into a question with a subject. Who had veto power, and what would they lose if the technology diffused?

The last mile

Now suppose the most favorable scenario possible: no locked-in standard, no elite blocking, no incumbent with veto power. The technology is demonstrably better and available. Is that enough? No.

Development economics has studied precisely this point for decades, and the formulation is direct: the 'build it and they will come' approach may work with wealthy farmers, but it is not so simple with poor farmers. Even after high-yield varieties are available and locally appropriate, several constraints prevent small producers from adopting them. Understanding these constraints is one of the most active research areas in the field.

Beyond financial and economic constraints, there are information and cognition barriers. The producer needs to learn about the technology before adopting it; they usually hear about it through an extension agent, but only become willing to adopt after seeing another person's experience or testing it on a small plot of their own land. Social learning and learning by doing are therefore central themes—and shape the design of extension and training programs.

And there is a perverse feedback loop, perhaps the most interesting finding in this literature. Constraints at the farm level discourage investment in developing new technologies: why would a company invest in creating new varieties knowing that small producers are too constrained to adopt them? This is why international agricultural research is unpopular as a private investment and tends to arrive as development assistance—despite agricultural research being among the most cost-effective ways to reduce rural poverty.

The phenomenon is not exclusive to poor agriculture. Even within a single country, firms in the same narrowly defined industry use visibly different technologies. And it is hard to explain why, in a globalized world, some countries fail to import and use technologies that would significantly increase their productivity. Growth economics has advanced much more in modeling innovation than in modeling diffusion—which is revealing, since one might expect the opposite.

Inventing and transforming are different things

The historical case that most clearly separates the two stages is the so-called Great Divergence. The current explanation—that Asia had exhausted itself ecologically while Europe still had room to grow—is contested by a systematic comparison of ecological constraints: the overall pattern is mixed, and key regions of China appeared better positioned than their European counterparts in surprising respects, such as per capita fuel supply. Britain, where industrialization actually began, had few of the underutilized resources that remained in other parts of Europe and did not appear better than the Lower Yangtze Delta in timber supply, soil depletion, and other critical ecological measures.

The conclusion of this reading is harsh for narratives of technical merit: if we accept that population growth and its ecological effects made China 'fall,' we would have to say that Europe's internal processes had brought it very close to the same cliff—not to the brink of takeoff—before it was rescued by a combination of overseas resources and an English rupture in the use of underground energy stocks, a rupture partly conditioned by geographical luck. This is a disputed historiographical thesis, and it is worth reading it as such. But it well supports the economic point: inventive capacity and economic transformation are separate stages, linked by material and institutional conditions.

The questions that predict better

None of these mechanisms alone explains the fate of a technology, and they do not form a single causal chain. They can coexist, reinforce each other, or not even be present in a concrete case—network externalities are irrelevant where there is no complementarity, and path dependency is not destiny. What the gathered evidence permits us to say is not that technical quality is irrelevant: it is that it is only one of the variables, and rarely the one that varies most between cases of success and failure.

A possible reading from this is that whoever asks only 'Is this technology better?' is answering the wrong question. The set of questions with greater predictive power appears to be different:

• Who is already locked into the current standard, and does the value of the technology for each user depend on how many others adopt it?

• How much of the institutional framework—rules, contracts, organizations—would have to be remade to accommodate it?

• Does it fit within the routines of those who should adopt it, and does it survive the environment that selects, which is not the same as the one that evaluates technical performance?

• Who loses income from diffusion, and does that someone have economic, regulatory, or political veto power?

• Who bears the cost of the first step—credit, risk, learning, complementary inputs—and can afford it?

None of them appears on a technical spec sheet. All of them appear in the historical record and in economic theory. It is an uncomfortable list for engineers and a comfortable one for strategists—and perhaps the most useful conclusion is that optimizing the product while ignoring these five questions is often the most expensive way to build something excellent that nobody will use.