Why Large Organizations Struggle With Disruption, and What to Do About It

Posted on July 30, 2024 by steve blank

Seemingly overnight, disruption has allowed challengers to threaten the dominance of companies and government agencies as many of their existing systems have now been leapfrogged. How an organization reacts to this type of disruption determines whether they adapt or die.

I’ve been working with a large organization whose very existence is being challenged by an onslaught of technology (AI, autonomy, quantum, cyberattacks, access to space, et al) from aggressive competitors, both existing and new. These competitors are deploying these new technologies to challenge the expensive (and until now incredibly effective) legacy systems that this organization has built for decades. (And they are doing it at speed that looks like a blur to this organization.) But the organization is also challenged by the inaction of its own leaders, who cannot let go of the expensive systems and suppliers they built over decades. It’s a textbook case of the Innovators Dilemma.

In the commercial world creative destruction happens all the time. You get good, you get complacent, and eventually you get punched in the face. The same holds true for Government organizations, albeit with more serious consequences.

This organization’s fate is not yet sealed. Inside it, I’ve watched incredibly innovative groups create autonomous systems and software platforms that rival anything a startup is doing. They’ve found champions in the field organizations, and they’ve run experiments with them. They’ve provided evidence that their organization could adapt to the changing competitive environment and even regain the lead. Simultaneously, they’ve worked with outside organizations to complement and accelerate their internal offerings. They’re on the cusp of a potential transformation – but leadership hesitates to make substantive changes.

The “Do Nothing” Feedback Loop
I’ve seen this play out time and again in commercial and government organizations. There’s nothing more frustrating for innovators than to watch their organization being disrupted while its senior leaders hesitate to take more than token actions. On the other hand, no one who leads a large organization wants it to go out of business. So, why is adapting to changed circumstances so hard for existing organizations?

The answer starts at the top. Responding to disruption requires action from senior leadership: e.g. the CEO, board, Secretary, etc. Fearful that a premature pivot can put their legacy business or forces at risk, senior leaders delay deciding – often until it’s too late.

My time with this organization helped me appreciate why adopting and widely deploying something disruptive is difficult and painful in companies and government agencies. Here are the reasons:

Disconnected Innovators – Most leaders of large organizations are not fluent in the new technologies and the disruptive operating concepts/business models they can create. They depend on guidance from their staff and trusted advisors – most of whom have been hired and promoted for their expertise in delivering incremental improvements in existing systems. The innovators in their organization, by contrast, rarely have direct access to senior leaders. Innovators who embrace radically new technologies and concepts that challenge the status quo and dogma are not welcomed, let alone promoted, or funded.

Legacy – The organization I’ve been working with, like many others, has decades of investment in existing concepts, systems, platforms, R&D labs, training, and a known set of external contractors. Building and sustaining their existing platforms and systems has left little money for creating and deploying new ones at the same scale (problems that new entrants/adversaries may not have.) Advocating that one or more of their platforms or systems are at risk or may no longer be effective is considered heresy and likely the end of a career.

The “Frozen Middle” – A common refrain I hear from innovators in large organizations is that too many people are resistant to change (“they just don’t get it”.) After seeing this behavior for decades, I’ve learned that the frozen middle occurs because of what’s called the“Semmelweis effect” – the unconscious tendency of people to stick to preexisting beliefs and reject new ideas that contradict them – because it undermines their established norms and/or beliefs. (They really don’t get it.) This group is most comfortable sticking with existing process and procedures and hires and promotes people who execute the status quo. This works well when the system can continue to succeed with incremental growth, but in the face of more radical change, this normal human reaction shuts out new learning and limits an organizations’ ability to rapidly adapt to new circumstances. The result is organizational blinders and frustrated innovators. And you end up with world-class people and organizations for a world that no longer exists.

Not everyone is affected by the Semmelweis effect. It’s often mid-grade managers / officers in this same “middle” who come up with disruptive solutions and concepts. However, unless they have senior champions (VP’s, Generals / Admirals) and are part of an organization with a mission to solve operational problems, these solutions die. These innovators lack alternate places where the culture encourages and funds experimentation and non-consensus ideas. Ironically, organizations tend to chase these employees out because they don’t conform, or if forced to conform, they grow disillusioned and leave for more innovative work in industry.

Hubris is managerial behavior of overconfidence and complacency. Unlike the unconscious Semmelweis effect, this is an active and conscious denial of facts. It occurs as some leaders/managers believe change threatens their jobs as decision-makers or that new programs, vendors or ideas increase the risk of failure, which may hurt their image and professional or promotional standing.

In the organization I’ve been working with, the internal engineering group offers senior leaders reassurances that they are responding to disruption by touting incremental upgrades to their existing platforms and systems.

Meanwhile because their budget is a zero-sum game, they starve innovators of funds and organizational support for deployment of disruptive new concepts at scale. The result is “innovation theater.” In the commercial world this behavior results in innovation demos but no shipping products and a company on the path to irrelevance or bankruptcy. In the military it’s demos but no funding for deployments at scale.

Fear of Failure/Risk Aversion – Large organizations are built around repeatable and scalable processes that are designed to be “fail safe.” Here new initiatives need to match existing budgeting, legal, HR and acquisition, processes and procedures. However, disruptive projects can only succeed in organizations that have a “safe-to-fail” culture. This is where learning and discovery happens via incremental and iterative experimentation with a portfolio of new ideas and failure is considered part of the process. “Fail safe” versus “safe-to-fail” organizations need to be separate and require different culture, different people, different development processes and risk tolerance.

Activist Investors Kill Transformation in Commercial Companies
A limit on transformation speed unique to commercial organizations is the fear of “Activist Investors.”  “Activist investors” push public companies to optimize short-term profit, by avoiding or limiting major investments in new opportunities and technology. When these investors gain control of a company, innovation investments are reduced, staff is cut, factories and R&D centers closed, and profitable parts of the company and other valuable assets sold.

Unique Barriers for Government Organizations
Government organizations face additional constraints that make them even slower to respond to change than large companies.

To start, leaders of the largest government organizations are often political appointees. Many have decades of relevant experience, but others are acting way above their experience level. This kind of mismatch tends to happen more frequently in government than in private industry.

Leaders’ tenures are too short – All but a few political appointees last only as long as their president in the White House, while leaders of programs and commands in the military services often serve 2- or 3-year tours. This is way too short to deeply understand and effectively execute organizational change. Because most government organizations lack a culture of formal innovation doctrine or playbook – a body of knowledge that establishes a common frame of reference and common professional language – institutional learning tends to be ephemeral rather than enduring. Little of the knowledge, practices, shared beliefs, theory, tactics, tools, procedures, language, and resources that the organization built under the last leader gets forwarded. Instead each new leader relearns and imposes their own plans and policies.

Getting Along Gets Rewarded – Career promotion in all services is primarily driven by “getting along” with the status quo. This leads to things like not cancelling a failing program, not looking for new suppliers who might be cheaper/ better/ more responsive, pursuing existing force design and operating concepts even when all available evidence suggests they’re no longer viable, selecting existing primes/contractors, or not pointing out that a major platform or weapon is no longer effective. The incentives are to not take risks. Doing so is likely the end of a career. Few get promoted for those behaviors. This discourages non-consensus thinking. Yet disruption requires risk.

Revolving doors – Senior leaders leave government service and go to work for the very companies whose programs they managed, and who they had purchased systems from (often Prime contractors). The result is that few who contemplate leaving the service and want a well-paying job with a contractor will hold them to account or suggest an alternate vendor while in the service.

Prime Contractors – are one of our nation’s greatest assets while being our greatest obstacles to disruptive change. In the 20th century platforms/weapons were mostly hardware with software components. In the 21st century, platforms/weapons are increasingly software with hardware added. Most primes still use Waterfall development with distinct planning, design, development, and testing phases rather than Agile (iterative and incremental development with daily software releases). The result is that primes have a demonstrated inability to deliver complex systems on time. (Moving primes to software upgradable systems/or cloud-based breaks their financial model.)

As well, prime contractors typically have a “lock” on existing government contracts. That’s because it’s less risky for acquisition officials to choose them for follow-on work– and primes have decades of experience in working through the byzantine and complex government purchasing process; and they have tons of people and money to influence all parts of the government acquisition system—from the requirements writers to program managers, to congressional staffers to the members of the Armed Services and Appropriations committees. New entrants have little chance to compete.

Congress – Lawmakers have incentives to support the status quo but few inducements to change it. Congress has a major say in what systems and platforms suppliers get used, with a bias to the status quo. To keep their own jobs, lawmakers shape military appropriations bills to support their constituents’ jobs and to attract donations from the contractors who hire them. (They and their staffers are also keeping the revolving door in mind for their next job.) Many congressional decisions that appear in the National Defense Authorization Act (NDAA) and in appropriations are to support companies that provide the most jobs in their districts and the most funds for their reelection. These come from the Prime contractors.

What to Do About It?
It starts at the top. Confronted with disruptive threats, senior leaders must actively work to understand:

  • The timing of the threat – disruption never comes with a memo, and when it happens its impact is exponential. When will disruption happen that will make our core business or operating concepts/force design obsolete? Will our competitors get there first?
  • The magnitude of the threat – will this put a small part of our business/capabilities at risk or will it affect our entire organization?
  • The impact of the threat – will this have a minor impact or does it threaten the leadership or the very existence of the organization. What happens if our competitors/adversaries adopt this first?
  • The response to the threat- Small experiments, department transformation, and company or organization-wide transformation – and its timeline.

Increase Visibility of Disruptive Tech and Concepts/Add Outside Opinions

  • To counter disruptive threats, the typical reporting relationship of innovators filtered through multiple layers of management must be put aside.
    • Senior leaders need a direct and unfiltered pipeline to their internal innovation groups for monthly updates and demos of evidenced-based experiments in operational settings.
    • And the new operating concepts to go with it.
  • Create a “Red Team” of advisors from outside their organization.
    • This group should update senior leaders on the progress of competitors
    • And offer unbiased assessment of their own internal engineering/R&D progress.
  • Stand up a strategic studies group that can develop new business models/ new strategic concepts usable at the operational level – ensure its connection with external sources of technical innovation
  • Create a “sensing” and “response” organization that takes actual company/agency/service problems out to VC’s and startups and seeing how they would solve them
    • However, unless senior leaders 1) actively make a point of seeing these first hand (at least biannually), and have the mechanism to “respond” with purchase orders/ OTA’s, this effort will have little impact.

Actively and Urgently Gather Evidence

  • Run real-world experiments – simulations, war games, – using disruptive tech and operating concepts (in offense and defense.)
  • See and actively seek out the impact of disruption in adjacent areas e.g. AI’s impact on protein modeling, drones in the battlefield and Black Sea in Ukraine, et al.
  • Ask the pointy end of the organization (e.g the sales force, fleet admirals) if they are willing to take more risk on new capabilities.

These activities need happen in months not years. Possible recommendations from these groups include do nothing, run small experiments, transform a single function or department, or a company or organization-wide transformation.

What Does Organization-wide Transformation look like?

  • What outcome do we desire?
  • When do we need it?
  • What budget, people, capital equipment are needed?
    • What would need to be divested?
  • How to communicate this to all stakeholders and get them aligned?
  • In the face of disruption/ crisis/ wartime advanced R&D groups now need a seat at the table with budgets sufficient for deployment at scale.
  • Finally, encourage more imagination. How can we use partners and other outside resources for technology and capital?

Examples of leaders who transformed their organization in the face of disruption include Microsoft CEO Satya Nadella and Steve Jobs from Apple, in defense, Bill Perry, Harold Brown and Ash Carter. Each dealt with disruption with acceptance, acknowledgment, imagination and action.

Much more to be said about transformation in future posts.

Filed under: Innovation Doctrine, National Security | 6 Comments »

Why The Government is Isn’t a Bigger Version of a Startup

Posted on November 12, 2019 by steve blank

This article previously appeared in War On The Rocks

There was a time when much of U.S. academia was engaged in weapon systems research for the Defense Department and intelligence community. Some of the best and brightest wanted to work for defense contractors or corporate research and development labs. And the best startups spun out of Stanford were building components for weapon systems.

Indeed, Silicon Valley was born as a center for weapon systems development and its software and silicon helped end the Cold War.

During World War II the United States did something its adversaries did not; it enlisted professors and graduate students as civilians in 105 colleges and universities to build advanced weapon systems — nuclear weapons, radar, etc. After World War II, the military-academic relationship that was so effective against Germany and Japan mobilized to face the Soviet threat and almost every research university (Massachusetts Institute of Technology, Stanford, Caltech, Harvard, Columbia, Johns Hopkins, University of Michigan, University of Wisconsin, Cornell, University of Chicago, and many others) continued to engage in weapon systems research during the Cold War.

Unique among them was Stanford, which provost Fred Terman (the father of American electronic warfare and electronic intelligence in World War II)) built as a center of excellence in microwaves and electronics. Rather than focus the university inward on research, Terman took the radical step of encouraging Stanford professors and graduate students to start companies applying engineering to pressing military problems. The companies they started in the 1950s and 60s were based on Stanford’s defense contacts and contracts — microwave components, electronic warfare, and intelligence systems, and then the first wave of semiconductor companies. As there was no venture capital, these early startups were funded by early sales to weapon systems prime contractors and subcontractors.

But this quarter-century relationship between the military and universities ended with a bang in 1969. In the middle of the Vietnam War, student riots protesting military research forced the end of classified work on most college campuses. One of the unintended consequences was that many of the academics went off to found a wave of startups selling their technology to the military. For example, at Stanford after student riots in April 1969 shut down the Applied Electronics Laboratory, James de Broekert ,who was building electronic intelligence satellites, left the university and co-founded three Silicon Valley military intelligence companies: Argo Systems, Signal Science, and Advent Systems.

Within a decade, the rise of venture capital in Silicon Valley enabled startups to find commercial customers rather than military ones. And from then on, innovation in semiconductors, supercomputers, and software would be driven by startups, not the government.

After 9/11, with the memories of the fall of the twin towers, this ecosystem of military, academic, corporate, and startup actors coalesced for the decade as U.S. companies felt a patriotic duty to help their country defeat a common enemy.

But the 2013 Snowden revelations damaged that tenuous relationship yet again. In hindsight the damage wasn’t the result of what the United States was doing, but over the Pentagon’s inability and unwillingness to own up to why it was doing it: After the intelligence failure of 9/11, security agencies overcompensated by widespread, warrantless datamining as well as electronic and telephonic surveillance, including on U.S. persons.

Without a clear explanation of why this had been done, startups, which were already being funded by ever-increasing pools of venture capital, abandoned cooperation with the Defense Department and focused on high returns on social media and commercial applications. The commercial applications of big data, machine learning, artificial intelligence, drones, robotics, cyber, quantum computing, and biotechnology are the core foundations on which the Pentagon needs to build the weapons of the 21st century. Yet the development of these advanced technologies is now being driven by commercial interests, not the Defense Department.

America’s adversaries understand this. China is tightly integrating its defense establishment with startups, companies, and academia in “military-civilian fusion.” Russia, Iran, and North Korea have also fused those activities.

Reconstituting the tightly connected military-academic-commercial ecosystem that the Defense Department once had requires the Pentagon to relearn skills it once had, overcoming decades of avoiding the political and social issues of what it takes to rally the nation against a common threat. Today, every government agency, service branch, and combatant command is adopting innovation activities (hackathons, design thinking classes, innovation workshops, et al.) to tap into the creativity of a new generation of soldier — born into a digital world, comfortable with technology, and willing to improve and enhance America’s ability to fight and win.

The Government Can’t Act Like a Startup
However, those activities are not enough. The government isn’t a bigger version of a startup and can’t act like a startup does. Innovation activities in government agencies most often result in innovation theater. While these activities shape and build culture, they don’t win wars, and rarely deliver shippable or deployable products.

Startups dream in years, plan in months, evaluate in weeks, and ship in days. At times this means startups operate at speeds so fast they appear to be a blur to government agencies. It’s not that these companies are smarter than Defense Department employees, but they operate with different philosophies, different product development methodologies, and with different constraints.

The table below summarizes a few of the salient differences. Some of the most important are the least obvious. Startups can do anything. They can break the law and apologize later (as Uber, Airbnb, and Tesla did), but a government official taking the same type of risks can go to jail.

Urgency and risk-taking in a startup are integral parts of the culture, felt by 100 percent of early-stage employees. The urgency servicemembers feel on the battlefield is felt by few in government agencies, and most often there are negative incentives for risk-taking. In a startup cluster (Silicon Valley, Beijing, Tel Aviv) a failed entrepreneur is known as “experienced.” In a government agency, they’re likely known as being out of a job.

Innovation at speed is a given at a startup but the exception in a government agency. Advances in commercial technologies are occurring at no less than two, and up to ten, times the speed of comparable Pentagon-developed or acquired systems. Some of the speed is simply due to development methodologies. Waterfall development is still used by most defense contractors, resulting in updates of systems measured in years. With Agile development, used by all startups, updates can occur in weeks or sometimes days, or even hours. Some of the speed differences are because commercial companies and academics face Darwinian competitive pressures for revenue or recognition. These force rapid technical advances in fields such as machine learning, artificial intelligence, robotics, big data, and analytics.

The very definition of a contractor implies a contract. And a government contract starts with fixed requirements that only change with contract modifications. That makes sense when the problem and solution are known. But when they are unknown the traditional methods of contracting fail. Startups recognize that when new circumstances arise, they can pivot — make substantive changes to their products without any new contracts.Existing contractors have learned how to master the arcane defense acquisition system and live with the slow decision-making and payment processes. In some agencies, large contractors seem to “own” sections, offices, organizations, or programs. Often former government employees, at the level of GS-15 and below, will leave as staffers and return the next day working for large Beltway contractors, working or managing the same programs they previously worked. This relationship between government agency and contractor further impedes and often rejects innovation or disruption. Officers know they will likely lose their post-retirement future if they seek radical change.

This symbiotic relationship between government agencies and incumbent contractors is also a barrier to new entrants, in particular to startups with the very technologies the Department of Defense now needs. While the Pentagon has made efforts to reform the process (Other Transaction Authorities, TechFAR, mid-tier contracting, accelerators…) there is still a fundamental misunderstanding of what financial incentives would attract the best and brightest investors to guide their companies to work with the Defense Department. There are no incentives for prime contractors to invest in new ventures and none to acquire new ventures. And there is no plan for how to rapidly insert and deploy startup technologies into weapon systems.

So, the question is: What’s next? How do leaders in government think about and organize innovation in a way that makes a difference?

The answer is that, yes, government agencies need to be more agile. And yes, they need to fix the systemic internal issues that hinder their own innovators’ contributions. But, in addition, what they are missing is a comprehensive plan to build a 21st-century defense innovation ecosystem — reintegrating the military, academia, and private enterprise. To harness both their own internal innovators and this new external ecosystem the Defense Department needs what I call an innovation doctrine to organize their efforts to rapidly access and mobilize talent and technology. The Pentagon can build a mindset, culture, and process to fix this. This doctrine would let the country again capture the untapped power and passion of the best and brightest to leapfrog adversaries and win wars.

Filed under: Corporate/Gov't Innovation, Innovation Doctrine | 10 Comments »

Why Companies and Government Do “Innovation Theater” Instead of Actual Innovation

Posted on October 15, 2019 by steve blank

This article previously appeared in the Harvard Business Review.

The type of disruption most companies and government agencies are facing is a once-in-every-few-centuries event. Disruption today is more than just changes in technology, or channel, or competitors – it’s all of them, all at once. And these forces are completely reshaping both commerce and defense.

Today, as large organizations are facing continuous disruption, they’ve recognized that their existing strategy and organizational structures aren’t nimble enough to access and mobilize the innovative talent and technology they need to meet these challenges. These organizations know they need to change but often the result has been a form of organizational Whack-A-Mole– a futile attempt at trying to swat every problem as they pop-up without understanding their root cause.

Ultimately, companies and government agencies need to stop doing this or they will fail.

We can build a mindset, culture and process to fix this – what I call an Innovation Doctrine. But first we need to step back and recognize one of the problems.

I just spent a few days with a large organization with a great history, who like most of their peers is dealing with new and rapidly evolving external threats. However, their biggest obstacle is internal. What had previously been a strength – their great management processes – now holds back their ability to respond to new challenges.

Companies Run on Process
Once upon a time every great organization was a scrappy startup willing to take risks – new ideas, new methods, new customers, targets, and mission. If they were a commercial company, they figured out product/market fit; or if a government organization, it focused on solution/mission fit. Over time as these organizations got large, they built process. By process I mean all the tools that allow companies and government to scale repeatable execution. HR processes, legal processes, financial processes, acquisition and contracting processes, security processes, product development and management processes, and types of organizational forms etc. All of these are great strategies and tools that business schools build, and consulting firms help implement.

Process is great when you live in a world where both the problem and solution are known. Process helps ensure that you can deliver solutions that scale without breaking other parts of the organization.

These processes reduce risk to an overall organization, but each layer of process reduces the ability to be agile and lean and – most importantly – responsive to new opportunities and threats.

Process Versus Product
As companies and government agencies get larger, they start to value the importance of “process” over the “product.” And by product, I mean the creation of new hardware, services, software, tools, operations, tradecraft, etc.  People who manage processes are not the same people as those who create product. Product people are often messy, hate paperwork and prefer to spend their time creating stuff rather than documenting it. Over time as organizations grow, they become risk averse. The process people dominate management, and the product people end up reporting to them.

If the company is large enough it will become a “rent-seeker” and look to the government and regulators as their first line of defense against innovative competition. They’ll use government regulation and lawsuits to keep out new entrants with more innovative business models.

The result of monopolist behavior is that innovation in that sector dies – until technology/consumer behavior passes them by.    By then the company has lost the ability to compete as an innovator.

In government agencies process versus product has gone further. Many agencies outsource product development to private contractors, leaving the government with mostly process people – who write requirements, and oversee acquisition, program management, and contracts.

However, when the government is faced with new adversaries, new threats, or new problems, both the internal process people as well as the external contractors are loath to obsolete their own systems and develop radically new solutions. For the contractors, anything new offers the real risk of losing a lucrative existing stream of revenue. For the process people, the status quo is a known and comfortable space and failure and risk-taking is considered career retarding. Metrics are used to manage process rather than creation of new capabilities, outcomes and speed to deployment. And if the contract and contractor are large enough, they put their thumb on the scale and use the political process and lobbying to maintain the status quo.

The result is that legacy systems live on as an albatross and an impediment to making the country safer and more secure.

Organizational and Innovation Theater
A competitive environment should drive a company/government agency into new forms of organization that can rapidly respond to these new threats. Instead, most organizations look to create even more process. This typically plays out in three ways:

  1. Often the first plan from leadership for innovation is hiring management consultants who bring out their 20th-century playbook. The consultants reorganize the company (surprise!), often from a functional organization into a matrixed organization. The result is organizational theater. The reorg keeps everyone busy for a year, perhaps provides new focus on new regions or targets, but in the end is an inadequate response to the need for rapid innovation for product.
  2. At the same time, companies and government agencies typically adopt innovation activities (hackathons, design thinking classes, innovation workshops, et al.) that result in innovation theater. While these activities shape, and build culture, but they don’t win wars, and they rarely deliver shippable/deployable product.
  3. Finally, companies and government agencies have realized that the processes and metrics they put in place to optimize execution (Procurement, Personnel, Security, Legal, etc.) are obstacles for innovation. Efforts to reform and recast these are well meaning, but without an overall innovation strategy it’s like building sandcastles on the beach. The result is process theater.

For most large organizations these reorgs, activities and reforms don’t increase revenue, profit or market share for companies, nor does it keep our government agencies ahead of our adversaries. One can generously describe them as innovation dead ends.

Between a Rock and A Hard Place
Today, companies and government agencies are not able to access and mobilize the innovative talent and technology they need to meet these challenges. The very processes that made them successful impede them.

Organizational redesign, innovation activities, and process reform need to be part of an overall plan.

In sum, large organizations lack shared beliefs, validated principles, tactics, techniques, procedures, organization, budget, etc. to explain how and where innovation will be applied and its relationship to the rapid delivery of new product.

We can build a mindset, culture and process to fix this.

More in future posts about Innovation Doctrine.

Lessons Learned

  • As companies and agencies get larger, they start to value the importance of process over the “content.”
  • When disruption happens, no process or process manager in the world is going to save your company/or government agency
    • It’s going to take those “product” creators
    • But they have no organization, authority, budget or resources
  • We can build a mindset, culture and process to fix this.
    • A shared set of beliefs and principles of how and where innovation will be used and rapidly delivered
  • Innovation Doctrine

Filed under: Corporate/Gov't Innovation, Customer Development, Harvard Business Review, Innovation Doctrine | 6 Comments »