The Red Queen Problem – Innovation in the DoD and Intelligence Community

“…it takes all the running you can do to keep in the same place. ”
The Red Queen Alice in Wonderland

Innovation, disruption, accelerators, have all become urgent buzzwords in the Department of Defense and Intelligence community. They are a reaction to the “red queen problem” but aren’t actually solving the problem. Here’s why.

In the 20th century our nation faced a single adversary – the Soviet Union. During the Cold War the threat from the Soviets was quantifiable and often predictable. We could specify requirements, budget and acquire weapons based on a known foe. We could design warfighting tactics based on knowing the tactics of our opponent. Our defense department and intelligence community owned proprietary advanced tools and technology. We and our contractors had the best technology domain experts. We could design and manufacture the best systems. We used these tools to keep pace with the Soviet threats and eventually used silicon, semiconductors and stealth to create an offset strategy to leapfrog their military.

That approach doesn’t work anymore. In the 21st century you need a scorecard to keep track of the threats: Russia, China, North Korea, Iran, ISIS in Yemen/Libya/Philippines, Taliban, Al-Qaeda, hackers for hire, etc. Some are strategic peers, some are near peers in specific areas, some are threats as non-state disrupters operating with no rules.

In addition to the proliferation of threats, most of the tools and technologies that were uniquely held by the DoD/IC or only within the reach of large nation states are now commercially available (Cyber, GPS, semiconductors, analytics, centrifuges, drones, genetic engineering, agile and lean methodologies, ubiquitous Internet, crypto and smartphones, etc.). In most industries, manufacturing is no longer a core competence of the U.S.

U.S. agencies that historically owned technology superiority and fielded cutting-edge technologies now find that off-the-shelf solutions may be more advanced than the solutions they are working on, or that adversaries can rapidly create asymmetric responses using these readily available technologies.

The result is that our systems, organizations, headcount and budget – designed for 20th century weapons procurements and warfighting tactics on a predictable basis – can’t scale to meet all these simultaneous and unpredictable challenges. Today, our DoD and national security agencies are running as hard as they can just to stay in place, but our adversaries are continually innovating faster than our traditional systems can respond. They have gotten inside our OODA loop (Observe, Orient, Decide and Act).

We believe that continuous disruption can only be met with a commitment to continuous innovation.

Pete Newell and I have spent a lot of time bringing continuous innovation to government organizations. Newell ran the U.S. Army’s Rapid Equipping Force on the battlefields of Iraq and Afghanistan finding and deploying technology solutions against agile insurgents. He’s spent the last four years in Silicon Valley out of uniform continuing that work. I’ve spent the last six years teaching our country’s scientists how to rapidly turn scientific breakthroughs into deliverable products by creating the curriculum for the National Science Foundation Innovation Corps – now taught in 53 universities. Together Pete, Joe Felter and I created Hacking for Defense, a nationwide program to teach university students how use Lean methodologies to solve defense and national security problems.

The solution to continuous disruption requires new ways to think about, organize, and build and deploy national security people, organizations and solutions.

Here are our thoughts about how to confront the Red Queen trap and adapt a government agency to infuse continuous innovation in its culture and practices.

Problem 1: Regardless of a high-level understanding that business as usual can’t go on, all agencies are given “guidance and metrics (what they are supposed to do (their “mission”) and how they are supposed to measure success). To no one’s surprise the guidance is “business as usual but more of it.” And to fulfill that guidance agencies create structure (divisions, directorates, etc.) designed to execute repeatable processes and procedures to deliver solutions that meet the requirements of the overall guidance.

Inevitably, while all of our defense and national security agencies will tell you that innovation is one of their pillars, innovation actually is an ill-defined and amorphous aspirational goal, while the people, budget and organization continue to flow to execution of mission (as per guidance.)

There is no guidance or acknowledgement that in our national security agencies, even as we execute our current mission, our capabilities decline every year due to security breaches, technology timing out, tradecraft obsolescence, etc. And there is no explicit requirement for creation of new capabilities that give us the advantage.

Solution 1: Extend agency guidance to include the requirements to create a continuous innovation process that a) resupplies the continual attrition of capabilities and b) creates new capabilities that gives us a mission advantage. The result will be agency leadership creating new organizational structures that make innovation a continual process rather than an ad hoc series of heroic efforts.

Problem 2: The word “Innovation” actually describes three very different types of activities.

Solution 2: Use the McKinsey Three Horizons Model to differentiate among the three types. Horizon 1 ideas provide continuous innovation to a company’s existing mission model and core capabilities. Horizon 2 ideas extend a company’s existing mission model and core capabilities to new stakeholders, customers, or targets. Horizon 3 is the creation of new capabilities to take advantage of or respond to disruptive technologies/opportunities or to counter disruption.

We’d add a new category, Horizon 0, which kills ideas that are not viable or feasible (something that Silicon Valley is tremendously efficient at doing).

These Horizons also apply to government agencies and other large organizations. Agencies and commands need to support all three horizons.

Problem 3: Risk equals failure and failure is to be avoided as it indicates a lack of competence.

Solution 3: The three-horizon model allows everyone to understand that failure in a Horizon 1/existing mission activity is different than failure in a Horizon 3 “never been done before” activity. We want to take risks in Horizon 3. If we aren’t failing with some efforts, we aren’t trying hard enough. An innovation process embraces and understands the different types of failure and risk.

Problem 4: Innovators tend to create activities rather than deployable solutions that can be used on the battlefield or by the mission. Accelerators, hubs, cafes, open-sourcing, crowd-souring, maker spaces, Chief Innovation Officers, etc. are all great but they tend to create innovation theater – lots of motion but no action. Great demos are shown and there are lots of coffee cups and posters, but if you look at the deliverables for the mission over a period of years the result is disappointing. Most of the executors and operators have seen little or no value from any of these activities. While the activities individually may produce things of value, they aren’t valued within the communities they serve because they aren’t connected to a complete pipeline that harnesses that value and turns it into a deliverable on the battlefield where it matters.

Solution 4: What we have been missing is an innovation pipeline focused on deployment not demos.

The Lean Innovation process is a self-regulating, evidence-based innovation pipeline. It is a process that operates with speed and urgency, where innovators and stakeholders curate and prioritize their own problems/Challenges/ideas/technology. It is evidence based, data driven, accountable, disciplined, rapid and mission- and deployment-focused.

The process recognizes that Innovation isn’t a single activity (an incubator, a class, etc.) it is a process from start to deployment.
The canonical innovation pipeline:

As you see in the diagram, there are 6 steps to the innovation pipeline: sourcing, challenge/curation, prioritization, solution exploration and hypothesis testing, incubation and integration.

Innovation sourcing: a list of problems/challenges, ideas, and technologies that might be worth investing in. These can come from hackathons, research groups, needs from operators in the field, etc.

Challenge/Curation: innovators get out of their own offices and talk to colleagues and customers with the goal of finding other places in the DoD where a problem or challenge might exist in a slightly different form, to identify related internal projects already in existence, and to find commercially available solutions to problems. It also seeks to identify legal issues, security issues, and support issues.

This process also helps identify who the customers for possible solutions would be, who the internal stakeholders would be, and even what initial minimum viable products might look like.

This phase also includes building initial minimal viable products (MVPs.) Some ideas drop out when the team recognizes that they may be technically, financially, or legally unfeasible or they may discover that other groups have already built a similar product.

Prioritization: Once a list of innovation ideas has been refined by curation, it needs to be prioritized using the McKinsey Three Horizons Model.

Once projects have been classified, the team prioritizes them, starting by asking: is this project worth pursing for another few months full time? This prioritization is not done by a committee of executives but by the innovation teams themselves.

Solution exploration and hypotheses testing: The ideas that pass through the prioritization filter enter an incubation process like Hacking for Defense/I-Corps, the system adopted by all U.S. government federal research agencies to turn ideas into products.

This six- to ten-week process delivers evidence for defensible, data-based decisions. For each idea, the innovation team fills out a mission model canvas. Everything on that canvas is a hypothesis. This not only includes the obvious – is there solution/mission fit? — but the other “gotchas” that innovators always seem to forget. The framework has the team talking not just to potential customers but also with people responsible for legal, support, contracting, policy, and finance. It also requires that they think through compatibility, scalability and deployment long before this gets presented to engineering. There is now another major milestone for the team: to show compelling evidence that this project deserves to be a new mainstream capability. Alternatively, the team might decide that it should be spun into its own organization or that it should be killed.

Incubation: Once hypothesis testing is complete, many projects will still need a period of incubation as the teams championing the projects gather additional data about the application, further build the minimum viable product (MVP), and get used to working together. Incubation requires dedicated leadership oversight from the horizon 1 organization to insure the fledgling project does not die of malnutrition (a lack of access to resources) or become an orphan (continue to work with no parent to guide them).

Integration and refactoring: At this point, if the innovation is Horizon 1 or 2, its time to integrate it into the existing organization. (Horizon 3 innovations are more likely set up as their own entities or at least divisions.) Trying to integrate new, unbudgeted, and unscheduled innovation projects into an engineering organization that has line item budgets for people and resources results in chaos and frustration. In addition, innovation projects carry both technical and organizational debt. This creates an impedance mismatch between the organizations that can be easily be resolved with a small dedicated refactoring team. Innovation then becomes a continuous cycle rather than a bottleneck.

Problem 5: The question being asked across the Department of Defense and national security community is, “Can we innovate like startups in Silicon Valley” and insert speed, urgency and agility into our work?

Solution 5: The reality is that the DoD/IC is not Silicon Valley. In fact, it’s much more like a large company with existing customers, existing products and the organizations built to support and service them. And much like large companies they are being disrupted by forces outside their control.

But what’s unique is, that unlike a large company that doesn’t know how to move rapidly, on the battlefields of Iraq and Afghanistan our combatant commands and national security community were more agile, creative and Lean than any startup. They wrote the book on how to collaborate (read Team of Teams) or adopt new technologies (see the Rapid Equipping Force.) The problem isn’t that these agencies and commands don’t know how to be innovative. The problem is they don’t know how to be innovative in peacetime when innovation succumbs to the daily demands of execution. Part of the reason is that large agencies are run by leaders who tend to be excellent Horizon 1 managers of existing people, process and resources but have no experience in building and leading Horizon 3 organizations.

The solution is to understand that an innovation pipeline requires different people, processes, procedures, and metrics, then execution.

Problem 6: How to get started? How to get leadership behind continuous innovation?

Solution 6: To leadership, incubators, cafes, accelerators and hackathons appear to be just background noise unrelated to their guidance and mission. Part of the problem lies with the innovators themselves. Lots of innovation activities celebrate the creation of demos, funding, new makerspaces, etc. but there is little accountability for the actual rapid deployment of useful tools. Once we can convince and demonstrate to leadership that continuous innovation can solve the Red Queen problem, we’ll have their attention and support.

We know how to do this. Our country requires it.
Let’s get started.

Lessons Learned

  • Organizations must constantly adapt and evolve, to survive when pitted against ever-evolving opposition in an ever-changing environment
  • Government agencies need to both innovate and execute
  • In peacetime innovation succumbs to the demands of execution
  • We need explicit guidance for innovation to agencies and their leadership requiring an innovation organization and process, that operates in parallel with the execution of current mission
  • We need an innovation pipeline that delivers rapid results, not separate, disconnected innovation activities

15 Responses

  1. Hi Steve,

    Innovation in the Intelligence community means just one thing: how to effectively reconcile offence and defense needs. From the news of recent years it seams that they have brokwn their own tools and those of the civilian government in order to sustainably keep an access everywere.

    For 4 years we’ve been build a new standard and certification body for end-to-end communication IT systems – and an initial open target architecture – that achieve ultra-high levels of assurance and concurrently solidly prevent malevolent abuse, through the Trustless Computing Association and its spin off startup TRUSTLESS.AI


  2. For additional background reading on the Red Queen problem and the DoD/IC, I recommend a 2007 paper by Dr. Calvin Andrus of the Central Intelligence Agency: Dancing with the Red Queen: Linear Intelligence in an Exponential World. Its available here: The paper explains the Red Queen issue in a bit more depth and also highlights 2007-era trends like “social software” as “creat(ing) a world for which the IC is grossly unprepared.”

  3. Steve,
    have you ever thought that lean startup and Valley dogmas are self-referential and self-full filling prophecies that overall hugely constrain innovation in appraches and methods?

    • all the time.

      • Because that dogma-prone attitude of the Valley and US in general – with its (“lean startup”, the perfect 2-5 minutes “stand up pitch”, the “slide deck” done Valley-style) tons of great talents and startups have been shunned from funding for a decade or more.

        You guys can takle the merit for the ICO model, whereby brilliant researcher, ideas, exeutioners and team can get funded with living 30 minutes from a VC, or spending 6-9monts fitting into the dogmas.

        That said, It think lean startup method is very good concept, “customer development process” (i.e. user-driven innovation) is fantastic, and that 98% of Valley people I’ve met still do not even understand what is meant for an MVP.

    • Could you be more specific about how do they “hugely constrain innovation”? Thanks

      • Let me better elaborate then.

        Because of the dogma-prone attitude of the Valley and US in general – with its “lean startup”, the perfect 2-5 minutes “stand up pitch”, the “slide deck” done Valley-style – tons of great talents and startups have been shunned from funding for a decades or more.

        The fact that most those that followed the dogms succeeded is, to a large extent, a self-fulfilling prophecy, as those that fund the next round reason the same way.

        Lean Startup and other dogma leaders of the Valley can take much of the merit for the ICO model, whereby brilliant researchers, ideas, executioners and teams cannot get funded without living 30 minutes drive from a top VC, or spending 6-9 months fitting into the dogmas. Although the ICO badly needs some widely-adopted vetting/due-diligence standards which are more that an easily swayable crowd of crypto investors

        That said, It think lean startup method is very good concept, “customer development process” (i.e. user-driven innovation) is fantastic, and that 98% of Valley people I’ve met still do not even understand what is meant for an MVP.

        • Lean Startup with Customer Development is (IMO) an evidence-based system-focussed (i.e. business model focussed rather than just product and customers focussed) approach to innovation, much like a “science of innovation” (as opposed to innovation theatre). Would you suggest the scientific process is dogma-prone as well?

          Please note I agree that a lot of SV and startup culture is dogma-prone (and similarly the culture in science) but that this is a flawed human implementation of the approach (and science) rather than a problem with the approach itself.

          PS 1. I call what most in my (the) startup community think of as Lean Startup, Faux Lean Startup, because as you suggest it comes (IMO) from a superficial and often mostly incorrect understanding. I think the introductory chapters to TSOM does the best job at really explaining Lean Startup with Customer Development.

          PS 2. I suggest call Lean Startup is more than user-driven or even customer-driven because users/customers are only involved in a part (admittedly a most important and significant part, but still only a part) of the business model. I tend to think of Design Thinking as more purely user-driven, whereas Lean Startup is more data-validated.

          • Hi Ashley, you raise good point. “customer development” process is great, and it is possibly the best implementation of “user-driven innovation” models.
            What I am saying is that many investor in the Valley are loosing opportunities because they pretend foreign innvoators and startups spend month fitting in their formats, methods and appearances, which have sciientific nature, but then it is an ecosystem problem. That’s where crowdfunding and ICOs come in to fill a void. But they have done it wrong so far. Ciao r

          • I agree, the (eco)system is flawed (but what system isn’t flawed?). Unfortunately, that’s life and we have to get used to it 🙂 All the best for success! Cheers, Ashley.

  4. Perhaps Stan McCrystal’s “Team of Teams’ should be made required reading as well.

  5. Congress. Innovation dies with Congress. Jobs. Innovation must be tied to jobs. If you don’t address those two factors, DoD will never breakout of its own largesse. Congress needs to write laws that allow DoD to spend money differently, and Congress needs to manage its oversight of DoD with innovation in mind. Congress also needs to be taught how innovation will preserve jobs in their states and districts or innovation is a non-starter. The Red Queen dilemma is great and nothing you said is wrong, per se, but Congress controls two things that hamper DoD innovation: money and oversight. Until you have a Congress that is COMMITTED to building a DoD that is blanketed in innovation, you’ll find individual pockets of awesomeness that are lost in the grand enterprise. If Congress tied money to innovation and innovation to oversight (and by extension, strict accountability), DoD would be light-years ahead of where it is today.

  6. Hi Steve,

    I really enjoyed this article and your pod cast on the growth show. I work in a DoD lab as an engineer/scientist and I am curious how we should use lean innovation in the context of a “corporate laboratory”. Are we the Horizon 3 people for the rest of the DoD? Or are we more like a big company working on our own Horizon 1 activities?

    Do you ever think that the lean innovation process should be shelf-similar like a fractal for large organizations. Meaning that the “corporate labs” are the horizon 3 people relative to the DoD but inside the lab itself there should be horizon 1,2,3 activities with this methodology extending to individual projects which in analogous to startup.

    Just curious.

    Keep up the good work!

  7. Reblogged this on The Perils of Improbable Potholes and commented:
    A perfect topic for Improbable Potholes! My idea for the blog was to serve as a repository of my own writing snippets combined with topics such as this, the “Red Queen problem.” I note that this looks similar to the “Hedonic Treadmill.”

    See, also:

Leave a Reply to rguerreschiCancel reply

%d bloggers like this: