Artificial Intelligence and Machine Learning– Explained

Artificial Intelligence is a once-in-a lifetime commercial and defense game changer

(download a PDF of this article here)

Hundreds of billions in public and private capital is being invested in Artificial Intelligence (AI) and Machine Learning companies. The number of patents filed in 2021 is more than 30 times higher than in 2015 as companies and countries across the world have realized that AI and Machine Learning will be a major disruptor and potentially change the balance of military power.

Until recently, the hype exceeded reality. Today, however, advances in AI in several important areas (here, here, here, here and here) equal and even surpass human capabilities.

If you haven’t paid attention, now’s the time.

Artificial Intelligence and the Department of Defense (DoD)
The Department of Defense has thought that Artificial Intelligence is such a foundational set of technologies that they started a dedicated organization- the JAIC – to enable and implement artificial intelligence across the Department. They provide the infrastructure, tools, and technical expertise for DoD users to successfully build and deploy their AI-accelerated projects.

Some specific defense related AI applications are listed later in this document.

We’re in the Middle of a Revolution
Imagine it’s 1950, and you’re a visitor who traveled back in time from today. Your job is to explain the impact computers will have on business, defense and society to people who are using manual calculators and slide rules. You succeed in convincing one company and a government to adopt computers and learn to code much faster than their competitors /adversaries. And they figure out how they could digitally enable their business – supply chain, customer interactions, etc. Think about the competitive edge they’d have by today in business or as a nation. They’d steamroll everyone.

That’s where we are today with Artificial Intelligence and Machine Learning. These technologies will transform businesses and government agencies. Today, 100s of billions of dollars in private capital have been invested in 1,000s of AI startups. The U.S. Department of Defense has created a dedicated organization to ensure its deployment.

But What Is It?
Compared to the classic computing we’ve had for the last 75 years, AI has led to new types of applications, e.g. facial recognition; new types of algorithms, e.g. machine learning; new types of computer architectures, e.g. neural nets; new hardware, e.g. GPUs; new types of software developers, e.g. data scientists; all under the overarching theme of artificial intelligence. The sum of these feels like buzzword bingo. But they herald a sea change in what computers are capable of doing, how they do it, and what hardware and software is needed to do it.

This brief will attempt to describe all of it.

New Words to Define Old Things
One of the reasons the world of AI/ML is confusing is that it’s created its own language and vocabulary. It uses new words to define programming steps, job descriptions, development tools, etc. But once you understand how the new world maps onto the classic computing world, it starts to make sense. So first a short list of some key definitions.

AI/ML – a shorthand for Artificial Intelligence/Machine Learning

Artificial Intelligence (AI) – a catchall term used to describe “Intelligent machines” which can solve problems, make/suggest decisions and perform tasks that have traditionally required humans to do. AI is not a single thing, but a constellation of different technologies.

Machine Learning (ML) – a subfield of artificial intelligence. Humans combine data with algorithms (see here for a list) to train a model using that data. This trained model can then make predications on new data (is this picture a cat, a dog or a person?) or decision-making processes (like understanding text and images) without being explicitly programmed to do so.

Machine learning algorithms – computer programs that adjust themselves to perform better as they are exposed to more data. The “learning” part of machine learning means these programs change how they process data over time. In other words, a machine-learning algorithm can adjust its own settings, given feedback on its previous performance in making predictions about a collection of data (images, text, etc.).

Deep Learning/Neural Nets – a subfield of machine learning. Neural networks make up the backbone of deep learning. (The “deep” in deep learning refers to the depth of layers in a neural network.) Neural nets are effective at a variety of tasks (e.g., image classification, speech recognition). A deep learning neural net algorithm is given massive volumes of data, and a task to perform – such as classification. The resulting model is capable of solving complex tasks such as recognizing objects within an image and translating speech in real time. In reality, the neural net is a logical concept that gets mapped onto a physical set of specialized processors. See here.)

Data Science – a new field of computer science. Broadly it encompasses data systems and processes aimed at maintaining data sets and deriving meaning out of them. In the context of AI, it’s the practice of people who are doing machine learning.

Data Scientists – responsible for extracting insights that help businesses make decisions. They explore and analyze data using machine learning platforms to create models about customers, processes, risks, or whatever they’re trying to predict.

What’s Different? Why is Machine Learning Possible Now?
To understand why AI/Machine Learning can do these things, let’s compare them to computers before AI came on the scene. (Warning – simplified examples below.)

Classic Computers

For the last 75 years computers (we’ll call these classic computers) have both shrunk to pocket size (iPhones) and grown to the size of warehouses (cloud data centers), yet they all continued to operate essentially the same way.

Classic Computers – Programming
Classic computers are designed to do anything a human explicitly tells them to do. People (programmers) write software code (programming) to develop applications, thinking a priori about all the rules, logic and knowledge that need to be built in to an application so that it can deliver a specific result. These rules are explicitly coded into a program using a software language (Python, JavaScript, C#, Rust, …).

Classic Computers –  Compiling
The code is then compiled using software to translate the programmer’s source code into a version that can be run on a target computer/browser/phone. For most of today’s programs, the computer used to develop and compile the code does not have to be that much faster than the one that will run it.

Classic Computers – Running/Executing Programs
Once a program is coded and compiled, it can be deployed and run (executed) on a desktop computer, phone, in a browser window, a data center cluster, in special hardware, etc. Programs/applications can be games, social media, office applications, missile guidance systems, bitcoin mining, or even operating systems e.g. Linux, Windows, IOS. These programs run on the same type of classic computer architectures they were programmed in.

Classic Computers – Software Updates, New Features
For programs written for classic computers, software developers receive bug reports, monitor for security breaches, and send out regular software updates that fix bugs, increase performance and at times add new features.

Classic Computers-  Hardware
The CPUs (Central Processing Units) that write and run these Classic Computer applications all have the same basic design (architecture). The CPUs are designed to handle a wide range of tasks quickly in a serial fashion. These CPUs range from Intel X86 chips, and the ARM cores on Apple M1 SoC, to the z15 in IBM mainframes.

Machine Learning

In contrast to programming on classic computing with fixed rules, machine learning is just like it sounds – we can train/teach a computer to “learn by example” by feeding it lots and lots of examples. (For images a rule of thumb is that a machine learning algorithm needs at least 5,000 labeled examples of each category in order to produce an AI model with decent performance.) Once it is trained, the computer runs on its own and can make predictions and/or complex decisions.

Just as traditional programming has three steps – first coding a program, next compiling it and then running it – machine learning also has three steps: training (teaching), pruning and inference (predicting by itself.)

Machine Learning – Training
Unlike programing classic computers with explicit rules, training is the process of “teaching” a computer to perform a task e.g. recognize faces, signals, understand text, etc. (Now you know why you’re asked to click on images of traffic lights, cross walks, stop signs, and buses or type the text of scanned image in ReCaptcha.) Humans provide massive volumes of “training data” (the more data, the better the model’s performance) and select the appropriate algorithm to find the best optimized outcome. (See the detailed “machine learning pipeline” section for the gory details.)

By running an algorithm selected by a data scientist on a set of training data, the Machine Learning system generates the rules embedded in a trained model. The system learns from examples (training data), rather than being explicitly programmed. (See the “Types of Machine Learning” section for more detail.) This self-correction is pretty cool. An input to a neural net results in a guess about what that input is. The neural net then takes its guess and compares it to a ground-truth about the data, effectively asking an expert “Did I get this right?” The difference between the network’s guess and the ground truth is its error. The network measures that error, and walks the error back over its model, adjusting weights to the extent that they contributed to the error.)

Just to make the point again: The algorithms combined with the training data – not external human computer programmers – create the rules that the AI uses. The resulting model is capable of solving complex tasks such as recognizing objects it’s never seen before, translating text or speech, or controlling a drone swarm.

(Instead of building a model from scratch you can now buy, for common machine learning tasks, pretrained models from others and here, much like chip designers buying IP Cores.)

Machine Learning Training – Hardware
Training a machine learning model is a very computationally intensive task. AI hardware must be able to perform thousands of multiplications and additions in a mathematical process called matrix multiplication. It requires specialized chips to run fast. (See the AI semiconductor section for details.)

Machine Learning – Simplification via pruning, quantization, distillation
Just like classic computer code needs to be compiled and optimized before it is deployed on its target hardware, the machine learning models are simplified and modified (pruned) to use less computing power, energy, and  memory before they’re deployed to run on their hardware.

Machine Learning – Inference Phase
Once the system has been trained it can be copied to other devices and run. And the computing hardware can now make inferences (predictions) on new data that the model has never seen before.

Inference can even occur locally on edge devices where physical devices meet the digital world (routers, sensors, IOT devices), close to the source of where the data is generated. This reduces network bandwidth issues and eliminates latency issues.

Machine Learning Inference – Hardware
Inference (running the model) requires substantially less compute power than training. But inference also benefits from specialized AI chips. (See the AI semiconductor section for details.)

Machine Learning – Performance Monitoring and Retraining
Just like classic computers where software developers do regular software updates to fix bugs and increase performance and add features, machine learning models also need to be updated regularly by adding new data to the old training pipelines and running them again. Why?

Over time machine learning models get stale. Their real-world performance generally degrades over time if they are not updated regularly with new training data that matches the changing state of the world. The models need to be monitored and retrained regularly for data and/or concept drift, harmful predictions, performance drops, etc. To stay up to date, the models need to re-learn the patterns by looking at the most recent data that better reflects reality.

One Last Thing – “Verifiability/Explainability”
Understanding how an AI works is essential to fostering trust and confidence in AI production models.

Neural Networks and Deep Learning differ from other types of Machine Learning algorithms in that they have low explainability. They can generate a prediction, but it is very difficult to understand or explain how it arrived at its prediction. This “explainability problem” is often described as a problem for all of AI, but it’s primarily a problem for Neural Networks and Deep Learning. Other types of Machine Learning algorithms – for example decision trees or linear regression– have very high explainability. The results of the five-year DARPA Explainable AI Program (XAI) are worth reading here.

So What Can Machine Learning Do?

It’s taken decades but as of today, on its simplest implementations, machine learning applications can do some tasks better and/or faster than humans. Machine Learning is most advanced and widely applied today in processing text (through Natural Language Processing) followed by understanding images and videos (through Computer Vision) and analytics and anomaly detection. For example:

Recognize and Understand Text/Natural Language Processing
AI is better than humans on basic reading comprehension benchmarks like SuperGLUE and SQuAD and their performance on complex linguistic tasks is almost there. Applications: GPT-3, M6, OPT-175B, Google Translate, Gmail Autocomplete, Chatbots, Text summarization.

Write Human-like Answers to Questions and Assist in Writing Computer Code
An AI can write original text that is indistinguishable from that created by humans. Examples GPT-3, Wu Dao 2.0 or generate computer code. Example GitHub Copilot, Wordtune

Recognize and Understand Images and video streams
An AI can see and understand what it sees. It can identify and detect an object or a feature in an image or video. It can even identify faces. It can scan news broadcasts or read and assess text that appears in videos. It has uses in threat detection –  airport security, banks, and sporting events. In medicine to interpret MRI’s or to design drugs. And in retail to scan and analyze in-store imagery to intuitively determine inventory movement. Examples of ImageNet benchmarks here and here

Turn 2D Images into 3D Rendered Scenes
AI using “NeRFs “neural radiance fields” can take 2d snapshots and render a finished 3D scene in realtime to create avatars or scenes for virtual worlds, to capture video conference participants and their environments in 3D, or to reconstruct scenes for 3D digital maps. The technology is an enabler of the metaverse, generating digital representations of real environments that creators can modify and build on. And self driving cars are using NeRF’s to render city-scale scenes spanning multiple blocks.

Detect Changes in Patterns/Recognize Anomalies
An AI can recognize patterns which don’t match the behaviors expected for a particular system, out of millions of different inputs or transactions. These applications can discover evidence of an attack on financial networks, fraud detection in insurance filings or credit card purchases; identify fake reviews; even tag sensor data in industrial facilities that mean there’s a safety issue. Examples here, here and here.

Power Recommendation Engines
An AI can provide recommendations based on user behaviors used in ecommerce to provide accurate suggestions of products to users for future purchases based on their shopping history. Examples: Netflix, TikTok, CrossingMinds and Recommendations AI

Recognize and Understand Your Voice
An AI can understand spoken language. Then it can comprehend what is being said and in what context. This can enable chatbots to have a conversation with people. It can record and transcribe meetings. (Some versions can even read lips to increase accuracy.) Applications: Siri/Alexa/Google Assistant. Example here

Create Artificial Images
AI can ​create artificial ​images​ (DeepFakes) that ​are​ indistinguishable ​from​ real ​ones using Generative Adversarial Networks.​ Useful in ​entertainment​, virtual worlds, gaming, fashion​ design, etc. Synthetic faces are now indistinguishable and more trustworthy than photos of real people. Paper here.

Create Artist Quality Illustrations from A Written Description
AI can generate images from text descriptions, creating anthropomorphized versions of animals and objects, combining unrelated concepts in plausible ways. An example application is Dall-E

Generative Design of Physical Products
Engineers can input design goals into AI-driven generative design software, along with parameters such as performance or spatial requirements, materials, manufacturing methods, and cost constraints. The software explores all the possible permutations of a solution, quickly generating design alternatives. Example here.

Sentiment Analysis
An AI leverages deep natural language processing, text analysis, and computational linguistics to gain insight into customer opinion, understanding of consumer sentiment, and measuring the impact of marketing strategies. Examples: Brand24, MonkeyLearn

What Does this Mean for Businesses?

Skip this section if you’re interested in national security applications

Hang on to your seat. We’re just at the beginning of the revolution. The next phase of AI, powered by ever increasing powerful AI hardware and cloud clusters, will combine some of these basic algorithms into applications that do things no human can. It will transform business and defense in ways that will create new applications and opportunities.

Human-Machine Teaming
Applications with embedded intelligence have already begun to appear thanks to massive language models. For example – Copilot as a pair-programmer in Microsoft Visual Studio VSCode. It’s not hard to imagine DALL-E 2 as an illustration assistant in a photo editing application, or GPT-3 as a writing assistant in Google Docs.

AI in Medicine
AI applications are already appearing in radiology, dermatology, and oncology. Examples: IDx-DR,OsteoDetect, Embrace2.  AI Medical image identification can automatically detect lesions, and tumors with diagnostics equal to or greater than humans. For Pharma, AI will power drug discovery design for finding new drug candidates. The FDA has a plan for approving AI software here and a list of AI-enabled medical devices here.

Autonomous Vehicles
Harder than it first seemed, but car companies like Tesla will eventually get better than human autonomy for highway driving and eventually city streets.

Decision support
Advanced virtual assistants can listen to and observe behaviors, build and maintain data models, and predict and recommend actions to assist people with and automate tasks that were previously only possible for humans to accomplish.

Supply chain management
AI applications are already appearing in predictive maintenance, risk management, procurement, order fulfillment, supply chain planning and promotion management.

Marketing
AI applications are already appearing in real-time personalization, content and media optimization and campaign orchestration to augment, streamline and automate marketing processes and tasks constrained by human costs and capability, and to uncover new customer insights and accelerate deployment at scale.

Making business smarter: Customer Support
AI applications are already appearing in virtual customer assistants with speech recognition, sentiment analysis, automated/augmented quality assurance and other technologies providing customers with 24/7 self- and assisted-service options across channels.

AI in National Security

Much like the dual-use/dual-nature of classical computers AI developed for commercial applications can also be used for national security.

AI/ML and Ubiquitous Technical Surveillance
AI/ML have made most cities untenable for traditional tradecraft. Machine learning can integrate travel data (customs, airline, train, car rental, hotel, license plate readers…,) integrate feeds from CCTV cameras for facial recognition and gait recognition, breadcrumbs from wireless devices and then combine it with DNA sampling. The result is automated persistent surveillance.

China’s employment of AI as a tool of repression and surveillance of the Uyghurs is a reminder of a dystopian future of how totalitarian regimes will use AI-enabled ubiquitous surveillance to repress and monitor its own populace.

AI/ML on the Battlefield
AI will enable new levels of performance and autonomy for weapon systems. Autonomously collaborating assets (e.g., drone swarms, ground vehicles) that can coordinate attacks, ISR missions, & more.

Fusing and making sense of sensor data (detecting threats in optical /SAR imagery, classifying aircraft based on radar returns, searching for anomalies in radio frequency signatures, etc.) Machine learning is better and faster than humans in finding targets hidden in a high-clutter background. Automated target detection and fires from satellite/UAV.

For example, an Unmanned Aerial Vehicle (UAV) or Unmanned Ground Vehicles with on board AI edge computers could use deep learning to detect and locate concealed chemical, biological and explosive threats by fusing imaging sensors and chemical/biological sensors.

Other examples include:

Use AI/ML countermeasures against adversarial, low probability of intercept/low probability of detection (LPI/LPD) radar techniques in radar and communication systems.

Given sequences of observations of unknown radar waveforms from arbitrary emitters without a priori knowledge, use machine learning to develop behavioral models to enable inference of radar intent and threat level, and to enable prediction of future behaviors.

For objects in space, use machine learning to predict and characterize a spacecrafts possible actions, its subsequent trajectory, and what threats it can pose from along that trajectory. Predict the outcomes of finite burn, continuous thrust, and impulsive maneuvers.

AI empowers other applications such as:

AI/ML in Collection
The front end of intelligence collection platforms has created a firehose of data that have overwhelmed human analysts. “Smart” sensors coupled with inference engines that can pre-process raw intelligence and prioritize what data to transmit and store –helpful in degraded or low-bandwidth environments.

Human-Machine Teaming in Signals Intelligence
Applications with embedded intelligence have already begun to appear in commercial applications thanks to massive language models. For example – Copilot as a pair-programmer in Microsoft Visual Studio VSCode. It’s not hard to imagine an AI that can detect and isolate anomalies and other patterns of interest in all sorts of signal data faster and more reliably than human operators.

AI-enabled natural language processing, computer vision, and audiovisual analysis can vastly reduce manual data processing. Advances in speech-to-text transcription and language analytics now enable reading comprehension, question answering, and automated summarization of large quantities of text. This not only prioritizes the work of human analysts, it’s a major force multiplier

AI can also be used to automate data conversion such as translations and decryptions, accelerating the ability to derive actionable insights.

Human-Machine Teaming in Tasking and Dissemination
AI-enabled systems will automate and optimize tasking and collection for platforms, sensors, and assets in near-real time in response to dynamic intelligence requirements or changes in the environment.

AI will be able to automatically generate machine-readable versions of intelligence products and disseminate them at machine speed so that computer systems across the IC and the military can ingest and use them in real time without manual intervention.

Human-Machine Teaming in Exploitation and Analytics
AI-enabled tools can augment filtering, flagging, and triage across multiple data sets. They can identify connections and correlations more efficiently and at a greater scale than human analysts, and can flag those findings and the most important content for human analysis.

AI can fuse data from multiple sources, types of intelligence, and classification levels to produce accurate predictive analysis in a way that is not currently possible. This can improve indications and warnings for military operations and active cyber defense.

AI/ML Information warfare
Nation states have used AI systems to enhance disinformation campaigns and cyberattacks. This included using “DeepFakes” (fake videos generated by a neural network that are nearly indistinguishable from reality). They are harvesting data on Americans to build profiles of our beliefs, behavior, and biological makeup for tailored attempts to manipulate or coerce individuals.

But because a large percentage of it is open-source AI is not limited to nation states, AI-powered cyber-attacks, deepfakes and AI software paired with commercially available drones can create “poor-man’s smart weapons” for use by rogue states, terrorists and criminals.

AI/ML Cyberwarfare
AI-enabled malware can learn and adapt to a system’s defensive measures, by probing a target system to look for system configuration and operational patterns and customize the attack payload to determine the most opportune time to execute the payload so to maximize the impact. Conversely, AI-enabled cyber-defensive tools can proactively locate and address network anomalies and system vulnerabilities.

Attacks Against AI – Adversarial AI
As AI proliferates, defeating adversaries will be predicated on defeating their AI and vice versa. As Neural Networks take over sensor processing and triage tasks, a human may only be alerted if the AI deems it suspicious. Therefore, we only need to defeat the AI to evade detection, not necessarily a human.

Adversarial attacks against AI fall into three types:

AI Attack Surfaces
Electronic Attack (EA), Electronic Protection (EP), Electronic Support (ES) all have analogues in the AI algorithmic domain. In the future, we may play the same game about the “Algorithmic Spectrum,” denying our adversaries their AI capabilities while defending ours. Other can steal or poison our models  or manipulate our training data.

What Makes AI Possible Now?

 Four changes make Machine Learning possible now:

  1. Massive Data Sets
  2. Improved Machine Learning algorithms
  3. Open-Source Code, Pretrained Models and Frameworks
  4. More computing power

Massive Data Sets
Machine Learning algorithms tend to require large quantities of training data in order to produce high-performance AI models. (Training OpenAI’s GPT-3 Natural Language Model with 175 billion parameters takes 1,024 Nvidia A100 GPUs more than one month.) Today, strategic and tactical sensors pour in a firehose of images, signals and other data. Billions of computers, digital devices and sensors connected to the Internet, producing and storing large volumes of data, which provide other sources of intelligence. For example facial recognition requires millions of labeled images of faces for training data.

Of course more data only helps if the data is relevant to your desired application. Training data needs to match the real-world operational data very, very closely to train a high-performing AI model.

Improved Machine Learning algorithms
The first Machine Learning algorithms are decades old, and some remain incredibly useful. However, researchers have discovered new algorithms that have greatly sped up the fields cutting-edge. These new algorithms have made Machine Learning models more flexible, more robust, and more capable of solving different types of problems.

Open-Source Code, Pretrained Models and Frameworks
Previously, developing Machine Learning systems required a lot of expertise and custom software development that made it out of reach for most organizations. Now open-source code libraries and developer tools allow organizations to use and build upon the work of external communities. No team or organization has to start from scratch, and many parts that used to require highly specialized expertise have been automated. Even non-experts and beginners can create useful AI tools. In some cases, open-source ML models can be entirely reused and purchased. Combined with standard competitions, open source, pretrained models and frameworks have moved the field forward faster than any federal lab or contractor. It’s been a feeding frenzy with the best and brightest researchers trying to one-up each other to prove which ideas are best.

The downside is that, unlike past DoD technology development – where the DoD leads it, can control it, and has the most advanced technology (like stealth and electronic warfare), in most cases the DoD will not have the most advanced algorithms or models. The analogy for AI is closer to microelectronics than it is EW. The path forward for the DoD should be supporting open research, but optimizing on data set collection, harvesting research results, and fast application. 

More computing power – special chips
Machine Learning systems require a lot of computing power. Today, it’s possible to run Machine Learning algorithms on massive datasets using commodity Graphics Processing Units (GPUs). While many of the AI performance improvements have been due to human cleverness on better models and algorithms, most of the performance gains have been the massive increase in compute performance.  (See the semiconductor section.)

More computing power – AI In the Cloud
The rapid growth in the size of machine learning models has been achieved by the move to large data center clusters. The size of machine learning models are limited by time to train them. For example, in training images, the size of the model scales with the number of pixels in an image. ImageNet Model sizes are 224×224 pixels. But HD (1920×1080) images require 40x more computation/memory. Large Natural Language Processing models – e.g. summarizing articles, English-to-Chinese translation like OpenAI’s GPT-3 require enormous models. GPT-3 uses 175 billion parameters and was trained on a cluster with 1,024 Nvidia A100 GPUs that cost ~$25 million! (Which is why large clusters exist in the cloud, or the largest companies/ government agencies.) Facebook’s Deep Learning and Recommendation Model (DLRM) was trained on 1TB data and has 24 billion parameters. Some cloud vendors train on >10TB data sets.

Instead of investing in massive amounts of computers needed for training companies can use the enormous on-demand, off-premises hardware in the cloud (e.g. Amazon AWS, Microsoft Azure) for both training machine learning models and deploying inferences.

We’re Just Getting Started
Progress in AI has been growing exponentially. The next 10 years will see a massive improvement on AI inference and training capabilities. This will require regular refreshes of the hardware– on the chip and cloud clusters – to take advantage. This is the AI version of Moore’s Law on steroids – applications that are completely infeasible today will be easy in 5 years.

What Can’t AI Do?

While AI can do a lot of things better than humans when focused on a narrow objective, there are many things it still can’t do. AI works well in specific domain where you have lots of data, time/resources to train, domain expertise to set the right goals/rewards during training, but that is not always the case.

For example AI models are only as good as the fidelity and quality of the training data. Having bad labels can wreak havoc on your training results. Protecting the integrity of the training data is critical.

In addition, AI is easily fooled by out-of-domain data (things it hasn’t seen before). This can happen by “overfitting” – when a model trains for too long on sample data or when the model is too complex, it can start to learn the “noise,” or irrelevant information, within the dataset. When the model memorizes the noise and fits too closely to the training set, the model becomes “overfitted,” and it is unable to generalize well to new data. If a model cannot generalize well to new data, then it will not be able to perform the classification or prediction tasks it was intended for. However, if you pause too early or exclude too many important features, you may encounter the opposite problem, and instead, you may “underfit” your model. Underfitting occurs when the model has not trained for enough time, or the input variables are not significant enough to determine a meaningful relationship between the input and output variables.

AI is also poor at estimating uncertainty /confidence (and explaining its decision-making). It can’t choose its own goals. (Executives need to define the decision that the AI will execute.  Without well-defined decisions to be made, data scientists will waste time, energy and money.) Except for simple cases an AI can’t (yet) figure out cause and effect or why something happened. It can’t think creatively or apply common sense.

AI is not very good at creating a strategy (unless it can pull from previous examples and mimic them, but then fails with the unexpected.) And it lacks generalized intelligence e.g. that can generalize knowledge and translate learning across domains.

All of these are research topics actively being worked on. Solving these will take a combination of high-performance computing, advanced AI/ML semiconductors, creative machine learning implementations and decision science. Some may be solved in the next decade, at least to a level where a human can’t tell the difference.

Where is AI in Business Going Next?

Skip this section if you’re interested in national security applications

Just as classic computers were applied to a broad set of business, science and military applications, AI is doing the same. AI is exploding not only in research and infrastructure (which go wide) but also in the application of AI to vertical problems (which go deep and depend more than ever on expertise). Some of the new applications on the horizon include Human AI/Teaming (AI helping in programming and decision making), smarter robotics and autonomous vehicles, AI-driven drug discovery and design, healthcare diagnostics, chip electronic design automation, and basic science research.

Advances in language understanding are being pursued to create systems that can summarize complex inputs and engage through human-like conversation, a critical component of next-generation teaming.

Where is AI and National Security Going Next?

In the near future AI may be able to predict the future actions an adversary could take and the actions a friendly force could take to counter these. The 20th century model loop of Observe–Orient–Decide and Act (OODA) is retrospective; an observation cannot be made until after the event has occurred. An AI-enabled decision-making cycle might be ‘sense–predict–agree–act’: AI senses the environment; predicts what the adversary might do and offers what a future friendly force response should be; the human part of the human–machine team agrees with this assessment; and AI acts by sending machine-to-machine instructions to the small, agile and many autonomous warfighting assets deployed en masse across the battlefield.

An example of this is DARPA’s ACE (Air Combat Evolution) program that is developing a warfighting concept for combined arms using a manned and unmanned systems. Humans will fight in close collaboration with autonomous weapon systems in complex environments with tactics informed by artificial intelligence.

A Once-in-a-Generation Event
Imagine it’s the 1980’s and you’re in charge of an intelligence agency. SIGINT and COMINT were analog and RF. You had worldwide collection systems with bespoke systems in space, air, underwater, etc. And you wake up to a world that shifts from copper to fiber. Most of your people, and equipment are going to be obsolete, and you need to learn how to capture those new bits. Almost every business processes needed to change, new organizations needed to be created, new skills were needed, and old ones were obsoleted. That’s what AI/ML is going to do to you and your agency.

The primary obstacle to innovation in national security is not technology, it is culture. The DoD and IC must overcome a host of institutional, bureaucratic, and policy challenges to adopting and integrating these new technologies. Many parts of our culture are resistant to change, reliant on traditional tradecraft and means of collection, and averse to risk-taking, (particularly acquiring and adopting new technologies and integrating outside information sources.)

History tells us that late adopters fall by the wayside as more agile and opportunistic governments master new technologies.

Carpe Diem.

Want more Detail?

Read on if you want to know about Machine Learning chips, see a sample Machine Learning Pipeline and learn about the four types of Machine Learning.

 

Artificial Intelligence/Machine Learning Semiconductors

Skip this section if all you need to know is that special chips are used for AI/ML.

AI/ML, semiconductors, and high-performance computing are intimately intertwined  – and progress in each is dependent on the others.  (See the “Semiconductor Ecosystem” report.)

Some machine learning models can have trillions of parameters and require a massive number of specialized AI chips to run. Edge computers are significantly less powerful than the massive compute power that’s located at data centers and the cloud. They need low power and specialized silicon.

Why Dedicated AI Chips and Chip Speed Matter
Dedicated chips for neutral nets (e.g. Nvidia GPUs, Xilinx FPUs, Google TPUs) are faster than conventional CPUs for three reasons: 1) they use parallelization, 2) they have larger memory bandwidth and 3) they have fast memory access.

There are three types of AI Chips:

  • Graphics Processing Units (GPUs) – Thousands of cores, parallel workloads, widespread use in machine learning
  • Field-Programmable Gate Arrays (FPGAs) – Good for algorithms; compression, video encoding, cryptocurrency,  genomics, search. Needs specialists to program
  • Application-Specific Integrated Circuits (ASICs) – custom chips e.g. Google TPU’s

Matrix multiplication plays a big part in neural network computations, especially if there are many layers and nodes. Graphics Processing Units (GPUs) contain 100s or 1,000s of cores that can do these multiplications simultaneously. And neural networks are inherently parallel which means that it’s easy to run a program across the cores and clusters of these processors. That makes AI chips 10s or even 1,000s of times faster and more efficient than classic CPUs for training and inference of AI algorithms. State-of-the-art AI chips are dramatically more cost-effective than state-of-the-art CPUs as a result of their greater efficiency for AI algorithms.

Cutting-edge AI systems require not only AI-specific chips, but state-of-the-art AI chips. Older AI chips incur huge energy consumption costs that quickly balloon to unaffordable levels. Using older AI chips today means overall costs and slowdowns at least an order of magnitude greater than for state-of- the-art AI chips.

Cost and speed make it virtually impossible to develop and deploy cutting-edge AI algorithms without state-of-the-art AI chips. Even with state-of-the-art AI chips, training a large AI algorithm can cost tens of millions of dollars and take weeks to complete. With general-purpose chips like CPUs or older AI chips, this training would take much longer and cost orders of magnitude more, making staying at the R&D frontier impossible. Similarly, performing inference using less advanced or less specialized chips could involve similar cost overruns and take orders of magnitude longer.

In addition to off-the-shelf AI chips from Nvidia, Xlinix and Intel, large companies like Facebook, Google, Amazon, have designed their own chips to accelerate AI. The opportunity is so large that there are hundreds of AI accelerator startups designing their own chips, funded by 10’s of billions of venture capital and private equity. None of these companies own a chip manufacturing plant (a fab) so they all use a foundry (an independent company that makes chips for others) like TSMC in Taiwan (or SMIC in China for for its defense related silicon.)

A Sample of AI GPU, FPGA and ASIC AI Chips and Where They’re Made

IP (Intellectual Property) Vendors Also Offer AI Accelerators
AI chip designers can buy AI IP Cores – prebuilt AI accelerators from Synopsys (EV7x,) Cadence (Tensilica AI,) Arm (Ethos,) Ceva (SensPro2, NeuPro), Imagination (Series4,) ThinkSilicon (Neox,) FlexLogic (eFPGA,) Edgecortix and others.

Other AI Hardware Architectures
Spiking Neural Networks (SNN) is a completely different approach from Deep Neural Nets. A form of Neuromorphic computing it tries to emulate how a brain works. SNN neurons use simple counters and adders—no matrix multiply hardware is needed and power consumption is much lower. SNNs are good at unsupervised learning – e.g. detecting patterns in unlabeled data streams. Combined with their low power they’re a good fit for sensors at the edge. Examples: BrainChip, GrAI Matter, Innatera, Intel.

Analog Machine Learning AI chips use analog circuits to do the matrix multiplication in memory. The result is extremely low power AI for always-on sensors. Examples: Mythic (AMP,) Aspinity (AML100,) Tetramem.

Optical (Photonics) AI Computation promise performance gains over standard digital silicon, and some are nearing production. They use intersecting coherent light beams rather than switching transistors to perform matrix multiplies. Computation happens in picoseconds and requires only power for the laser. (Though off-chip digital transitions still limit power savings.) Examples: Lightmatter, Lightelligence, Luminous, Lighton.

AI Hardware for the Edge
As more AI moves to the edge, the Edge AI accelerator market is segmenting into high-end chips for camera-based systems and low-power chips for simple sensors. For example:

AI Chips in Autonomous vehicles, Augmented Reality and multicamera surveillance systems These inference engines require high performance. Examples: Nvidia (Orin,) AMD (Versal,) Qualcomm (Cloud AI 100,) and acquired Arriver for automotive software.

AI Chips in Cameras for facial recognition, surveillance. These inference chips require a balance of processing power with low power. Putting an AI chip in each camera reduces latency and bandwidth. Examples: Hailo-8, Ambarella CV5S,  Quadric (Q16), (RealTek 3916N).

Ultralow-Power AI Chips Target IoT Sensors – IoT devices require very simple neural networks and can run for years on a single battery. Example applications: Presence detection, wakeword detection, gunshot detection… Examples: Syntiant (NDP,) Innatera, BrainChip

Running on the edge devices are deep learning models such as OmniMLFoghorn, specifically designed for edge accelerators.

AI/ML Hardware Benchmarks
While there are lots of claims about how much faster each of these chips are for AI/ML there are now a set of standard benchmarks –  MLCommons. These benchmarks were created by Google, Baidu, Stanford, Harvard and U.C. Berkeley.

One Last Thing – Non-Nvidia AI Chips and the “Nvidia Software Moat”
New AI accelerator chips have to cross the software moat that Nvidia has built around their GPU’s. As popular AI applications and frameworks are built on Nvidia CUDA software platform,  if new AI Accelerator vendors want to port these applications to their chips they have to build their own drivers, compiler, debugger, and other tools.

Details of a machine learning pipeline

This is a sample of the workflow (a pipeline) data scientists use to develop, deploy and maintain a machine learning model (see the detailed description here.)

The Types of Machine Learning

skip this section if you want to believe it’s magic.

Machine Learning algorithms fall into four classes:

  1. Supervised Learning
  2. Unsupervised Learning
  3. Semi-supervised Learning
  4. Reinforcement Learning

They differ based on:

  • What types of data their algorithms can work with
  • For supervised and unsupervised learning, whether or not the training data is labeled or unlabeled
  • How the system receives its data inputs

Supervised Learning

  • A “supervisor” (a human or a software system) accurately labels each of the training data inputs with its correct associated output
  • Note that pre-labeled data is only required for the training data that the algorithm uses to train the AI mode
  • In operation in the inference phase the AI will be generating its own labels, the accuracy of which will depend on the AI’s training
  • Supervised Learning can achieve extremely high performance, but they require very large, labeled datasets
  • Using labeled inputs and outputs, the model can measure its accuracy and learn over time
  • For images a rule of thumb is that the algorithm needs at least 5,000 labeled examples of each category in order to produce an AI model with decent performance
  • In supervised learning, the algorithm “learns” from the training dataset by iteratively making predictions on the data and adjusting for the correct answer.
  • While supervised learning models tend to be more accurate than unsupervised learning models, they require upfront human intervention to label the data appropriately.

Supervised Machine Learning – Categories and Examples:

  • Classification problems – use an algorithm to assign data into specific categories, such as separating apples from oranges. Or classify spam in a separate folder from your inbox. Linear classifiers, support vector machines, decision trees and random forest are all common types of classification algorithms.
  • Regression– understands the relationship between dependent and independent variables. Helpful for predicting numerical values based on different data points, such as sales revenue projections for a given business. Some popular regression algorithms are linear regression, logistic regression and polynomial regression.
  • Example algorithms include: Logistic Regression and Back Propagation Neural Networks

Unsupervised Learning

  • These algorithms can analyze and cluster unlabeled data sets. They discover hidden patterns in data without the need for human intervention (hence, they are “unsupervised”)
  • They can extract features from the data without a label for the results
  • For an image classifier, an unsupervised algorithm would not identify the image as a “cat” or a “dog.” Instead, it would sort the training dataset into various groups based on their similarity
  • Unsupervised Learning systems are often less predictable, but as unlabeled data is usually more available than labeled data, they are important
  • Unsupervised algorithms are useful when developers want to understand their own datasets and see what properties might be useful in either developing automation or change operational practices and policies
  • They still require some human intervention for validating the output 

Unsupervised Machine Learning – Categories and Examples

  • Clustering groups unlabeled data based on their similarities or differences. For example, K-means clustering algorithms assign similar data points into groups, where the K value represents the size of the grouping and granularity. This technique is helpful for market segmentation, image compression, etc.
  • Association finds relationships between variables in a given dataset. These methods are frequently used for market basket analysis and recommendation engines, along the lines of “Customers Who Bought This Item Also Bought” recommendations.
  • Dimensionality reduction is used when the number of features  (or dimensions) in a given dataset is too high. It reduces the number of data inputs to a manageable size while also preserving the data integrity. Often, this technique is used in the preprocessing data stage, such as when autoencoders remove noise from visual data to improve picture quality.
  • Example algorithms include: Apriori algorithm and K-Means

Difference between supervised and unsupervised learning

The main difference: Labeled data

  • Goals: In supervised learning, the goal is to predict outcomes for new data. You know up front the type of results to expect. With an unsupervised learning algorithm, the goal is to get insights from large volumes of new data. The machine learning itself determines what is different or interesting from the dataset.
  • Applications: Supervised learning models are ideal for spam detection, sentiment analysis, weather forecasting and pricing predictions, among other things. In contrast, unsupervised learning is a great fit for anomaly detection, recommendation engines, customer personas and medical imaging.
  • ComplexitySupervised learning is a simple method for machine learning, typically calculated through the use of programs like R or Python. In unsupervised learning, you need powerful tools for working with large amounts of unclassified data. Unsupervised learning models are computationally complex because they need a large training set to produce intended outcomes.
  • Drawbacks: Supervised learning models can be time-consuming to train, and the labels for input and output variables require expertise. Meanwhile, unsupervised learning methods can have wildly inaccurate results unless you have human intervention to validate the output variables.

Semi-Supervised Learning

  • “Semi- Supervised” algorithms combine techniques from Supervised and Unsupervised algorithms for applications with a small set of labeled data and a large set of unlabeled data.
  • In practice, using them leads to exactly what you would expect, a mix of some of both of the strengths and weaknesses of Supervised and Unsupervised approaches
  • Typical algorithms are extensions to other flexible methods that make assumptions about how to model the unlabeled data. An example is Generative Adversarial Networks trained on photographs can generate new photographs that look authentic to human observers (deep fakes)

Reinforcement Learning

  • Training data is collected by an autonomous, self-directed AI agent as it perceives its environment and performs goal-directed actions
  • The rewards are input data received by the AI agent when certain criteria are satisfied.
  • These criteria are typically unknown to the agent at the start of training
  • Rewards often contain only partial information. They don’t signal which inputs were good or not
  • The system is learning to take actions to maximize its receipt of cumulative rewards
  • Reinforcement AI can defeat humans– in chess, Go…
  • There are no labeled datasets for every possible move
  • There is no assessment of whether it was a “good or bad move
  • Instead, partial labels reveal the final outcome “win” or “lose”
  • The algorithms explore the space of possible actions to learn the optimal set of rules for determining the best action that maximize wins

Reinforcement Machine Learning – Categories and Examples

  • Algorithm examples include: DQN (Deep Q Network), DDPG (Deep Deterministic Policy Gradient), A3C (Asynchronous Advantage Actor-Critic Algorithm), NAF (Q-Learning with Normalized Advantage Functions), …
  • AlphaGo, a Reinforcement system played 4.9 million games of Go in 3 days against itself to learn how to play the game at a world-champion level
  • Reinforcement is challenging to use in the real world, as the real world is not as heavily bounded as video games and time cannot be sped up in the real world
  • There are consequences to failure in the real world

(download a PDF of this article here)

Sources:

Lessons for the DoD – From Ukraine and China

 Portions of this post previously appeared in War On the Rocks.


Looking at a satellite image of Ukraine online I realized it was from Capella Space – one of our Hacking for Defense student teams who now has 7 satellites in orbit.

National Security is Now Dependent on Commercial Technology
They’re not the only startup in this fight. An entire wave of new startups and scaleups are providing satellite imagery and analysis, satellite communications, and unmanned aerial vehicles supporting the struggle.

For decades, satellites that took detailed pictures of Earth were only available to governments and the high-resolution images were classified. Today, commercial companies have their own satellites providing unclassified imagery. The government buys and distributes commercial images from startups to supplement their own and shares them with Ukraine as part of a broader intelligence-sharing arrangement that the head of Defense Intelligence Agency described as “revolutionary.” By the end of the decade, there will be 1000 commercial satellites for every U.S. government satellite in orbit.

At the onset of the war in Ukraine, Russia launched a cyber-attack on Viasat’s KA-SAT satellite, which supplies Internet across Europe, including to Ukraine. In response, to a (tweeted) request from Ukraine’s vice prime minister, Elon Musk’s Starlink satellite company shipped thousands of their satellite dishes and got Ukraine back on the Internet. Other startups are providing portable cell towers – “backpackable” and fixed.  When these connect via satellite link, they can provide phone service and WIFI capability. Another startup is providing a resilient, mesh local area network for secure tactical communications supporting ground units.

Drone technology was initially only available to national governments and militaries but is now democratized to low price points and available as internet purchases. In Ukraine, drones from startups are being used as automated delivery vehicles for resupply, and for tactical reconnaissance to discover where threats are. When combined with commercial satellite imagery, this enables pinpoint accuracy to deliver maximum kinetic impact in stopping opposing forces.

Equipment from large military contractors and other countries is also part of the effort. However, the equipment listed above is available commercially off-the-shelf, at dramatically cheaper prices than what’s offered by the large existing defense contractors, and developed and delivered in a fraction of the time. The Ukraine conflict is demonstrating the changing character of war such that low-cost emerging commercial technology is extremely effective when deployed against a larger 20th-century industrialized force that Russia is fielding.

While we should celebrate the organizations that have created and fielded these systems, the battle for the Ukraine illustrates much larger issues in the Department of Defense.

For the first time ever our national security is inexorably intertwined with commercial technology (drones, AI, machine learning, autonomy, biotech, cyber, semiconductors, quantum, high-performance computing, commercial access to space, et al.) And as we’re seeing on the Ukrainian battlefield they are changing the balance of power.

The DoD’s traditional suppliers of defense tools, technologies, and weapons – the prime contractors and federal labs – are no longer the leaders in these next-generation technologies – drones, AI, machine learning, semiconductors, quantum, autonomy, biotech, cyber, quantum, high performance computing, et al. They know this and know that weapons that can be built at a fraction of the cost and upgraded via software will destroy their existing business models.

Venture capital and startups have spent 50 years institutionalizing the rapid delivery of disruptive innovation. In the U.S., private investors spent $300 billion last year to fund new ventures that can move with the speed and urgency that the DoD now requires. Meanwhile China has been engaged in a Civil/Military Fusion program since 2015 to harness these disruptive commercial technologies for its national security needs.

China – Civil/Military Fusion
Every year the Secretary of Defense has to issue a formal report to Congress: Military and Security Developments Involving the People’s Republic of China. Six pages of this year’s report describe how China is combining its military-civilian sectors as a national effort for the PRC to develop a “world-class” military and become a world leader in science and technology. A key part of Beijing’s strategy includes developing and acquiring advanced dual-use technology. It’s worth thinking about what this means – China is not just using its traditional military contractors to build its defense ecosystem; they’re mobilizing their entire economy – commercial plus military suppliers. And we’re not.

DoD’s Civil/Military Orphan-Child – the Defense Innovation Unit
In 2015, before China started its Civil/Military effort, then-Secretary of Defense Ash Carter, saw the need for the DoD to understand, embrace and acquire commercial technology. To do so he started the Defense Innovation Unit (DIU). With offices in Silicon Valley, Austin, Boston, Chicago and Washington, DC, this is the one DoD organization with the staffing and mandate to match commercial startups or scaleups to pressing national security problems. DIU bridges the divide between DOD requirements and the commercial technology needed to address them with speed and urgency. It accelerates the connection of commercial technology to the military. Just as importantly, DIU helps the Department of Defense learn how to innovate at the same speed as tech-driven companies.

Many of the startups providing Ukraine satellite imagery and analysis, satellite communications, and unmanned aerial vehicles were found by the Defense Innovation Unit (DIU). Given that DIU is the Department of Defense’s most successful organization in developing and acquiring advanced dual-use technology, one would expect the department to scale the Defense Innovation Unit by a factor of ten. (Two years ago, the House Armed Services Committee in its Future of Defense Task Force report recommended exactly that—a 10X increase in budget.) The threats are too imminent and stakes too high not to do so.

So what happened?

Congress cut their budget by 20%.

And their well-regarded director just resigned in frustration because the Department is not resourcing DIU nor moving fast enough or broadly enough in adopting commercial technology.

Why? The Defense Ecosystem is at a turning point. Defense innovation threatens entrenched interests. Given that the Pentagon budget is essentially fixed, creating new vendors and new national champions of the next generation of defense technologies becomes a zero-sum game.

The Defense Innovation Unit (DIU) had no advocates in its chain of command willing to go to bat for it, let alone scale it.

The Department of Defense has world-class people and organization for a world that no longer exists
The Pentagon’s relationship with startups and commercial companies, already an arms-length one, is hindered by a profound lack of understanding about how the commercial innovation ecosystem works and its failure of imagination about what venture and private equity funded innovation could offer. In the last few years new venture capital and private equity firms have raised money to invest in dual-use startups. New startups focused on national security have sprung up and they and their investors have been banging on the closed doors of the defense department.

If we want to keep pace with our adversaries, we need to stop acting like we can compete with one hand tied behind our back. We need a radical reinvention of our civil/military innovation relationship. This would use Department of Defense funding, private capital, dual-use startups, existing prime contractors and federal labs in a new configuration that could look like this:


Create a new defense ecosystem encompassing startups, and mid-sized companies at the bleeding edge, prime contractors as integrators of advanced technology, federally funded R&D centers refocused on areas not covered by commercial tech (nuclear and hypersonics). Make it permanent by creating an innovation doctrine/policy.

Reorganize DoD Research and Engineering to allocate its budget and resources equally between traditional sources of innovation and new commercial sources of innovation.

  • Scale new entrants to the defense industrial base in dual-use commercial tech – AI/ML, Quantum, Space, drones, autonomy, biotech, underwater vehicles, shipyards, etc. that are not the traditional vendors. Do this by picking winners. Don’t give out door prizes. Contracts should be >$100M so high-quality venture-funded companies will play. And issue debt/loans to startups.

Reorganize DoD Acquisition and Sustainment to create and buy from new 21st century arsenals – new shipyards, drone manufacturers, etc. that can make 1,000’s of extremely low cost, attritable systems – “the small, the agile and the many.”

  • Acquire at Speed. Today, the average Department of Defense major acquisition program takes anywhere from nine to 26 years to get a weapon in the hands of a warfighter. DoD needs a requirements, budgeting and acquisition process that operates at commercial speed (18 months or less) which is 10x faster than DoD procurement cycles. Instead of writing requirements, the department should rapidly assess solutions and engage warfighters in assessing and prototyping commercial solutions. We’ll know we’ve built the right ecosystem when a significant number of major defense acquisition programs are from new entrants.

  • Acquire with a commercially oriented process. Congress has already granted the Department of Defense “Other Transaction Authority” (OTA) as a way to streamline acquisitions so they do not need to use Federal Acquisition Regulations (FAR). DIU has created a “Commercial Solutions Opening” to mirror a commercial procurement process that leverages OTA. DoD could be applying Commercial Solutions Openings on a much faster and broader scale.

Integrate and create incentives for the Venture Capital/Private Equity ecosystem to invest at scale. The most important incentive would be for DoD to provide significant contracts for new entrants. (One new entrant which DIU introduced, Anduril, just received a follow-on contract for $1 billion. This should be one of many such contracts and not an isolated example.) More examples could include: matching dollars for national security investments (similar to the SBIR program but for investors), public/private partnership investment funds, incentivize venture capital funds with no-carry loans (debt funding) to, or tax holidays and incentives – to get $10’s of billions of private investment dollars in technology areas of national interest.

Buy where we can; build where we must. Congress mandated that the Department of Defense should use commercial off-the-shelf technology wherever possible, but the department fails to do this (see industry letter to the Department of Defense).

Coordinate with Allies. Expand the National Security Innovation Base (NSIB) to an Allied Security Innovation Base. Source commercial technology from allies.

This is a politically impossible problem for the Defense Department to solve alone. Changes at this scale will require Congressional and executive office action. Hard to imagine in the polarized political environment. But not impossible.

Put Different People in Charge and reorganize around this new ecosystem. The threats, speed of change, and technologies the United States faces in this century require radically different mindsets and approaches than those it faced in the 20th century. Today’s leaders in the DoD, executive branch and Congress haven’t fully grasped the size, scale, and opportunity of the commercial innovation ecosystem or how to build innovation processes to move with the speed and urgency to match the pace China has set.


Change is hard – on the people and organizations inside the DoD who’ve spent years operating with one mindset to be asked to pivot to a new one.

But America’s adversaries have exploited the boundaries and borders between its defense and commercial and economic interests. Current approaches to innovation across the government — both in the past and under the current administration —  are piecemeal, incremental, increasingly less relevant, and insufficient.

These are not problems of technology. It takes imagination, vision and the willingness to confront the status quo. So far, all are currently lacking.

Russia’s Black Sea flagship Moskva on the bottom of the ocean and the thousands of its destroyed tanks illustrate the consequences of a defense ecosystem living in the past. We need transformation not half-measures. The U.S. Department of Defense needs to change.

Historically, major defense reforms have come from inside the DoD, at other times Congress (National Security Act of 1947, Goldwater-Nichols Act of 1986) and others from the President (Roosevelt’s creation of the Joint Chiefs in 1942, Eisenhower and the Department of Defense Reorganization Act of 1958.)

It may be that the changes needed are so broad that the DoD can’t make them and Congress needs to act. If so, it’s their time to step up.

Carpe diem. Seize the day.

The Quantum Technology Ecosystem – Explained

If you think you understand quantum mechanics,
you don’t understand quantum mechanics

Richard Feynman

IBM Quantum Computer

Tens of billions of public and private capital are being invested in Quantum technologies. Countries across the world have realized that quantum technologies can be a major disruptor of existing businesses and change the balance of military power. So much so, that they have collectively invested ~$24 billion in in quantum research and applications.

At the same time, a week doesn’t go by without another story about a quantum technology milestone or another quantum company getting funded. Quantum has moved out of the lab and is now the focus of commercial companies and investors. In 2021 venture capital funds invested over $2 billion in 90+ Quantum technology companies. Over a $1 billion of it going to Quantum computing companies. In the last six months quantum computing companies IonQ, D-Wave and Rigetti went public at valuations close to a billion and half dollars. Pretty amazing for computers that won’t be any better than existing systems for at least another decade – or more.  So why the excitement about quantum?

The Quantum Market Opportunity

While most of the IPOs have been in Quantum Computing, Quantum technologies are used in three very different and distinct markets: Quantum Computing, Quantum Communications and Quantum Sensing and Metrology.

All of three of these markets have the potential for being disruptive. In time Quantum computing could obsolete existing cryptography systems, but viable commercial applications are still speculative. Quantum communications could allow secure networking but are not a viable near-term business. Quantum sensors could create new types of medical devices, as well as new classes of military applications, but are still far from a scalable business.

It’s a pretty safe bet that 1) the largest commercial applications of quantum technologies won’t be the ones these companies currently think they’re going to be, and 2) defense applications using quantum technologies will come first. 3) if and when they do show up they’ll destroy existing businesses and create new ones.

We’ll describe each of these market segments in detail. But first a description of some quantum concepts.

Key Quantum Concepts

Skip this section if all you want to know is that 1) quantum works, 2) yes, it is magic.

Quantum  – The word “Quantum” refers to quantum mechanics which explains the behavior and properties of atomic or subatomic particles, such as electrons, neutrinos, and photons.

Superposition – quantum particles exist in many possible states at the same time. So a particle is described as a “superposition” of all those possible states. They fluctuate until observed and measured. Superposition underpins a number of potential quantum computing applications.

Entanglement – is what Einstein called “spooky action at a distance.” Two or more quantum objects can be linked so that measurement of one dictates the outcomes for the other, regardless of how far apart they are. Entanglement underpins a number of potential quantum communications applications.

Observation – Superposition and entanglement only exist as long as quantum particles are not observed or measured. If you observe the quantum state you can get information, but it results in the collapse of the quantum system.

Qubit – is short for a quantum bit. It is a quantum computing element that leverages the principle of superposition to encode information via one of four methods: spin, trapped atoms and ions, photons, or superconducting circuits.

Quantum Computers – Background

Quantum computers are a really cool idea. They harness the unique behavior of quantum physics—such as superposition, entanglement, and quantum interference—and apply it to computing.

In a classical computer transistors can represent two states – either a 0 or 1. Instead of transistors Quantum computers use quantum bits (called qubits.) Qubits exist in superposition – both in 0 and 1 state simultaneously.

Classic computers use transistors as the physical building blocks of logic. In quantum computers they may use trapped ions, superconducting loops, quantum dots or vacancies in a diamond. The jury is still out.

In a classic computer 2-14 transistors make up the seven basic logic gates (AND, OR, NAND, etc.) In a quantum computer building a single logical Qubit require a minimum of 9 but more likely 100’s or thousands of physical Qubits (to make up for error correction, stability, decoherence and fault tolerance.)

In a classical computer compute-power increases linearly with the number of transistors and clock speed. In a Quantum computer compute-power increases exponentially with the addition of each logical qubit.

But qubits have high error rates and need to be ultracold. In contrast classical computers have very low error rates and operate at room temperature.

Finally, classical computers are great for general purpose computing. But quantum computers can theoretically solve some complex algorithms/ problems exponentially faster than a classical computer. And with a sufficient number of logical Qubits they can become a Cryptographically Relevant Quantum Computer (CRQC).  And this is where Quantum computers become very interesting and relevant for both commercial and national security. (More below.)

Types of Quantum Computers

Quantum computers could potentially do things at speeds current computers cannot. Think of the difference of how fast you can count on your fingers versus how fast today’s computers can count. That’s the same order of magnitude speed-up a quantum computer could have over today’s computers for certain applications.

Quantum computers fall into four categories:

  1. Quantum Emulator/Simulator
  2. Quantum Annealer
  3. NISQ – Noisy Intermediate Scale Quantum
  4. Universal Quantum Computer – which can be a Cryptographically Relevant Quantum Computer (CRQC)

When you remove all the marketing hype, the only type that matters is #4 – a Universal Quantum Computer. And we’re at least a decade or more away from having those.

Quantum Emulator/Simulator
These are classical computers that you can buy today that simulate quantum algorithms. They make it easy to test and debug a quantum algorithm that someday may be able to run on a Universal Quantum Computer. Since they don’t use any quantum hardware they are no faster than standard computers.

Quantum Annealer is a special purpose quantum computer designed to only run combinatorial optimization problems, not general-purpose computing, or cryptography problems. D-Wave has defined and owned this space. While they have more physical Qubits than any other current system they are not organized as gate-based logical qubits. Currently this is a nascent commercial technology in search of a future viable market.

Noisy Intermediate-Scale Quantum (NISQ) computers. Think of these as prototypes of a Universal Quantum Computer – with several orders of magnitude fewer bits. (They currently have 50-100 qubits, limited gate depths, and short coherence times.) As they are short several orders of magnitude of Qubits, NISQ computers cannot perform any useful computation, however they are a necessary phase in the learning, especially to drive total system and software learning in parallel to the hardware development. Think of them as the training wheels for future universal quantum computers.

Universal Quantum Computers / Cryptographically Relevant Quantum Computers (CRQC)
This is the ultimate goal. If you could build a universal quantum computer with fault tolerance (i.e. millions of error corrected physical qubits resulting in thousands of logical Qubits), you could run quantum algorithms in cryptography, search and optimization, quantum systems simulations, and linear equations solvers. (See here for a list of hundreds quantum algorithms.) These all would dramatically outperform classical computation on large complex problems that grow exponentially as more variables are considered. Classical computers can’t attack these problems in reasonable times without so many approximations that the result is useless. We simply run out of time and transistors with classical computing on these problems. These special algorithms are what make quantum computers potentially valuable. For example, Grover’s algorithm solves the problem for the unstructured search of data. Further, quantum computers are very good at minimization / optimizations…think optimizing complex supply chains, energy states to form complex molecules, financial models, etc.

However, while all of these algorithms might have commercial potential one day, no one has yet to come up with a use for them that would radically transform any business or military application. Except for one – and that one keeps people awake at night.

It’s Shor’s algorithm for integer factorization – an algorithm that underlies much of existing public cryptography systems.

The security of today’s public key cryptography systems rests on the assumption that breaking into those with a thousand or more digits is practically impossible. It requires factoring into large prime numbers (e.g., RSA) or elliptic curve (e.g., ECDSA, ECDH) or finite fields (DSA) that can’t be done with any type of classic computer regardless of how large. Shor’s factorization algorithm can crack these codes if run on a Universal Quantum Computer. Uh-oh!

Impact of a Cryptographically Relevant Quantum Computer (CRQC) Skip this section if you don’t care about cryptography.

Not only would a Universal Quantum Computer running Shor’s algorithm make today’s public key algorithms (used for asymmetric key exchanges and digital signatures) useless, someone can implement a “harvest-now-and-decrypt-later” attack to record encrypted documents now with intent to decrypt them in the future. That means everything you send encrypted today will be able to be read retrospectively. Many applications – from ATMs to emails – would be vulnerable—unless we replace those algorithms with those that are “quantum-safe”.

When Will Current Cryptographic Systems Be Vulnerable?

The good news is that we’re nowhere near having any viable Cryptographically Relevant Quantum Computer, now or in the next few years. However, you can estimate when this will happen by calculating how many logical Qubits are needed to run Shor’s Algorthim and how long it will it take to break these crypto systems. There are lots of people tracking these numbers (see here and here). Their estimate is that using 8,194 logical qubits using 22.27 million physical qubits, it would take a quantum computer 20 minutes to break RSA-2048. The best estimate is that this might be possible in 8 to 20 years.

Post-Quantum / Quantum-Resistant Codes

That means if you want to protect the content you’re sending now, you need to migrate to new Post-Quantum /Quantum-Resistant Codes. But there are three things to consider in doing so:

  1. shelf-life time: the number of years the information must be protected by cyber-systems
  2. migration time: the number of years needed to properly and safely migrate the system to a quantum-safe solution
  3. threat timeline: the number of years before threat actors will be able to break the quantum-vulnerable systems

These new cryptographic systems would secure against both quantum and conventional computers and can interoperate with existing communication protocols and networks. The symmetric key algorithms of the Commercial National Security Algorithm (CNSA) Suite were selected to be secure for national security systems usage even if a CRQC is developed.

Cryptographic schemes that commercial industry believes are quantum-safe include lattice-based cryptography, hash trees, multivariate equations, and super-singular isogeny elliptic curves.

Estimates of when you can actually buy a fully error-corrected quantum computers vary from “never” to somewhere between 8 to 20 years from now. (Some optimists believe even earlier.)

Quantum Communication

Quantum communications quantum computers. A quantum network’s value comes from its ability to distribute entanglement. These communication devices manipulate the quantum properties of photons/particles of light to build Quantum Networks.

This market includes secure quantum key distribution, clock synchronization, random number generation and networking of quantum military sensors, computers, and other systems.

Quantum Cryptography/Quantum Key Distribution
Quantum Cryptography/Quantum Key Distribution can distribute keys between authorized partners connected by a quantum channel and a classical authenticated channel. It can be implemented via fiber optics or free space transmission. China transmitted entangled photons (at one pair of entangled particles per second) over 1,200 km in a satellite link, using the Micius satellite.

The Good: it can detect the presence of an eavesdropper, a feature not provided in standard cryptography. The Bad: Quantum Key Distribution can’t be implemented in software or as a service on a network and cannot be easily integrated into existing network equipment. It lacks flexibility for upgrades or security patches. Securing and validating Quantum Key Distribution is hard and it’s only one part of a cryptographic system.

The view from the National Security Agency (NSA) is that quantum-resistant (or post-quantum) cryptography is a more cost effective and easily maintained solution than quantum key distribution. NSA does not support the usage of QKD or QC to protect communications in National Security Systems. (See here.) They do not anticipate certifying or approving any Quantum Cryptography/Quantum Key Distribution security products for usage by National Security System customers unless these limitations are overcome. However, if you’re a commercial company these systems may be worth exploring.

Quantum Random Number Generators (GRGs)
Commercial Quantum Random Number Generators that use quantum effects (entanglement) to generate nondeterministic randomness are available today. (Government agencies can already make quality random numbers and don’t need these devices.)

Random number generators will remain secure even when a Cryptographically Relevant Quantum Computer is built.

Quantum Sensing and Metrology

Quantum sensors  Quantum computers.

This segment consists of Quantum Sensing (quantum magnetometers, gravimeters, …), Quantum Timing (precise time measurement and distribution), and Quantum Imaging (quantum radar, low-SNR imaging, …) Each of these areas can create entirely new commercial products or entire new industries e.g. new classes of medical devices and military systems, e.g. anti-submarine warfare, detecting stealth aircraft, finding hidden tunnels and weapons of mass destruction. Some of these are achievable in the near term.

Quantum Timing
First-generation quantum timing devices already exist as microwave atomic clocks. They are used in GPS satellites to triangulate accurate positioning. The Internet and computer networks use network time servers and the NTP protocol to receive the atomic clock time from either the GPS system or a radio transmission.

The next generation of quantum clocks are even more accurate and use laser-cooled single ions confined together in an electromagnetic ion trap. This increased accuracy is not only important for scientists attempting to measure dark matter and gravitational waves, but miniaturized/ more accurate atomic clocks will allow precision navigation in GPS- degraded/denied areas, e.g. in commercial and military aircraft, in tunnels and caves, etc.

Quantum Imaging
Quantum imaging is one of the most interesting and near-term applications. First generation magnetometers such as superconducting quantum interference devices (SQUIDs) already exist. New quantum sensor types of imaging devices use entangled light, accelerometers, magnetometers, electrometers, gravity sensors. These allow measurements of frequency, acceleration, rotation rates, electric and magnetic fields, photons, or temperature with levels of extreme sensitivity and accuracy.

These new sensors use a variety of quantum effects: electronic, magnetic, or vibrational states or spin qubits, neutral atoms, or trapped ions. Or they use quantum coherence to measure a physical quantity. Or use quantum entanglement to improve the sensitivity or precision of a measurement, beyond what is possible classically.

Quantum Imaging applications can have immediate uses in archeology,  and profound military applications. For example, submarine detection using quantum magnetometers or satellite gravimeters could make the ocean transparent. It would compromise the survivability of sea-based nuclear deterrent by detecting and tracking subs deep underwater.

Quantum sensors and quantum radar from companies like Rydberg can be game changers.

Gravimeters or quantum magnetometers could also detect concealed tunnels, bunkers, and nuclear materials. Magnetic resonance imaging could remotely ID chemical and biological agents. Quantum radar or LIDAR would enable extreme detection of electromagnetic emissions, enhancing ELINT and electronic warfare capabilities. It can use fewer emissions to get the same detection result, for better detection accuracy at the same power levels – even detecting stealth aircraft.

Finally, Ghost imaging uses the quantum properties of light to detect distant objects using very weak illumination beams that are difficult for the imaged target to detect. It can increase the accuracy and lessen the amount of radiation exposed to a patient during x-rays. It can see through smoke and clouds. Quantum illumination is similar to ghost imaging but could provide an even greater sensitivity.

National and Commercial Efforts
Countries across the world are making major investments ~$24 billion in 2021 – in quantum research and applications.

Lessons Learned

  • Quantum technologies are emerging and disruptive to companies and defense
  • Quantum technologies cover Quantum Computing, Quantum Communications and Quantum Sensing and Metrology
    • Quantum computing could obsolete existing cryptography systems
    • Quantum communication could allow secure cryptography key distribution and networking of quantum sensors and computers
    • Quantum sensors could make the ocean transparent for Anti-submarine warfare, create unjammable A2/AD, detect stealth aircraft, find hidden tunnels and weapons of mass destruction, etc.
  • A few of these technologies are available now, some in the next 5 years and a few are a decade or more out
  • Tens of billions of public and private capital dollars are being invested in them
  • Defense applications will come first
  • The largest commercial applications won’t be the ones we currently think they’re going to be
    • when they do show up they’ll destroy existing businesses and create new ones

The Semiconductor Ecosystem – Explained

The last year has seen a ton written about the semiconductor industry: chip shortages, the CHIPS Act, our dependence on Taiwan and TSMC, China, etc.

But despite all this talk about chips and semiconductors, few understand how the industry is structured. I’ve found the best way to understand something complicated is to diagram it out, step by step. So here’s a quick pictorial tutorial on how the industry works.


The Semiconductor Ecosystem

We’re seeing the digital transformation of everything. Semiconductors – chips that process digital information — are in almost everything: computers, cars, home appliances, medical equipment, etc. Semiconductor companies will sell $600 billion worth of chips this year.

Looking at the figure below, the industry seems pretty simple. Companies in the semiconductor ecosystem make chips (the triangle on the left) and sell them to companies and government agencies (on the right). Those companies and government agencies then design the chips into systems and devices (e.g. iPhones, PCs, airplanes, cloud computing, etc.), and sell them to consumers, businesses, and governments. The revenue of products that contain chips is worth tens of trillions of dollars.

Yet, given how large it is, the industry remains a mystery to most.  If you do think of the semiconductor industry at all, you may picture workers in bunny suits in a fab clean room (the chip factory) holding a 12” wafer. Yet it is a business that manipulates materials an atom at a time and its factories cost 10s of billions of dollars to build.  (By the way, that wafer has two trillion transistors on it.)

If you were able to look inside the simple triangle representing the semiconductor industry, instead of a single company making chips, you would find an industry with hundreds of companies, all dependent on each other. Taken as a whole it’s pretty overwhelming, so let’s describe one part of the ecosystem at a time.  (Warning –  this is a simplified view of a very complex industry.)

Semiconductor Industry Segments

The semiconductor industry has seven different types of companies. Each of these distinct industry segments feeds its resources up the value chain to the next until finally a chip factory (a “Fab”) has all the designs, equipment, and materials necessary to manufacture a chip. Taken from the bottom up these semiconductor industry segments are:

  1. Chip Intellectual Property (IP) Cores
  2. Electronic Design Automation (EDA) Tools
  3. Specialized Materials
  4. Wafer Fab Equipment (WFE)
  5. “Fabless” Chip Companies
  6. Integrated Device Manufacturers (IDMs)
  7. Chip Foundries
  8. Outsourced Semiconductor Assembly and Test (OSAT)

The following sections below provide more detail about each of these eight semiconductor industry segments.

Chip Intellectual Property (IP) Cores

  • The design of a chip may be owned by a single company, or…
  • Some companies license their chip designs – as software building blocks, called IP Cores – for wide use
  • There are over 150 companies that sell chip IP Cores
  • For example, Apple licenses IP Cores from ARM as a building block of their microprocessors in their iPhones and Computers

Electronic Design Automation (EDA) Tools

  • Engineers design chips (adding their own designs on top of any IP cores they’ve bought) using specialized Electronic Design Automation (EDA) software
  • The industry is dominated by three U.S. vendors – Cadence, Mentor (now part of Siemens) and Synopsys
  • It takes a large engineering team using these EDA tools 2-3 years to design a complex logic chip like a microprocessor used inside a phone, computer or server. (See the figure of the design process below.)

  • Today, as logic chips continue to become more complex, all Electronic Design Automation companies are beginning to insert Artificial Intelligence aids to automate and speed up the process

Specialized Materials and Chemicals

So far our chip is still in software. But to turn it into something tangible we’re going to have to physically produce it in a chip factory called a “fab.” The factories that make chips need to buy specialized materials and chemicals:

  • Silicon wafers – and to make those they need crystal growing furnaces
  • Over 100 Gases are used – bulk gases (oxygen, nitrogen, carbon dioxide, hydrogen, argon, helium), and other exotic/toxic gases (fluorine, nitrogen trifluoride, arsine, phosphine, boron trifluoride, diborane, silane, and the list goes on…)
  • Fluids (photoresists, top coats, CMP slurries)
  • Photomasks
  • Wafer handling equipment, dicing
  • RF Generators


Wafer Fab Equipment (WFE) Make the Chips

  • These machines physically manufacture the chips
  • Five companies dominate the industry – Applied Materials, KLA, LAM, Tokyo Electron and ASML
  • These are some of the most complicated (and expensive) machines on Earth. They take a slice of an ingot of silicon and manipulate its atoms on and below its surface
  • We’ll explain how these machines are used a bit later on

 “Fabless” Chip Companies

  • Systems companies (Apple, Qualcomm, Nvidia, Amazon, Facebook, etc.) that previously used off-the-shelf chips now design their own chips.
  • They create chip designs (using IP Cores and their own designs) and send the designs to “foundries” that have “fabs” that manufacture them
  • They may use the chips exclusively in their own devices e.g. Apple, Google, Amazon ….
  • Or they may sell the chips to everyone e.g. AMD, Nvidia, Qualcomm, Broadcom…
  • They do not own Wafer Fab Equipment or use specialized materials or chemicals
  • They do use Chip IP and Electronic Design Software to design the chips


Integrated Device Manufacturers (IDMs)

  • Integrated Device Manufacturers (IDMs) design, manufacture (in their own fabs), and sell their own chips
    • They do not make chips for other companies (this is changing rapidly – see here.)
    • There are three categories of IDMs– Memory (e.g. Micron, SK Hynix), Logic (e.g. Intel), Analog (TI, Analog Devices)
  • They have their own “fabs” but may also use foundries
    • They use Chip IP and Electronic Design Software to design their chips
    • They buy Wafer Fab Equipment and use specialized materials and chemicals
  • The average cost of taping out a new leading-edge chip (3nm) is now $500 million

 Chip Foundries

  • Foundries make chips for others in their “fabs”
  • They buy and integrate equipment from a variety of manufacturers
    • Wafer Fab Equipment and specialized materials and chemicals
  • They design unique processes using this equipment to make the chips
  • But they don’t design chips
  • TSMC in Taiwan is the leader in logic, Samsung is second
  • Other fabs specialize in making chips for analog, power, rf, displays, secure military, etc.
  • It costs $20 billon to build a new generation chip (3nm) fabrication plant

Fabs

  • Fabs are short for fabrication plants – the factory that makes chips
  • Integrated Device Manufacturers (IDMs) and Foundries both have fabs. The only difference is whether they make chips for others to use or sell or make them for themselves to sell.
  • Think of a Fab as analogous to a book printing plant (see figure below)
  1. Just as an author writes a book using a word processor, an engineer designs a chip using electronic design automation tools
  2. An author contracts with a publisher who specializes in their genre and then sends the text to a printing plant. An engineer selects a fab appropriate for their type of chip (memory, logic, RF, analog)
  3. The printing plant buys paper and ink. A fab buys raw materials; silicon, chemicals, gases
  4. The printing plant buys printing machinery, presses, binders, trimmers. The fab buys wafer fab equipment, etchers, deposition, lithography, testers, packaging
  5. The printing process for a book uses offset lithography, filming, stripping, blueprints, plate making, binding and trimming. Chips are manufactured in a complicated process manipulating atoms using etchers, deposition, lithography. Think of it as an atomic level offset printing. The wafers are then cut up and the chips are packaged
  6. The plant turns out millions of copies of the same book. The plant turns out millions of copies of the same chip

While this sounds simple, it’s not. Chips are probably the most complicated products ever manufactured.  The diagram below is a simplified version of the 1000+ steps it takes to make a chip.

Outsourced Semiconductor Assembly and Test (OSAT)

  • Companies that package and test chips made by foundries and IDMs
  • OSAT companies take the wafer made by foundries, dice (cut) them up into individual chips, test them and then package them and ship them to the customer

 

Fab Issues

  • As chips have become denser (with trillions of transistors on a single wafer) the cost of building fabs have skyrocketed – now >$10 billion for one chip factory
  • One reason is that the cost of the equipment needed to make the chips has skyrocketed
    • Just one advanced lithography machine from ASML, a Dutch company, costs $150 million
    • There are ~500+ machines in a fab (not all as expensive as ASML)
    • The fab building is incredibly complex. The clean room where the chips are made is just the tip of the iceberg of a complex set of plumbing feeding gases, power, liquids all at the right time and temperature into the wafer fab equipment
  • The multi-billion-dollar cost of staying at the leading edge has meant most companies have dropped out. In 2001 there were 17 companies making the most advanced chips.  Today there are only two – Samsung in Korea and TSMC in Taiwan.
    • Given that China believes Taiwan is a province of China this could be problematic for the West.

What’s Next – Technology

It’s getting much harder to build chips that are denser, faster, and use less power, so what’s next?

  • Instead of making a single processor do all the work, logic chip designers have put multiple specialized processors inside of a chip
  • Memory chips are now made denser by stacking them 100+ layers high
  • As chips are getting more complex to design, which means larger design teams, and longer time to market, Electronic Design Automation companies are embedding artificial intelligence to automate parts of the design process
  • Wafer equipment manufacturers are designing new equipment to help fabs make chips with lower power, better performance, optimum area-to-cost, and faster time to market

What’s Next – Business

The business model of Integrated Device Manufacturers (IDMs) like Intel is rapidly changing. In the past there was a huge competitive advantage in being vertically integrated i.e. having your own design tools and fabs. Today, it’s a disadvantage.

  • Foundries have economies of scale and standardization. Rather than having to invent it all themselves, they can utilize the entire stack of innovation in the ecosystem. And just focus on manufacturing
  • AMD has proven that it’s possible to shift from an IDM to a fabless foundry model. Intel is trying. They are going to use TSMC as a foundry for their own chips as well as set up their own foundry

What’s Next – Geopolitics

Controlling advanced chip manufacturing in the 21st century may well prove to be like controlling the oil supply in the 20th. The country that controls this manufacturing can throttle the military and economic power of others.

  • Ensuring a steady supply of chips has become a national priority. (China’s largest import by $’s are semiconductors – larger than oil)
  • Today, both the U.S. and China are rapidly trying to decouple their semiconductor ecosystems from each other; China is pouring $100+ billion of government incentives in building Chinese fabs, while simultaneously trying to create indigenous supplies of wafer fab equipment and electronic design automation software
  • Over the last few decades the U.S. moved most of its fabs to Asia. Today we are incentivizing bringing fabs and chip production back to the U.S.

An industry that previously was only of interest to technologists is now one of the largest pieces in great power competition.

Driven to Distraction – the future of car safety

If you haven’t gotten a new car in a while you may not have noticed that the future of the dashboard looks like this:


That’s it. A single screen replacing all the dashboard gauges, knobs and switches. But behind that screen is an increasing level of automation that hides a ton of complexity.

At times everything you need is on the screen with a glance. At other times you have to page through menus and poke at the screen while driving. And while driving at 70mph, try to understand if you or your automated driving system is in control of your car. All while figuring out how to use any of the new features, menus or rearranged user interface that might have been updated overnight.

In the beginning of any technology revolution the technology gets ahead of the institutions designed to measure and regulate safety and standards. Both the vehicle’s designers and regulators will eventually catch up, but in the meantime we’re on the steep part of a learning curve – part of a million-person beta test – about what’s the right driver-to-vehicle interface.

We went through this with airplanes. And we’re reliving that transition in cars. Things will break, but in a few decades we’ll come out out the other side, look back and wonder how people ever drove any other way.

Here’s how we got here, what it’s going to cost us, and where we’ll end up.


Cars, Computers and Safety
Two massive changes are occurring in automobiles: 1) the transition from internal combustion engines to electric, and 2) the introduction of automated driving.

But a third equally important change that’s also underway is the (r)evolution of car dashboards from dials and buttons to computer screens. For the first 100 years cars were essentially a mechanical platform – an internal combustion engine and transmission with seats – controlled by mechanical steering, accelerator and brakes. Instrumentation to monitor the car was made up of dials and gauges; a speedometer, tachometer, and fuel, water and battery gauges.
By the 1970’s driving became easier as automatic transmissions replaced manual gear shifting and hydraulically assisted steering and brakes became standard. Comfort features evolved as well: climate control – first heat, later air-conditioning; and entertainment – AM radio, FM radio, 8-track tape, CD’s, and today streaming media. In the last decade GPS-driven navigation systems began to appear.

Safety
At the same time cars were improving, automobile companies fought safety improvements tooth and nail. By the 1970’s auto deaths in the U.S averaged 50,000 a year. Over 3.7 million people have died in cars in the U.S. since they appeared – more than all U.S. war deaths combined. (This puts auto companies in the rarified class of companies – along with tobacco companies – that have killed millions of their own customers.) Car companies argued that talking safety would scare off customers, or that the added cost of safety features would put them in a competitive price disadvantage. But in reality, style was valued over safety.

Safety systems in automobiles have gone through three generations – passive systems and two generations of active systems. Today we’re about to enter a fourth generation – autonomous systems.

Passive safety systems are features that protect the occupants after a crash has occurred. They started appearing in cars in the 1930’s. Safety glass in windshields appeared in the 1930’s in response to horrific disfiguring crashes. Padded dashboards were added in the 1950’s but it took Ralph Nader’s book, Unsafe at Any Speedto spur federally mandated passive safety features in the U.S. beginning in the 1960’s: seat belts, crumple zones, collapsible steering wheels, four-way flashers and even better windshields. The Department of Transportation was created in 1966 but it wasn’t until 1979 that the National Highway Traffic Safety Administration (NHTSA) started crash-testing cars (the Insurance Institute for Highway Safety started their testing in 1995). In 1984 New York State mandated seat belt use (now required in 49 of the 50 states.)

These passive safety features started to pay off in the mid-1970’s as overall auto deaths in the U.S. began to decline.

Active safety systems try to prevent crashes before they happen. These depended on the invention of low-cost, automotive-grade computers and sensors. For example, accelerometers-on-a-chip made airbags possible as they were able to detect a crash in progress. These began to appear in cars in the late 1980’s/1990’s and were required in 1998. In the 1990’s computers capable of real-time analysis of wheel sensors (position and slip) made ABS (anti-lock braking systems) possible. This feature was finally required in 2013.

Since 2005 a second generation of active safety features have appeared. They run in the background and constantly monitor the vehicle and space around it for potential hazards. They include: Electronic Stability Control, Blind Spot Detection, Forward Collision Warning, Lane Departure Warning, Rearview Video Systems, Automatic Emergency Braking, Pedestrian Automatic Emergency Braking, Rear Automatic Emergency Braking, Rear Cross Traffic Alert and Lane Centering Assist.

Autonomous Cars
Today, a fourth wave of safety features is appearing as Autonomous/Self-Driving features. These include Lane Centering/Auto Steer, Adaptive cruise control, Traffic jam assist, Self-parking, full self-driving. The National Highway Traffic Safety Administration (NHTSA) has adopted the six-level SAE standard to describe these vehicle automation features:

Getting above Level 2 is a really hard technical problem and has been discussed ad infinitum in other places. But what hasn’t got much attention is how drivers interact with these systems as the level of automation increases, and as the driving role shifts from the driver to the vehicle. Today, we don’t know whether there are times these features make cars less safe rather than more.

For example, Tesla and other cars have Level 2 and some Level 3 auto-driving features. Under Level 2 automation, drivers are supposed to monitor the automated driving because the system can hand back control of the car to you with little or no warning. In Level 3 automation drivers are not expected to monitor the environment, but again they are expected to be prepared to take control of the vehicle at all times, this time with notice.

Research suggests that drivers, when they aren’t actively controlling the vehicle, may be reading their phone, eating, looking at the scenery, etc. We really don’t know how drivers will perform in Level 2 and 3 automation. Drivers can lose situational awareness when they’re surprised by the behavior of the automation – asking: What is it doing now? Why did it do that? Or, what is it going to do next? There are open questions as to whether drivers can attain/sustain sufficient attention to take control before they hit something. (Trust me, at highway speeds having a “take over immediately” symbol pop up while you are gazing at the scenery raises your blood pressure, and hopefully your reaction time.)If these technical challenges weren’t enough for drivers to manage, these autonomous driving features are appearing at the same time that car dashboards are becoming computer displays.

We never had cars that worked like this. Not only will users have to get used to dashboards that are now computer displays, they are going to have understand the subtle differences between automated and semi-automated features and do so as auto makers are developing and constantly updating them. They may not have much help mastering the changes. Most users don’t read the manual, and, in some cars, the manuals aren’t even keeping up with the new features.

But while we never had cars that worked like this, we already have planes that do.
Let’s see what we’ve learned in 100 years of designing controls and automation for aircraft cockpits and pilots, and what it might mean for cars.

Aircraft Cockpits
Airplanes have gone through multiple generations of aircraft and cockpit automation. But unlike cars which are just first seeing automated systems, automation was first introduced in airplanes during the 1920s and 1930s.

For their first 35 years airplane cockpits, much like early car dashboards, were simple – a few mechanical instruments for speed, altitude, relative heading and fuel. By the late 1930’s the British Royal Air Force (RAF) standardized on a set of flight instruments. Over the next decade this evolved into the “Basic T” instrument layout – the de facto standard of how aircraft flight instruments were laid out.

Engine instruments were added to measure the health of the aircraft engines – fuel and oil quantity, pressure, and temperature and engine speed.

Next, as airplanes became bigger, and the aerodynamic forces increased, it became difficult to manually move the control surfaces so pneumatic or hydraulic motors were added to increase the pilots’ physical force. Mechanical devices like yaw dampers and Mach trim compensators corrected the behavior of the plane.

Over time, navigation instruments were added to cockpits. At first, they were simple autopilots to just keep the plane straight and level and on a compass course. The next addition was a radio receiver to pick up signals from navigation stations. This was so pilots could set the desired bearing to the ground station into a course deviation display, and the autopilot would fly the displayed course.

In the 1960s, electrical systems began to replace the mechanical systems:

  • electric gyroscopes (INS) and autopilots using VOR (Very High Frequency Omni-directional Range) radio beacons to follow a track
  • auto-throttle – to manage engine power in order to maintain a selected speed
  • flight director displays – to show pilots how to fly the aircraft to achieve a preselected speed and flight path
  • weather radars – to see and avoid storms
  • Instrument Landing Systems – to help automate landings by giving the aircraft horizontal and vertical guidance.

By 1960 a modern jet cockpit (the Boeing 707) looked like this:

While it might look complicated, each of the aircraft instruments displayed a single piece of data. Switches and knobs were all electromechanical.

Enter the Glass Cockpit and Autonomous Flying
Fast forward to today and the third generation of aircraft automation. Today’s aircraft might look similar from the outside but on the inside four things are radically different:

  1. The clutter of instruments in the cockpit has been replaced with color displays creating a “glass cockpit”
  2. The airplanes engines got their own dedicated computer systems – FADEC (Full Authority Digital Engine Control) – to autonomously control the engines
  3. The engines themselves are an order of magnitude more reliable
  4. Navigation systems have turned into full-blown autonomous flight management systems

So today a modern airplane cockpit (an Airbus 320) looks like this:

Today, airplane navigation is a real-world example of autonomous driving – in the sky. Two additional systems, the Terrain Awareness and Warning Systems (TAWS) and Traffic Condition Avoidance System (TCAS) gave pilots a view of what’s underneath and around them dramatically increasing pilots’ situation awareness and flight safety. (Autonomy in the air is technically a much simpler problem because in the cruise portion of flight there are a lot less things to worry about in the air than in a car.)

Navigation in planes has turned into autonomous “flight management.” Instead of a course deviation dial, navigation information is now presented as a “moving map” on a display showing the position of navigation waypoints, by latitude and longitude. The position of the airplane no longer uses ground radio stations, but rather is determined by Global Positioning System (GPS) satellites or autonomous inertial reference units. The route of flight is pre-programmed by the pilot (or uploaded automatically) and the pilot can connect the autopilot to autonomously fly the displayed route. Pilots enter navigation data into the Flight Management System, with a keyboard. The flight management system also automates vertical and lateral navigation, fuel and balance optimization, throttle settings, critical speed calculation and execution of take-offs and landings.

Automating the airplane cockpit relieved pilots from repetitive tasks and allowed less skilled pilots to fly safely. Commercial airline safety dramatically increased as the commercial jet airline fleet quadrupled in size from ~5,000 in 1980 to over 20,000 today. (Most passengers today would be surprised to find out how much of their flight was flown by the autopilot versus the pilot.)

Why Cars Are Like Airplanes
And here lies the connection between what’s happened to airplanes with what is about to happen to cars.

The downside of glass cockpits and cockpit automation means that pilots no longer actively operating the aircraft but instead monitor it. And humans are particularly poor at monitoring for long periods. Pilots have lost basic manual and cognitive flying skills because of a lack of practice and feel for the aircraft. In addition, the need to “manage” the automation, particularly when involving data entry or retrieval through a key-pad, increased rather than decreased the pilot workload. And when systems fail, poorly designed user interfaces reduce a pilot’s situational awareness and can create cognitive overload.

Today, pilot errors — not mechanical failures– cause at least 70-80% of commercial airplane accidents. The FAA and NTSB have been analyzing crashes and have been writing extensively on how flight deck automation is affecting pilots. (Crashes like Asiana 214 happened when pilots selected the wrong mode on a computer screen.) The FAA has written the definitive document how people and automated systems ought to interact.

In the meantime, the National Highway Traffic Safety Administration (NHTSA) has found that 94% of car crashes are due to human error – bad choices drivers make such as inattention, distraction, driving too fast, poor judgment/performance, drunk driving, lack of sleep.

NHTSA has begun to investigate how people will interact with both displays and automation in cars. They’re beginning to figure out:

  • What’s the right way to design a driver-to-vehicle interface on a screen to show:
    • Vehicle status gauges and knobs (speedometer, fuel/range, time, climate control)
    • Navigation maps and controls
    • Media/entertainment systems
  • How do you design for situation awareness?
    • What’s the best driver-to-vehicle interface to display the state of vehicle automation and Autonomous/Self-Driving features?
    • How do you manage the information available to understand what’s currently happening and project what will happen next?
  • What’s the right level of cognitive load when designing interfaces for decisions that have to be made in milliseconds?
    • What’s the distraction level from mobile devices? For example, how does your car handle your phone? Is it integrated into the system or do you have to fumble to use it?
  • How do you design a user interface for millions of users whose age may span from 16-90; with different eyesight, reaction time, and ability to learn new screen layouts and features?

Some of their findings are in the document Human-centric design guidance for driver-vehicle interfaces. But what’s striking is that very little of the NHSTA documents reference the decades of expensive lessons that the aircraft industry has learned. Glass cockpits and aircraft autonomy have traveled this road before. Even though aviation safety lessons have to be tuned to the different reaction times needed in cars (airplanes fly 10 times faster, yet pilots often have seconds or minutes to respond to problems, while in a car the decisions often have to be made in milliseconds) there’s a lot they can learn together. Aviation has gone 9 years in the U.S. with just one fatality, yet in 2017 37,000 people died in car crashes in the U.S.

There Are No Safety Ratings for Your Car As You Drive
In the U.S. aircraft safety has been proactive. Since 1927 new types aircraft (and each sub-assembly) are required to get a type approval from the FAA before it can be sold and be issued an Airworthiness Certificate.

Unlike aircraft, car safety in the U.S. has been reactive. New models don’t require a type approval, instead each car company self-certifies that their car meets federal safety standards. NHTSA waits until a defect has emerged and then can issue a recall.

If you want to know how safe your model of car will be during a crash, you can look at the National Highway Traffic Safety Administration (NHTSA) New Car Assessment Program (NCAP) crash-tests, or the Insurance Institute for Highway Safety (IIHS) safety ratings. Both summarize how well the active and passive safety systems will perform in frontal, side, and rollover crashes. But today, there are no equivalent ratings for how safe cars are while you’re driving them. What is considered a good vs. bad user interface and do they have different crash rates? Does the transition from Level 1, 2 and 3 autonomy confuse drivers to the point of causing crashes? How do you measure and test these systems? What’s the role of regulators in doing so?

Given the NHTSA and the FAA are both in the Department of Transportation (DoT), It makes you wonder whether these government agencies actively talk to and collaborate with each other and have integrated programs and common best practices. And whether they have extracted best practices from the NTSB. And from the early efforts of Tesla, Audi, Volvo, BMW, etc., it’s not clear they’ve looked at the airplane lessons either.

It seems like the logical thing for NHTSA to do during this autonomous transition is 1) start defining “best practices” in U/I and automation safety interfaces and 2) to test Level 2-4 cars for safety while you drive (like the crash tests but for situational awareness, cognitive load, etc. in a set of driving scenarios. (There are great university programs already doing that research.)

However, the DoT’s Automated Vehicles 3.0 plan moves the agency further from owning the role of “best practices” in U/I and automation safety interfaces. It assumes that car companies will do a good job self-certifying these new technologies. And has no plans for safety testing and rating these new Level 2-4 autonomous features.

(Keep in mind that publishing best practices and testing for autonomous safety features is not the same as imposing regulations to slow down innovation.)

It looks like it might take an independent agency like the SAE to propose some best practices and ratings. (Or the slim possibility that the auto industry comes together and set defacto standards.)

The Chaotic Transition
It took 30 years, from 1900 to 1930, to transition from horses and buggies in city streets to automobiles dominating traffic. During that time former buggy drivers had to learn a completely new set of rules to control their cars. And the roads in those 30 years were a mix of traffic – it was chaotic.
In New York City the tipping point was 1908 when the number of cars passed the number of horses. The last horse-drawn trolley left the streets of New York in 1917. (It took another decade or two to displace the horse from farms, public transport and wagon delivery systems.) Today, we’re about to undergo the same transition.

Cars are on the path for full autonomy, but we’re seeing two different approaches on how to achieve Level 4 and 5 “hands off” driverless cars. Existing car manufacturers, locked into the existing car designs, are approaching this step-wise – adding additional levels of autonomy over time – with new models or updates; while new car startups (Waymo, Zoox, Cruise, etc.) are attempting to go right to Level 4 and 5.

We’re going to have 20 or so years with the roads full of a mix of millions of cars – some being manually driven, some with Level 2 and 3 driver assistance features, and others autonomous vehicles with “hands-off” Level 4 and 5 autonomy. It may take at least 20 years before autonomous vehicles become the dominant platforms. In the meantime, this mix of traffic is going to be chaotic. (Some suggest that during this transition we require autonomous vehicles to have signs in their rear window, like student drivers, but this time saying, “Caution AI on board.”)

As there will be no government best practices for U/I or scores for autonomy safety, learning and discovery will be happening on the road. That makes the ability for car companies to have over-the-air updates for both the dashboard user interface and the automated driving features essential. Incremental and iterative updates will add new features, while fixing bad ones. Engaging customers to make them realize they’re part of the journey will ultimately make this a successful experiment.

My bet is much like when airplanes went to glass cockpits with increasingly automated systems, we’ll create new ways drivers crash their cars, while ultimately increasing overall vehicle safety.

But in the next decade or two, with the government telling car companies “roll your own”, it’s going to be one heck of a ride.

Lessons Learned

  • There’s a (r)evolution as car dashboards move from dials and buttons to computer screens and the introduction of automated driving
    • Computer screens and autonomy will both create new problems for drivers
    • There are no standards to measure the safety of these systems
    • There are no standards for how information is presented
  • Aircraft cockpits are 10 to 20 years ahead of car companies in studying and solving this problem
    • Car and aircraft regulators need to share their learnings
    • Car companies can reduce crashes and deaths if they look to aircraft cockpit design for car user interface lessons
  • The Department of Transportation has removed barriers to the rapid adoption of autonomous vehicles
    • Car companies “self-certify” whether their U/I and autonomy are safe
    • There are no equivalents of crash safety scores for driving safety with autonomous features
  • Over-the-air updates for car software will become essential
    • But the downside is they could dramatically change the U/I without warning
  • On the path for full autonomy we’ll have three generations of cars on the road
    • The transition will be chaotic, so hang on it’s going to a bumpy ride, but the destination – safety for everyone on the road – will be worth it

The Apple Watch – Tipping Point Time for Healthcare

I don’t own an Apple Watch. I do have a Fitbit. But the Apple Watch 4 announcement intrigued me in a way no other product has since the original IPhone. This wasn’t just another product announcement from Apple. It heralded the U.S. Food and Drug Administration’s (FDA) entrance into the 21stcentury. It is a harbinger of the future of healthcare and how the FDA approaches innovation.

Sooner than people think, virtually all home and outpatient diagnostics will be performed by consumer devices such as the Apple Watch, mobile phones, fitness trackers, etc. that have either become FDA cleared as medical devices or have apps that have received FDA clearance. Consumer devices will morph into medical grade devices, with some painful and well publicized mistakes along the way.

Let’s see how it turns out for Apple.


Smartwatches are the apex of the most sophisticated electronics on the planet. And the Apple Watch is the most complex of them all. Packed inside a 40mm wide, 10 mm deep package is a 64-bit computer, 16gbytes of memory, Wi-Fi, NFC, cellular, Bluetooth, GPS, accelerometer, altimeter, gyroscope, heart rate sensor, and an ECG sensor – displaying it all on a 448 by 368 OLED display.
When I was a kid, this was science fiction.  Heck, up until its first shipment in 2015, it was science fiction.

But as impressive as its technology is, the Apple’s smartwatch has been a product looking for a solution. At first, positioned as a fashion statement, it seemed like the watch was actually an excuse to sell expensive wristbands. Subsequent versions focused on fitness and sports – the watch was like a Fitbit– plus the ability to be annoyed by interruptions from your work. But now the fourth version of the Watch might have just found the beginnings of “gotta have it” killer applications – healthcare – specifically medical diagnostics and screening.

Healthcare on Your Wrist
Large tech companies like Google, Amazon, Apple recognize that the  multi-trillion dollar health care market is ripe for disruption and have poured billions of dollars into the space. Google has been investing in a broad healthcare portfolio, Amazon has been investing in pharmacy distribution and Apple…? Apple has been focused on turning the Apple Watch into the future of health screening and diagnostics.

Apples latest Watch – with three new healthcare diagnostics and screening apps – gives us a glimpse into what the future of healthcare diagnostics and screening could look like.

The first new healthcare app on the Watch is Fall Detection. Perhaps you’ve seen the old commercials where someone falls and can’t get up, and has a device that calls for help. Well this is it – built into the watch. The watch’s built-in accelerometer and gyroscope analyze your wrist trajectory and impact acceleration to figure out if you’ve taken a hard fall. You can dismiss the alert, or have it call 911. Or, if you haven’t moved after a minute, it can call emergency services, and send a message along with your location.

If you’re in Apple’s current demographic you might think, “Who cares?” But if you have an aged parent, you might start thinking, “How can I get them to wear this watch?”

The second new healthcare app also uses the existing optical sensor in the watch and running in the background, gathers heart data and has an algorithm that can detect irregular heart rhythms. If it senses something is not right, up pops up an alert. A serious and common type of irregular heart rhythm is atrial fibrillation (AFib). AFib happens when the atria—the top two chambers of the heart get out of sync, and instead of beating at a normal 60 beats a minute it may quiver at 300 beats per minute.

This rapid heartbeat allows blood to pool in the heart, which can cause clots to form and travel to the brain, causing a stroke. Between 2.7 and 6.1 million people in the US have AFib (2% of people under 65 have it, while 9% of people over 65 years have it.) It puts ~750,000 people a year in the hospital and contributes to ~130,000 deaths each year. But if you catch atrial fibrillation early, there’s an effective treatment — blood thinners.

If your watch gives you an irregular heart rhythm alert you can run the third new healthcare app – the Electrocardiogram.

The Electrocardiogram (ECG or EKG) is a visual presentation of whether your heart is working correctly. It records the electrical activity of the heart and shows doctors the rhythm of heartbeats, the size and position of the chambers of the heart, and any damage to the heart’s muscle. Today, ECGs are done in a doctor’s office by having you lie down, and sticking 10 electrodes to your arms, legs and chest. The heart’s electrical signals are then measured from twelve angles (called “leads”).

With the Apple Watch, you can take an ECG by just putting your finger on the crown for 30 seconds. To make this work Apple has added two electrodes (the equivalent of a single lead), one on the back of the watch and another on the crown. The ECG can tell you that you may have atrial fibrillation (AFib) and suggest you see a doctor. As the ECG is saved in a PDF file (surprisingly it’s not also in the HL7’s FHIR Format), you can send it to your doctor, who may decide no visit is necessary.

These two apps, the Electrocardiogram and the irregular heart rhythms, are serious health screening tools. They are supposed to ship in the U.S. by the end of 2018. By the end of next year, they can be on the wrists of tens of millions of people.

The question is are they are going to create millions of unnecessary doctors’ visits from unnecessarily concerned users or are they going to save thousands of lives?  My bet is both – until traditional healthcare catches up with the fact that in the next decade screening devices will be in everyone’s hands (or wrists.)

Apple and The FDA – Clinical Trials
In the U.S. medical devices, drugs and diagnostics are regulated by the Food and Drug Administration – the FDA. What’s unique about the Apple Watch is that both the Electrocardiogram and the irregular heart rhythms apps required Apple to get clearance from the FDA. This is a very big deal.

The FDA requires evidence that medical devices do what they claim. To gather that evidence companies enroll volunteers in a study – called a clinical trial – to see if the device does what the company thinks it will.

Stanford University has been running a clinical trial on irregular heart rhythms for Apple since 2017 with a completion date in 2019. The goal is to see if an irregular pulse notification is really atrial fibrillation, and how many of those notified contacted a doctor within 90 days. (The Stanford study appears to be using previous versions of the Apple Watch with just the optical sensor and not the new ECG sensors. They used someone else’s wearable heart monitor to detect the Afib.)

Nov 1 2018 Update – the design of the Stanford Apple Watch study published here

To get FDA clearance, Apple reportedly submitted two studies to the FDA (so far none of the data has been published or peer reviewed). In one trial with 588 people, half of whom were known to have AFib and the other half of whom were healthy, the app couldn’t read 10% of the recordings. But for the other 90%, it was able to identify over 98% of the patients who had AFib, and over 99% of patients that had healthy heart rates.

The second data set Apple sent the FDA was part of Stanford’s Apple Heart Study. The app first identified 226 people with an irregular heart rhythm. The goal was to see how well the Apple Watch could pick up an event that looked like atrial fibrillation compared to a wearable heart monitor. The traditional monitors identified that 41 percent of people had an atrial fibrillation event. In 79 percent of those cases, the Apple app also picked something up.

This was good enough for the FDA.

The FDA – Running Hard to Keep Up With Disruption
And “good enough” is a big idea for the FDA. In the past the FDA was viewed as inflexible and dogmatic by new companies while viewed as insufficiently protective by watchdog organizations.

For the FDA this announcement was as important for them as it was for Apple.

The FDA has to adjudicate between a whole host of conflicting constituents and priorities. Its purpose is to make sure that drugs, devices, diagnostics, and software products don’t harm thousands or even millions of people so the FDA wants a process to make sure they get it right. This is a continual trade-off between patient safety, good enough data and decision making, and complete clinical proof. On the other hand, for a company, a FDA clearance can be worth hundreds of millions or even billions of dollars. And a disapproval or delayed clearance can put a startup out of business. Finally, the rate of change of innovation for medical devices, diagnostics and digital health has moved faster than the FDA’s ability to adapt its regulatory processes. Frustrated by the FDA’s 20th century processes for 21st century technology, companies hired lobbyists to force a change in the laws that guide the FDA regulations.

So, the Apple announcement is a visible signal in Washington that the FDA is encouraging innovation. In the last two years the FDA has been trying to prove it could keep up with the rapid advancements in digital health, devices and diagnostics- while trying to prevent another Theranos.

Since the appointment of the new head of the FDA, there has been very substantial progress in speeding up mobile and digital device clearances with new guidelines and policies. For example, in the last year the FDA announced its Pre-Cert pilot program which allows companies making software as a medical device to build products without each new device undergoing the FDA clearance process. The pilot program allowed nine companies, including Apple, to begin developing products (like the Watch) using this regulatory shortcut. (The FDA has also proposed new rules for clinical support software that say if doctors can review and understand the basis of the software’s decision, the tool does not have to be regulated by the FDA.)

This rapid clearance process as the standard – rather than the exception – is a sea-change for the FDA. It’s close to de-facto adopting a Lean decision-making process and rapid clearances for things that minimally affect health. It’s how China approaches approvals and will allow U.S. companies to remain competitive in an area (medical devices) where China has declared the intent to dominate.

Did Apple Cut in Front of the Line?
Some have complained that the FDA has been too cozy with Apple over this announcement.

Apple got its two FDA Class II clearances through what’s called a “de novo” pathway, meaning Apple claimed these features were the first of its kind. (It may be the first one built into the watch, but it’s not the first Apple Watch ECG app cleared by the FDA – AliveCor, got over-the-counter-clearance in 2014 and Cardiac Designs in 2013.) Critics said that the De Novo process should only be used where there is no predicate (substantial equivalence to an already cleared device.) But Apple cited at least one predicate, so if they followed the conventional 510k approval process, that should have taken at least 100 days. Yet Apple got two software clearances in under 30 days, which uncannily appeared the day before their product announcement.

To be fair to Apple, they were likely holding pre-submission meetings with the FDA for quite some time, perhaps years. One could speculate that using the FDA Pre-Cert pilot program they consulted on the design of the clinical trial, trial endpoints, conduct, inclusion and exclusion criteria, etc. This is all proper medical device company thinking and exactly how consumer device companies need to approach and work with the FDA to get devices or software cleared. And it’s exactly how the FDA should be envisioning its future.

Given Apple sells ~15 million Apple Watches a year, the company is about to embark on a public trial at massive scale of these features – with its initial patient population at the least risk for these conditions. It will be interesting to see what happens. Will overly concerned 20- and 30-year-olds flood doctors with false positives? Or will we be reading about lives saved?

Why most consumer hardware companies aren’t medical device and diagnostic companies
Historically consumer electronics companies and medical device and diagnostic companies were very different companies. In the U.S. medical device and diagnostic products require both regulatory clearance from the FDA and reimbursement approval by different private and public insurers to get paid for the products.

These regulatory and reimbursement agencies have very different timelines and priorities than for-profit companies. Therefore, to get FDA clearance a critical part of a medical device company is spent building a staff and hiring consultants such as clinical research organizations who can master and navigate FDA regulations and clinical trials.

And just because a company gets the FDA to clear their device/diagnostic/software doesn’t mean they’ll get paid for it. In the U.S. medical devices are reimbursed by private insurance companies (Blue Cross/Blue Shield, etc.) and/or the U.S. government via Centers for Medicare & Medicaid Services (CMS). Getting these clearances to get the product covered, coded and paid is as hard as getting the FDA clearance, often taking another 2-3 years. Mastering the reimbursement path requires a company to have yet another group of specialists conduct expensive clinical cost outcomes studies.

The Watch announcement telegraphed something interesting about Apple – they’re one of the few consumer products company to crack the FDA clearance process (Philips being the other). And going forward, unless these new apps are a disaster, it opens the door for them to add additional FDA-cleared screening and diagnostic tools to the watch (and by extension a host of AI-driven imaging diagnostics (melanoma detection, etc.) to the iPhone.) This by itself is a key differentiator for the Watch as a healthcare device.

The other interesting observation: Unlike other medical device companies, Apple’s current Watch business model is not dependent on getting insurers to pay for the watch. Today consumers pay directly for the Watch. However, if the Apple Watch becomes a device eligible for reimbursement, there’s a huge revenue upside for Apple. When and if that happens, your insurance would pay for all or part of an Apple Watch as a diagnostic tool.

(After running cost outcome studies, insurers believe that preventative measures like staying fit brings down their overall expense for a variety of conditions. So today some life insurance companies are mandating the use of an activity tracker like Apple Watch.)

The Future of SmartWatches in Healthcare
Very few companies (probably less than five) have the prowess to integrate sensors, silicon and software with FDA regulatory clearance into a small package like the Apple Watch.

So what else can/will Apple offer on the next versions of the Watch? After looking through Apple’s patents, here’s my take on the list of medical diagnostics and screening apps Apple may add.

Sleep Tracking and Sleep Apnea Detection
Compared to the Fitbit, the lack of a sleep tracking app on the Apple Watch is a mystery (though third-party sleep apps are available.) Its absence is surprising as the Watch can theoretically do much more than just sleep tracking – it can potentially detect Sleep Apnea. Sleep apnea happens when you’re sleeping, and your upper airway becomes blocked, reducing or completely stopping air to your lungs. This can cause a host of complications including Type 2 diabetes, high blood pressure, liver problems, snoring, daytime fatigueToday diagnosing sleep apnea often requires an overnight stay in a sleep study clinicSleep apnea screening doesn’t appear to require any new sensors and would be a great app for the Watch. Perhaps the app is missing because you have to take the watch off and recharge it every night?

Pulse oximetry
Pulse oximetry is a test used to measure the oxygen level (oxygen saturation) of the blood. The current Apple Watch can already determine how much oxygen is contained in your blood based on the amount of infrared light it absorbs. But for some reason Apple hasn’t released this feature – FDA regulations? Inconsistent readings?  Another essential Watch health app that may or may not require any new sensors.

Respiration rate
Respiration rate (the number of breaths a person takes per minute) along with blood pressure, heart rate and temperature make up a person’s vital signs. Apple has a patent for this watch feature but for some reason hasn’t released it – FDA regulations?  Inconsistent readings?  Another essential Watch health app that doesn’t appear to require any new sensors.

Blood Pressure
About 1/3rd of Americans have high blood pressure. High blood pressure increases the risk of heart disease and stroke. It often has no warning signs or symptoms. Many people do not know they have it and only half of those have it under control. Traditionally measuring blood pressure requires a cuff on the arm and produces a single measurement at a single point in time. We’ve never had the ability to continually monitor a person’s blood pressure under stress or sleep. Apple filed two patents in 2017 to measure blood pressure by holding the watch against your chest. This is tough to do, but it would be another great health app for the Watch that may or may not require any new sensors.

Sunburn/UV Detector
Apple has patented a new type of sensor – a sunscreen detector to let you know what exposed areas of the skin of may be at elevated UV exposure risk. I’m not big on this, but the use of ever more powerful sunscreens has quadrupled, while at the same time, the incidence of skin cancers has also quadrupled, so there may be a market here.

Parkinson’s Disease Diagnosis and Monitoring
Parkinson’s Disease is a brain disorder that leads to shaking, stiffness, and difficulty with walking, balance, and coordination. It affects 1/% of people over 60. Today, there is no diagnostic test for the disease (i.e. blood test, brain scan or EEG). Instead, doctors look for four signs: tremor, rigidity, Bradykinesia/akinesia and Postural instability. Today patients have to go to a doctor for tests to rate the severity of their symptoms and keep a diary of their symptoms.

Apple added a new “Movement Disorder API” to its ResearchKit framework that supports movement and tremor detection. It allows an Apple Watch to continuously monitor for Parkinson’s disease symptoms; tremors and Dyskinesia, a side-effect of treatments for Parkinson’s that causes fidgeting and swaying motions in patients. Researchers have built a prototype Parkinson’s detection app on top of it. It appears that screening for Parkinson’s would not require any new sensors – but likely clinical trials and FDA clearance – and would be a great app for the Watch.

Glucose Monitoring
More than 100 million U.S. adults live with diabetes or prediabetes. If you’re a diabetic, monitoring your blood glucose level is essential to controlling the disease. However, it requires sticking your finger to draw blood multiple times a day. The holy grail of glucose monitoring has been a sensor that can detect glucose levels through the skin. This sensor has been the graveyard of tons of startups that have crashed and burned pursuing this. Apple has a patent application that looks suspiciously like a non-invasive glucose monitoring sensor for the Apple Watch. This is a really tough technical problem to solve, and even if the sensor works, there would be a long period of clinical trials for FDA clearance, but this app would be a game changer for diabetic patients – and Apple – if they can make it happen.

Sensor and Data Challenges
With many of these sensors just getting a signal is easy. Correlating that particular signal to an underlying condition and avoiding being confounded by other factors is what makes achieving medical device claims so hard.

As medical grade data acquisition becomes possible, continuous or real time transmission will store and report baseline data on tens of millions of “healthies” that will be vital in training the algorithms and eventually predicting disease earlier. This will eventually enable more accurate diagnostics on less data, and make the data itself – especially the transition from healthy to diseased – incredibly valuable.

However, this sucks electrons out of batteries and plays on the edge electrical design and the laws of physics, but Apple’s prowess in this area is close to making this possible.

What’s Not Working?
Apple has attempted to get medical researchers to create new health apps by developing ResearchKit, an open source framework for researchers. Great idea. However, given the huge potential for the Watch in diagnostics, ResearchKit and the recruitment of Principal Investigators feels dramatically under resourced. (It took three years to go from ResearchKit 1.0 to 2.0).  Currently, there are just 11 ResearchKit apps on the ITunes store. This effort – Apple software development and third-party app development – feels understaffed and underfunded. Given the potential size of the opportunity, the rhetoric doesn’t match the results and the results to date feel off by at least 10x.

Apple needs to act more proactively and directly fund some of these projects with grants to specific principal investigators and build a program of scale. (Much like the NIH SBIR program.) There should be as sustained commitment to at least several new FDA cleared screening/diagnostic apps every year for Watch and iPhone from Apple.

The Future
Although the current demographics of the Apple Watch skews young, the populations of the U.S., China, Europe and Japan continue to age, which in turn threatens to overwhelm healthcare systems. Having an always on, real-time streaming of medical data to clinicians, will change the current “diagnosis on a single data point and by appointment” paradigm. Wearable healthcare diagnostics and screening apps open an entirely new segment for Apple and will change the shape of healthcare forever.

Imagine a future when you get an Apple Watch (or equivalent) through your insurer to monitor your health for early warning signs of heart attack, stroke, Parkinson’s disease and to help you monitor and manage diabetes, as well as reminding you about medications and tracking your exercise. And when combined with an advanced iPhone with additional FDA cleared screening apps for early detection of skin cancer, glaucoma, cataracts, and other diseases, the future of your health will truly be in your own hands.

Outside the U.S., China is plowing into this with government support, private and public funding, and a China FDA (CFDA) approval process that favors local Chinese solutions. There are well over 100 companies in China alone focusing in this area, many with substantial financial and technical support.

Let’s hope Apple piles on the missing resources for diagnostics and screening apps and grabs the opportunity.

Lessons Learned

  • Apple’s new Watch has two heart diagnostic apps cleared by the FDA
    • This is a big deal
  • In a few years, home and outpatient diagnostics will be performed by wearable consumer devices – Apple Watch, mobile phones or fitness trackers
    • Collecting and sending health data to doctors as needed
    • Collecting baseline data on tens of millions of healthy people to train disease prediction algorithms
  • In the U.S. the FDA has changed their mobile and digital device guidelines and policies to make this happen
  • Insurers will ultimately will be paying for diagnostic wearables
  • Apple has a series of patents for additional Apple Watch sensors – glucose monitoring, blood pressure, UV detection, respiration
    • The watch is already capable of detecting blood oxygen level, sleep apnea, Parkinson’s disease
    • Getting a signal from a sensor is the easy part. Correlating that signal to an underlying condition is hard
    • They need to step up their game – money, software, people – with the medical research community
  • China has made building a local device and diagnostic industry one of their critical national initiatives

The End of More – The Death of Moore’s Law

 A version of this article first appeared in IEEE Spectrum.

For most of our lives the idea that computers and technology would get, better, faster, cheaper every year was as assured as the sun rising every morning. The story “GlobalFoundries Stops All 7nm Development“ doesn’t sound like the end of that era, but for anyone who uses an electronic device, it most certainly is.

Technology innovation is going to take a different direction.


GlobalFoundries was one of the three companies that made the most advanced silicon chips for other companies (AMD, IBM, Broadcom, Qualcomm, STM and the Department of Defense.) The other foundries are Samsung in South Korea and TSMC in Taiwan. Now there are only two pursuing the leading edge.

This is a big deal.

Since the invention of the integrated circuit ~60 years ago, computer chip manufacturers have been able to pack more transistors onto a single piece of silicon every year. In 1965, Gordon Moore, one of the founders of Intel, observed that the number of transistors was doubling every 24 months and would continue to do so. For 40 years the chip industry managed to live up to that prediction. The first integrated circuits in 1960 had ~10 transistors. Today the most complex silicon chips have 10 billion. Think about it. Silicon chips can now hold a billion times more transistors.

But Moore’s Law ended a decade ago. Consumers just didn’t get the memo.

No More Moore – The End of Process Technology Innovation
Chips are actually “printed,” not with a printing press but with lithography, using exotic chemicals and materials in a “fab” (a chip fabrication plant – the factory where chips are produced). Packing more transistors in each generation of chips requires the fab to “shrink” the size of the transistors. The first transistors were printed with lines 80 microns wide. Today Samsung and TSMC are pushing to produce chips with features few dozen nanometers across.That’s about a 2,000-to-1 reduction.

Each new generation of chips that shrinks the line widths requires fabs to invest enormous amounts of money in new chip-making equipment.  While the first fabs cost a few million dollars, current fabs – the ones that push the bleeding edge – are over $10 billion.

And the exploding cost of the fab is not the only issue with packing more transistors on chips. Each shrink of chip line widths requires more complexity. Features have to be precisely placed on exact locations on each layer of a device. At 7 nanometers this requires up to 80 separate mask layers.

Moore’s Law was an observation about process technology and economics. For half a century it drove the aspirations of the semiconductor industry. But the other limitation to packing more transistors onto to a chip is a physical limitation called Dennard scaling– as transistors get smaller, their power density stays constant, so that the power use stays in proportion with area. This basic law of physics has created a “Power Wall” – a barrier to clock speed – that has limited microprocessor frequency to around 4 GHz since 2005. It’s why clock speeds on your microprocessor stopped increasing with leaps and bounds 13 years ago.  And why memory density is not going to increase at the rate we saw a decade ago.

This problem of continuing to shrink transistors is so hard that even Intel, the leader in microprocessors and for decades the gold standard in leading fab technology, has had problems. Industry observers have suggested that Intel has hit several speed bumps on the way to their next generation push to 10- and 7-nanometer designs and now is trailing TSMC and Samsung.

This combination of spiraling fab cost, technology barriers, power density limits and diminishing returns is the reason GlobalFoundries threw in the towel on further shrinking line widths . It also means the future direction of innovation on silicon is no longer predictable.

It’s the End of the Beginning
The end of putting more transistors on a single chip doesn’t mean the end of innovation in computers or mobile devices. (To be clear, 1) the bleeding edge will advance, but almost imperceptibly year-to-year and 2) GlobalFoundaries isn’t shutting down, they’re just no longer going to be the ones pushing the edge 3) existing fabs can make current generation 14nm chips and their expensive tools have been paid for. Even older fabs at 28-, 45-, and 65nm can make a ton of money).

But what it does mean is that we’re at the end of guaranteed year-to-year growth in computing power. The result is the end of the type of innovation we’ve been used to for the last 60 years. Instead of just faster versions of what we’ve been used to seeing, device designers now need to get more creative with the 10 billion transistors they have to work with.

It’s worth remembering that human brains have had 100 billion neurons for at least the last 35,000 years. Yet we’ve learned to do a lot more with the same compute power. The same will hold true with semiconductors – we’re going to figure out radically new ways to use those 10 billion transistors.

For example, there are new chip architectures coming (multi-core CPUs, massively parallel CPUs and special purpose silicon for AI/machine learning and GPU’s like Nvidia), new ways to package the chips and to interconnect memory, and even new types of memory. And other designs are pushing for extreme low power usage and others for very low cost.

It’s a Whole New Game
So, what does this mean for consumers? First, high performance applications that needed very fast computing locally on your device will continue their move to the cloud (where data centers are measured in football field sizes) further enabled by new 5G networks. Second, while computing devices we buy will not be much faster on today’s off-the-shelf software, new features– facial recognition, augmented reality, autonomous navigation, and apps we haven’t even thought about –are going to come from new software using new technology like new displays and sensors.

The world of computing is moving into new and uncharted territory. For desktop and mobile devices, the need for a “must have” upgrade won’t be for speed, but because there’s a new capability or app.

For chip manufacturers, for the first time in half a century, all rules are off. There will be a new set of winners and losers in this transition. It will be exciting to watch and see what emerges from the fog.

Lessons Learned

  • Moore’s Law – the doubling of every two years of how many transistors can fit on a chip – has ended
  • Innovation will continue in new computer architectures, chip packaging, interconnects, and memory
  • 5G networks will move more high-performance consumer computing needs seamlessly to the cloud
  • New applications and hardware other than CPU speed (5G networks, displays, sensors) will now drive sales of consumer devices
  • New winners and losers will emerge in consumer devices and chip suppliers

The Difference Between Innovators and Entrepreneurs

I just received a thank-you note from a student who attended a fireside chat I held at the ranch. Something I said seemed to inspire her:

“I always thought you needed to be innovative, original to be an entrepreneur. Now I have a different perception. Entrepreneurs are the ones that make things happen. (That) takes focus, diligence, discipline, flexibility and perseverance. They can take an innovative idea and make it impactful. … successful entrepreneurs are also ones who take challenges in stride, adapt and adjust plans to accommodate whatever problems do come up.”


Over the last decade I’ve watched hundreds of my engineering students as well as ~1,500 of the country’s best scientists in the National Science Foundation Innovation Corps, cycle through the latest trends in startups: social media, new materials, big data, medical devices, diagnostics, digital health, therapeutics, drones, robotics, bitcoin, machine learning, etc.  Some of these world-class innovators get recruited by large companies like professional athletes, with paychecks to match. Others join startups to strike out on their own. But what I’ve noticed is that it’s rare that the smartest technical innovator is the most successful entrepreneur.

Being a domain expert in a technology field rarely makes you competent in commerce. Building a company takes very different skills than building a neural net in Python or decentralized blockchain apps in Ethereum.

Nothing makes me happier than to see my students getting great grades (and as they can tell you, I make them very work hard for them). But I remind them that customers don’t ask for your transcript. Until we start giving grades for resiliency, curiosity, agility, resourcefulness, pattern recognition, tenacity and having a passion for products and customers, great grades and successful entrepreneurs have at best a zero correlation (and anecdotal evidence suggests that the correlation may actually be negative.)

Most great technology startups – Oracle, Microsoft, Apple, Amazon, Tesla – were built by a team led by an entrepreneur.

It doesn’t mean that if you have technical skills you can’t build a successful company. It does mean that success in building a company that scales depends on finding product/market fit, enough customers, enough financing, enough great employees, distribution channels, etc. These are entrepreneurial skills you need to rapidly acquire or find a co-founder who already has them.

Lessons Learned

  • Entrepreneurship is a calling, not a job.
  • A calling is something you feel you need to follow, it gives you direction and purpose but no guarantee of a paycheck.
  • It’s what allows you to create a missionary zeal to recruit others, get customers to buy into a vision and gets VC’s to finance a set of slides.
  • It’s what makes you get up and do it again when customers say no, when investors laugh at your idea or when your rocket fails to make it to space.

Tesla Lost $700 Million Last Year, So Why Is Tesla’s Valuation $60 Billion?

Automobile manufacturers shipped 88 million cars in 2016. Tesla shipped 76,000. Yet Wall Street values Tesla higher than any other U.S. car manufacturer. What explains this more than 1,000 to 1 discrepancy in valuation?

The future.

Too many people compare Tesla to what already exists and that’s a mistake. Tesla is not another car company.

At the turn of the 20th century most people compared existing buggy and carriage manufacturers to the new automobile companies. They were both transportation, and they looked vaguely similar, with the only apparent difference that one was moved by horses attached to the front while the other had an unreliable and very noisy internal combustion engine.

They were different. And one is now only found in museums. Companies with business models built around internal combustion engines disrupted those built around horses.  That’s the likely outcome for every one of today’s automobile manufacturers. Tesla is a new form of transportation disrupting the incumbents.

Here are four reasons why.

Electric cars pollute less, have fewer moving parts, are quieter and faster than existing cars. Today, the technology necessary (affordable batteries with sufficient range) for them to be a viable business have all just come together. Most observers agree that autonomous electric cars will be the dominate form of transportation by mid-century. That’s bad news for existing car companies.

First, car companies have over a century of expertise in designing and building efficient mechanical propulsion systems – internal combustion engines for motive power and transmissions to drive the wheels. If existing car manufacturers want to build electric vehicles, all those design skills and most of the supply chain and manufacturing expertise are useless. And not only useless but they become this legacy of capital equipment and headcount that is now a burden to a company. In a few years, the only thing useful in existing factories building traditional cars will be the walls and roof.

Second, while the automotive industry might be 1000 times larger than Tesla, Tesla may actually have more expertise and dollars committed to the electric car ecosystem than any legacy car company. Tesla’s investment in Lithium/Ion battery factory (the Gigafactory), its electric drive train design and manufacturing output exceed the sum of the entire automotive industry.

Third, the future of transportation is not only electric, it’s autonomous and connected. A lot has been written about self-driving cars and as a reminder, automated driving comes in multiple levels:

  • Level 0: the car gives you warnings but driver maintains control of the car. For example, blind spot warning.
  • Level 1: the driver and the car share control. For example, Adaptive Cruise Control (ACC) where the driver controls steering and the automated system controls speed.
  • Level 2: The automated system takes full control of the vehicle (accelerating, braking, and steering). The driver monitors and intervenes if the automated system fails to respond.
  • Level 3: The driver can text or watch a movie. The vehicle will handle situations that call for an immediate response, like emergency braking. The driver must be prepared to intervene within some limited time, when called upon by the vehicle.
  • Level 4: No driver attention is ever required for safety, i.e. the driver may safely go to sleep or leave the driver’s seat.
  • Level 5: No human intervention is required. For example, a robotic taxi

Each level of autonomy requires an exponential amount of software engineering design and innovation. While cars have had an ever-increasing amount of software content, the next generation of transportation are literally computers on wheels. Much like in electric vehicle drive trains, autonomy and connectivity are not core competencies of existing car companies.

Fourth, large, existing companies are executing a known business model and have built processes, procedures and key performance indicators to measure progress to a known set of goals. But when technology disruption happens (electric drive trains, autonomous vehicles, etc.) changing a business model is extremely difficult. Very few companies manage to make the transition from one business model to another.

And while Tesla might be the first mover in disrupting transportation there is no guarantee they will be the ultimate leader. However, the question shouldn’t be why Tesla has such a high valuation.

The question should be why the existing automobile companies aren’t valued like horse and buggy companies.

Lesson Learned

  • Few market leaders in an industry being disrupted make the transition to the new industry
  • The assets, expertise, and mindset that made them leaders in the past are usually the baggage that prevents them from seeing the future

Innovation, Change and the Rest of Your Life

I gave the Alumni Day talk at U.C. Santa Cruz and had a few things to say about innovation.

—-

Even though I live just up the coast, I’ve never had the opportunity to start a talk by saying “Go Banana Slugs.”

I’m honored for the opportunity to speak here today.

We’re standing 15 air miles away from the epicenter of technology innovation. The home of some of the most valuable and fastest growing companies in the world.

I’ve spent my life in innovation, eight startups in 21 years, and the last 15 years in academia teaching it.

I lived through the time when working in my first job in Ann Arbor Michigan we had to get out a map to find out that San Jose was not only in Puerto Rico but there was a city with that same name in California.  And that’s where my plane ticket ought to take me to install some computer equipment.

39 years ago I got on that plane and never went back.

I’ve seen the Valley grow from Sunnyvale to Santa Clara to today where it stretches from San Jose to South of Market in San Francisco.  I’ve watched the Valley go from Microwave Valley – to Defense Valley – to Silicon Valley to Internet Valley. And to today, when its major product is simply innovation.  And I’ve been lucky enough to watch innovation happen not only in hardware and software but in Life Sciences – in Therapeutics, Medical Devices, Diagnostics and now Digital Health.

I’ve been asked to talk today about the future of Innovation – typically that involves giving you a list of hot technologies to pay attention to – technologies like machine learning.  The applications that will pour of this just one technology will transform every industry – from autonomous vehicles to automated radiology/oncology diagnostics.

Equally transformative on the life science side, CRISPR and CAS enable rapid editing of the genome, and that will change life sciences as radically as machine intelligence.

But today’s talk about the future of innovation is not about these technologies, or the applications or the new industries they will spawn.

In fact, it’s not about any specific new technologies.

The future of innovation is really about seven changes that have made innovation itself possible in a way that never existed before.

We’ve created a world where innovation is not just each hot new technology, but a perpetual motion machine.

So how did this happen?  Where is it going?

Silicon Valley emerged by the serendipitous intersection of:

  • Cold War research in microwaves and electronics at Stanford University,
  • a Stanford Dean of Engineering who encouraged startup culture over pure academic research,
  • Cold War military and intelligence funding driving microwave and military products for the defense industry in the 1950’s,
  • a single Bell Labs researcher deciding to start his semiconductor company next to Stanford in the 1950’s which led to
  • the wave of semiconductor startups in the 1960’s/70’s,
  • the emergence of Venture Capital as a professional industry,
  • the personal computer revolution in 1980’s,
  • the rise of the Internet in the 1990’s and finally
  • the wave of internet commerce applications in the first decade of the 21st century.
  • The flood of risk capital into startups at a size and scale that was not only unimaginable at its start, but in the middle of the 20th century would have seemed laughable.

Up until the beginning of this century, the pattern for the Valley seemed to be clear. Each new wave of innovation – microwaves, defense, silicon, disk drives, PCs, Internet, therapeutics, – was like punctuated equilibrium – just when you thought the wave had run its course into stasis, there emerged a sudden shift and radical change into a new family of technology. 

But in the 20th Century there were barriers to Entrepreneurship
In the last century, while startups continued to innovate in each new wave of technology, the rate of innovation was constrained by limitations we only now can understand. Startups in the past were constrained by:

  1. customers were initially the government and large companies and they adopted technology slowly,
  2. long technology development cycles (how long it takes to get from idea to product),
  3. disposable founders,
  4. the high cost of getting to first customers (how many dollars to build the product),
  5. the structure of the Venture Capital industry (there were a limited number of VC firms each needing to invest millions per startups),
  6. the failure rate of new ventures (startups had no formal rules and acted like smaller versions of large companies),
  7. the information and expertise about how to build startups (information was clustered in specific regions like Silicon Valley, Boston, New York, etc.), and there were no books, blogs or YouTube videos about entrepreneurship.

What we’re now seeing is The Democratization of Entrepreneurship
What’s happening today is something more profound than a change in technology. What’s happening is that these seven limits to startups and innovation have been removed.

The first thing that’s changed is that Consumer Internet and Genomics are Driving Innovation at scale
In the 1950’s and ‘60’s U.S. Defense and Intelligence organizations drove the pace of innovation in Silicon Valley by providing research and development dollars to universities, and defense companies built weapons systems that used the Valley’s first microwave devices and semiconductor components.

In the 1970’s, 80’s and 90’s, momentum shifted to the enterprise as large businesses supported innovation in PCs, communications hardware and enterprise software. Government and the enterprise are now followers rather than leaders.

Today, for hardware and software it’s consumers – specifically consumer Internet companies – that are the drivers of innovation. When the product and channel are bits, adoption by 10’s and 100’s of millions and even billions of users can happen in years versus decades.

For life sciences it was the Genentech IPO in 1980 that proved to investors that life science startups could make them a ton of money.

The second thing that’s changed is that we’re now Compressing the Product Development Cycle
In the 20th century startups I was part of, the time to build a first product release was measured in years as we turned out the founder’s vision of what customers wanted. This meant building every possible feature the founding team envisioned into a monolithic “release” of the product.

Yet time after time, after the product shipped, startups would find that customers didn’t use or want most of the features. The founders were simply wrong about their assumptions about customer needs. It turns out the term “visionary founder” was usually a synonym for someone who was hallucinating. The effort that went into making all those unused features was wasted.

Today startups build products differently. Instead of building the maximum number of features, founders treat their vision as a series of untested hypotheses, then get out of the building and test a minimum feature set in the shortest period of time.  This lets them deliver a series of minimal viable products to customers in a fraction of the time.

For products that are simply “bits” delivered over the web, a first product can be shipped in weeks rather than years.

The third thing is that Founders Need to Run the Company Longer
Today, we take for granted new mobile apps and consumer devices appearing seemingly overnight, reaching tens of millions of users – and just as quickly falling out of favor. But in the 20th century, dominated by hardware, software, and life sciences, technology swings inside an existing market happened slowly — taking years, not months. And while new markets were created (i.e. the desktop PC market), they were relatively infrequent.

This meant that disposing of the founder, and the startup culture responsible for the initial innovation, didn’t hurt a company’s short-term or even mid-term prospects.  So, almost like clockwork 20th century startups fired the innovators/founders when they scaled. A company could go public on its initial wave of innovation, then coast on its current technology for years. In this business environment, hiring a new CEO who had experience growing a company around a single technical innovation was a rational decision for venture investors.

That’s no longer the case.

The pace of technology change in the second decade of the 21st century is relentless. It’s hard to think of a hardware/software or life science technology that dominates its space for years. That means new companies face continuous disruption before their investors can cash out.

To stay in business in the 21st century, startups must do three things their 20th century counterparts didn’t:

  • A company is no longer built on a single innovation. It needs to be continuously innovating – and who best to do that? The founders.
  • To continually innovate, companies need to operate at startup speed and cycle time much longer their 20th century counterparts did. This requires retaining a startup culture for years – and who best to do that? The founders.
  • Continuous innovation requires the imagination and courage to challenge the initial hypotheses of your current business model (channel, cost, customers, products, supply chain, etc.) This might mean competing with and if necessary killing your own products. (Think of the relentless cycle of iPod then iPhone innovation.) Professional CEOs who excel at growing existing businesses find this extremely hard.  Who best to do that? The founders.

The fourth thing that’s changed is that you can start a company on your laptop For Thousands Rather than Millions of Dollars
Startups traditionally required millions of dollars of funding just to get their first product to customers. A company developing software would have to buy computers and license software from other companies and hire the staff to run and maintain it. A hardware startup had to spend money building prototypes and equipping a factory to manufacture the product.

Today open source software has slashed the cost of software development from millions of dollars to thousands. My students think of computing power as a utility like I think of electricity. They can get to more computing power via their laptop through Amazon Web Services than existed in the entire world when I started in Silicon Valley.

And for consumer hardware, no startup has to build their own factory as the costs are absorbed by offshore manufacturers.  China has simply become the factory.

The cost of getting the first product out the door for an Internet commerce startup has dropped by a factor of a 100 or more in the last decade.  Ironically, while the cost of getting the first product out the door has plummeted, it now can take 10’s or 100’s of millions of dollars to scale.

The fifth change is the New Structure of how startups get funded
The plummeting cost of getting a first product to market (particularly for Internet startups) has shaken up the Venture Capital industry.

Venture Capital used to be a tight club clustered around formal firms located in Silicon Valley, Boston, and New York. While those firms are still there (and getting larger), the pool of money that invests risk capital in startups has expanded, and a new class of investors has emerged.

First, Venture Capital and angel investing is no longer a U.S. or Euro-centric phenomenon. Risk capital has emerged in China, India and other countries where risk taking, innovation and liquidity are encouraged, on a scale previously only seen in the U.S.

Second, new groups of VCs, super angels, smaller than the traditional multi-hundred-million-dollar VC fund, can make small investments necessary to get a consumer Internet startup launched. These angels make lots of early bets and double-down when early results appear. (And the results do appear years earlier than in a traditional startup.)

Third, venture capital has now become Founder-friendly.

A 20th century VC was likely to have an MBA or finance background. A few, like John Doerr at Kleiner Perkins and Don Valentine at Sequoia, had operating experience in a large tech company. But out of the dot-com rubble at the turn of the 21st century, new VCs entered the game – this time with startup experience. The watershed moment was in 2009 when the co-founder of Netscape, Marc Andreessen, formed a venture firm and started to invest in founders with the goal to teach them how to be CEOs for the long term. Andreessen realized that the game had changed. Continuous innovation was here to stay and only founders – not hired execs – could play and win.  Founder-friendly became a competitive advantage for his firm Andreessen Horowitz. In a seller’s market, other VCs adopted this “invest in the founder” strategy.

Fourth, in the last decade, corporate investors and hedge funds have jumped into later stage investing with a passion. Their need to get into high-profile deals has driven late-stage valuations into unicorn territory.  A unicorn is a startup with a market capitalization north of a billion dollars.

What this means is that the emergence of incubators and super angels have dramatically expanded the sources of seed capital. VCs have now ceded more control to founders. Corporate investors and hedge funds have dramatically expanded the amount of money available. And the globalization of entrepreneurship means the worldwide pool of potential startups has increased at least 100-fold since the turn of this century.  And today there are over 200 startups worth over a billion dollars.

Change Number 6 is that Starting a Company means you no longer Act Like A Big Company
Since the turn of the century, there’s been a radical shift in how startups thought of themselves.  Until then investors and entrepreneurs acted like startups were simply smaller versions of large companies. Everything a large company did, a startup should do – write a business plan; hire sales, marketing, engineering; spec all the product features on day one and build everything for a big first customer ship.

We now understand that’s wrong.  Not kind of wrong but going out of business wrong.

What used to happen is you’d build the product, have a great launch event, everyone high-five the VP of Marketing for great press and then at the first board meeting ask the VP of Sales how he was doing versus the sales plan.  The response was inevitably “great pipeline.”  (Great pipeline means no real sales.)

This would continue for months, as customers weren’t behaving as per the business plan.  Meanwhile every other department in the company would be making their plan – meaning the company was burning cash without bringing in revenue.  Finally the board would fire the VP of sales.  This cycle would continue then you’d fire the VP of Marketing, then the CEO.

What we’ve learned is that while companies execute business models, startups search for a business model. It means that unlike in big companies startups are guessing about who their customers are, what features they want, where and how they want to buy the product, how much they want to pay.  We now understand that startups are just temporary organizations designed to search for a scalable and repeatable business models.

We now have specific management tools to grow startups. Entrepreneurs first map their assumptions and then test these hypotheses with customers out in the field (customer development) and use an iterative and incremental development methodology (agile development) to build the product. When founders discover their assumptions are wrong, as they inevitably will, the result isn’t a crisis, it’s a learning event called a pivot — and an opportunity to change the business model.

The result, startups now have tools that speed up the search for customers, reduce time to market and slash the cost of development. I’m glad to have been part of the team inventing the Lean Startup methodology.

Change number 7 – the last one – is perhaps the most profound and one students graduating today don’t even recognize. And it’s that Information is everywhere

In the 20th century learning the best practices of a startup CEO was limited by your coffee bandwidth. That is, you learned best practices from your board and by having coffee with other, more experienced CEOs. Today, every founder can read all there is to know about running a startup online. Incubators and accelerators like Y-Combinator have institutionalized experiential training in best practices (product/market fit, pivots, agile development, etc.); provide experienced and hands-on mentorship; and offer a growing network of founding CEOs.

The result is that today’s CEOs have exponentially more information than their predecessors. This is ironically part of the problem. Reading about, hearing about and learning about how to build a successful company is not the same as having done it. As we’ll see, information does not mean experience, maturity or wisdom. 

The Entrepreneurial Singularity
The barriers to entrepreneurship are not just being removed. In each case, they’re being replaced by innovations that are speeding up each step, some by a factor of ten.

And while innovation is moving at Internet speed, it’s not limited to just Internet commerce startups. It has spread to the enterprise and ultimately every other business segment. We’re seeing the effect of Amazon on retailers.  Malls are shutting down. Most students graduating today have no idea what a Blockbuster record/video store was. Many have never gotten their news from a physical newspaper.

If we are at the cusp of a revolution as important as the scientific and industrial revolutions what does it mean? Revolutions are not obvious when they happen. When James Watt started the industrial revolution with the steam engine in 1775 no one said, “This is the day everything changes.”  When Karl Benz drove around Mannheim in 1885, no one said, “There will be 500 million of these driving around in a century.” And certainly in 1958 when Noyce and Kilby invented the integrated circuit, the idea of a quintillion (10 to the 18th) transistors being produced each year seemed ludicrous.

It’s possible that we’ll look back to this decade as the beginning of our own revolution. We may remember this as the time when scientific discoveries and technological breakthroughs were integrated into the fabric of society faster than they had ever been before. When the speed of how businesses operated changed forever.

As the time when we reinvented the American economy and our Gross Domestic Product began to take off and the U.S. and the world reached a level of wealth never seen before.  It may be the dawn of a new era for a new American economy built on entrepreneurship and innovation.

%d bloggers like this: