Integrating AI for Combat Effects

Watch the Video

---

Read the Transcript

---

Col. Toni Gray:

Okay. Good afternoon, everyone. Thank you for joining us today, I am Colonel Toni Gray. I am the division chief for ACC A3 Artificial Intelligence Integration. We are a brand new organization stood up at headquarters Air Combat Command, three years in the making but we focus specifically on integration of AI tools and platforms for operations. And so up until about a year ago, anyone who’s been following trends in artificial intelligence, many of the discussions around artificial intelligence primarily trended toward what in the future that AI could make possible, and whether or not it was going to be operating in isolation or as part of an entire system. And what we’ve seen here this week is that we have trended from that isolation of artificial intelligence into a full scale stack in order to employ it for combat operations. And when we talk about artificial intelligence, a lot of the time it can be very difficult to put your finger on, because often artificial intelligence is talked about as if it is one thing.

It’s one capability to acquire, it’s one system to implement. It’s something that your organization either has or it doesn’t. But in reality, AI is just an umbrella term for a distinct set of advanced computing capabilities that must seamlessly integrate and work together. And much of that technology is invisible to the end user, because when integration of AI tools works well, it just fades into the mission and that’s the point, but that is also what makes the integration of AI very difficult and it’s a challenge that’s very easy to underestimate. So, what our panel of experts is going to walk us through today is how we get the greatest operational value out of the layers of AI enabled technology that need to work together. And so, we’re going to walk through today data, perception, action and coordination. And those all need to function together to create combat effectiveness.

And with that, I’m going to introduce our industry panelist. We have Mr. Peter Guerra, he’s going to open us up at the foundational level. At Oracle, his focus is unifying infrastructure, environments, and compute foundations that government and defense systems rely on. He’s led data and AI work at Microsoft, AWS, Accenture, and Booz Allen Hamilton. He has published work with the Institute of Electrical and Electronic Engineers, he’s spoken with NVIDIA and at CloudWorld events, and Peter’s going to talk to us about what it takes to make the data environment, what it looks like, has to look like architecturally before AI can perform reliable at any operational scale. William Duhe is going to pick up from there, because once the data exists, the next challenge is turning that flood of data, the different sensor inputs, cameras, radars, laser-based distance sensors into one single coherent picture of what is actually happening.

At SNC, Duhe works on vertically integrating AI across SNC’s portfolio. His career has spanned real-time sensor fusion, building three-dimensional models of the environment, and object detection and tracking, and those that run directly on devices in the field. And so, he’s going to take us into what it really takes to turn many sensors into one picture of the world. Matt George, he’s going to take us from understanding to action. Matt is the CEO and co-founder of Merlin, they’re an autonomous aviation technology company and Merlin builds the Merlin Pilot. It’s an AI autonomous flight system that integrates into new and existing airframes. And it does the four jobs of a pilot, aviate, navigate, communicate, and operate. Merlin’s under contract with USSOCOM and the C-130J autonomy program with critical design reviews already complete. He’s logged hundreds of hours in autonomous flights across the Mojave, New Zealand, Quonset Point, and he is going to walk us through what changes when a machine’s output becomes physical behavior in the real world.

And then Mr. Ryan Tseng, he’s going to close out our stack at the top. At Shield AI, his work operates at the level of multiple systems acting together toward a single mission, including work tied to the Air Force’s CCA program, and he’s going to bring us his perspective on what it takes for a whole set of capable systems to behave coherently toward one objective. And so, together we’re going to cover the whole stack. You got the data underneath everything, the perception that makes sense of it, the action that it acts on, and the coordination that ties multiple actions into one mission. And so with that, we’re going to get started at the foundation with Peter.

Okay, Peter. Before a system can perceive, act, or coordinate, it has to start with data, and that data needs to be available, usable, and relevant to the mission. So, if you could talk a little bit about what has to be true about the data environment before AI can produce reliable operational effect.

Peter Guerra:

Yes, and thank you. And thank you for having me, I appreciate that. And I appreciate the way that you’ve structured this, because from my perspective and having deployed multiple types of AI systems over the last 20 years, the most important essential condition is how we treat data. Data as a mission ready capability, not just something that an organization possesses. And a lot of times what I’ve seen is that we think about data as a volume problem, but it’s not just about how much we have, it’s about how we create the right operational effect from the information that we are derived to make sure they have trusted sources and a form that can be used with a context that can be learned. For example, we say you can’t shoot a missile you don’t have, so supply chain, super critical, but the data equivalent is you can’t make a reliable AI enabled decision with information the system can’t find, understand, trust or access.

And that’s often what I see with a lot of these pilots and POCs around AI, specifically around DOW, but also all of the global customers that I work with. AI that doesn’t know your data generally is not going to work. I mean, that’s just a foundational principle of how the technology works. An AI model can’t use information that’s siloed, it’s stale and inaccessible, it’s kind of stripped of its content. That’s not how it works. So, what we help our customers with, and obviously we have a lot of the world’s very sensitive data stored, processed and used within the Oracle system, we have three major tenants that we ask our customers to think through before deploying AI at scale. And the first is that data has to be discoverable, accessible across the mission, not trapped in applications, not stored in buckets randomly across the organization, et cetera. You have to have the clearly defined set of data that is accessible, governed, and that can actually be discoverable and that’s super important.

And the second that it has to be understandable, and this is something that we see a lot with customers, they know they have data but they don’t necessarily understand the context of how that data interacts with other data. So a model, like an AI model like a GenAI, it can’t reason over data that’s kind of spread all over and it’s only getting a part of the picture. It needs to be able to see the whole picture in order for context to work appropriately. And then third, the environment has to be built in a way that allows that data not only to be discoverable and understandable, but also it needs to be able to be connected together in the right way with the right security provisions in the right place and so forth to make sure that it works across tactical edge, different clouds, whether it’s degraded environments, et cetera.

Because if you have a model that’s running that relies on certain data and that data suddenly isn’t there, obviously the output of that model is not going to be good. So what we recommend, and those are the foundations that we recommend from a data perspective for AI, and we’ve done this with a lot of you all, we’ve been great trusted partners with Air Force and other parts of DOW for many, many years. And when we stick to those three tenants, that’s where we see really the AI models proliferate and provide a lot of value.

Col. Toni Gray:

Fantastic, thank you very much. So Duhe, Peter’s given us the foundation, but having the data is really only the beginning part of this. And so, the next challenge is turning that information into an understanding of the environment that’s useful enough and timely enough to operationally matter. And so, what changes when an AI output has to become action in the physical world, and what does it take to trust that action enough to field it?

William Duhe:

Thank you, Peter. And yeah, data is where everything starts and then we have to make perceptions about that data, and I want to define those terms. So, data is measurements about our environment, about the world around us, just like we see, we hear, we feel, our systems and our machines need to do the same thing. And in order to perceive, you have to understand, you have to interpret all the information coming in around you so that you can make a decision about something. But the understanding is the floor of what perception enables. Perception is the ceiling and understanding is the floor. So, more data gives us a higher ceiling of what is knowable, but perception gives us a higher base rate of what we can actually know in an environment. And AI is accelerating and creating more opportunity to understand our environments at scale. The challenges around using that system at scale boil down to a lot more human and unglamorous issues often.

And this comes in the form of rigorous systems integration, time, how much time drift happens between two clocks and a sensor fusion scenario. You have something traveling at 300 meters per second, two sensors detect it simultaneously, the clocks are 30 milliseconds apart, that’s nine meters of covariance before it’s even reached the neurons in the model. And so, the models are getting incredibly powerful, and being somebody that’s worked with them very in depth recently, I’m very amazed at how well they’re capable of understanding and reasoning through complex human language and the environment that we describe with it. But if we don’t integrate and tune and rigorously calibrate the systems that we deploy, and give our models and our machines the eyes and ears to look out into the world and understand things, then they’re going to fail. And so, it’s not sexy but it’s ultimately important. And the second big thing I’ll say is that we are starting to move towards a realm of technology where we have intelligent macro systems, and these intelligent macro systems are not deterministic in the way that we’re typically used to thinking about technology.

A calculator, two plus two is four. It is very deterministic. Every time you plug in two plus two you get four. With these frontier models and large language models and computer vision/vision language models, we’re starting to see increasingly complex and intelligent systems that are non-deterministic. And we’re going to want to use those as part of our production systems, but this is going to require a little bit of re-architecture of how we quantify and propagate error through our systems, and uncertainty. Uncertainty is a valuable asset by understanding how certain or uncertain a measurement is and a qualification is, we can then degrade or enhance the decisions we want to make with our systems. So, I think that to circle back, a useful perception from a machine is one that we can measure the uncertainty around and validate after the fact, and crosscheck.

Col. Toni Gray:

Very true. Okay, Matt, now that we’ve moved from heading with the data to being able to make sense of it, understanding the environment doesn’t by itself create operational effects, and the next step is turning that understanding into action. And so from your point of view, what changes when an AI’s output has to become reliable action in the physical world?

Matt George:

I think often we really focus some of these conversations around the really cool things that AI can do, and AI can do a bunch of really cool things. But just as a show of hands, who’s a pilot by the way in the room? Show of hands. All right. Who as a pilot who has dealt with an error or an emergency or a failure in flight? Same hands go up. Who as a pilot has had to deviate around weather in flight? Yeah, same hands go up. Yeah, often those things happen together. So, in the real world when we’re saying, “Hey, look, all the cool things that AI can do,” you still have to deal with some of the meat and potatoes, rather unsexy parts of autonomy that need to take a mission and actually go execute on the mission. What do you do when your autonomous system is flying to an objective but there’s a thunderstorm along the way, and flying through the thunderstorm would tear the aircraft apart, but deviating around the thunderstorm puts you in airspace that causes World War III?

What do you do? So, those are the problems and some of the questions that we think a lot about. You’ll hear from Ryan and some of the other really incredible things that them and others are doing on the really, really high end of autonomy, but there’s also a bunch of infrastructure that goes below that in order to make those missions possible. So that’s number one, really thinking about how to go take that environmental data and making gray area decisions in a highly contested environment about how to go execute on the mission. And second is certification. Once again, not a topic that gets the blood flowing, but at the end of the day is one of those things that if we’re going to go put our service members in an aircraft or we’re going to put our service members around an aircraft, or more importantly put the aircraft around folks who may not have bought into any of our crazy experiments on the ground, certification really matters.

So, some of the work that we’ve done at Merlin over the past eight years, which feels like eight dog years sometimes, has been how to go take highly non-explainable systems and put those into highly deterministic boxes, so we can go take those novel thinking systems, but do so in a way where those novel thinking systems have some constraints around them so that we can go to our friends at ENEZ or LCMC, or some of the other folks that we work with and say, “Here is why this is safe in order to go put on your aircraft, put onto your system and integrate it actually into the macro environment.” And the third thing I’ll talk about here is making it real as a team sport. None of us can do this on our own, and interoperability really matters. So once again, I’m hitting all the high points of things that people find boring at the end of the day, which is certification and interoperability MOSA and FACE.

But at the end of the day, this is a team sport. The amount of time that our team has spent on 1553 bus data, and figuring out how to go take that from legacy operators and legacy primes and work that together to actually get a capability out to a customer, that’s really, really, really important. So the gap between like, “Hey, look, what we can do to make it real and what we can do and what we need to do to get that in the hands of the folks that need the most, and how to actually get there, don’t forget about that part. It’s all those gray area dumb decisions around hey, is that landing gear light actually failed or is my landing gear not down? It’s how do I deviate around the weather? How do I make sure that I’m deploying AI in a way that is certifiable?

How am I deploying autonomy in a way that the folks at Wright-Patt can get comfortable? And how am I doing it in a way where we have systems that in our case, working on the C-130 and the KC-135 are sometimes from the late ’50s and the mid ’60s? And how do we do that in a way where we get the capability out to the war fighter and don’t forget about that hard middle of taking it from the possible to the real?

Col. Toni Gray:

Thank you. So Ryan, once individual systems can act effectively, we move beyond the behavior of just any one system. And so, can you talk to us about what changes when mission intent has to be translated into coordinated behavior across multiple systems?

Ryan Tseng:

Yeah, thank you. I think at a fundamental level there’s not a lot of change between translating commander’s intent to one system or a bunch of systems, there’s some additional networking complexity that has to be worked out. But I think the hardest thing is building the trust between the commander and what’s going to happen. So played sports, I was a wrestler. When you wrestle, you concern yourself with what your opponent’s going to do, you try to figure out what you’re going to do, and it’s a pretty contained space, a lot of complexity in that engagement. Also played team sports where you have to worry about not only what the person right across from you is going to do and what you’re going to do about it, but what everybody else on your team is going to do to keep the entire team objective moving forward. And that introduces a lot of complexity, not only for the people that are on the court or on the field, but also for the coach to try to anticipate what all the dumbasses in high school are actually going to do despite all the drilling and practice.

You practice it a million times and then we don’t listen to what the coach tried to teach us to do. So, the question that I think is really important for the joint force to figure out is as we think about translating commander’s intent to autonomous behaviors, how do you build that trust in the execution that’s going to unfold as the events are very dynamic? The thing in terms of concrete actions or recommendations for things that we think the joint force should be thinking about is taking simulation environments, like the joint simulation environment that are built today to enable the joint force to train humans to conduct missions together. Maybe we need to think about ways to open up that environment to AI pilots so that people can get familiar with what the AI pilots are going to do as they execute their missions.

They can start building trust between agents and then take it another step forward, consider changing the mandate of the joint simulation environment to not just be an environment to train pilots across the joint force to fight together, but to also train the machines that are going to fight with the pilots to fight with each other. The dimensionality of what an autonomous system can do in an unpredictable fight is absolutely enormous. And the only way that I think we can begin to get our heads around how it’s going to translate intent to action is extremely large scale simulation.

Col. Toni Gray:

Okay, great. So, now you can see we’ve moved through the stack one layer at a time. We started with the information foundation, then we moved into making sense of the information, translating it into action, and finally coordinating toward mission objectives. But when we’re dealing in operations, we know that none of these layers exist independently. So, this is where I want to bring you forward into your own conversation now. And so, I want to talk a little bit or have you guys talk a little bit about most of how your layers work together and what you need from the other layers to work well in operations toward combat effects. So, anyone want to volunteer to kick us off or otherwise going to pick someone?

Peter Guerra:

Sounds like I’m being picked. All right, awesome. Well, certainly for data, one of the most foundational things is the right compute layer and the right transport layers. I mean, in order for the data to be available to do what these folks are talking about, you have to be able to store, process, you have to be able to transport it, all of those things, key foundational layers for that. And then of course, when you’re talking anything AI, you have to have the right type of compute, TPUs and other types of things like that, that can actually interact with that data and process it in the right speed to be able to draw the connections that it needs in order to be able to operate effectively.

Col. Toni Gray:

Right, thank you.

William Duhe:

And to build on that, I think that providing well-structured context rich training data and well-labeled data upstream to each of our layers is one of the most integral parts of each of our pieces of the puzzle, because well-structured, well-labeled, and in the future context-rich labeled data can help facilitate things like robust simulation environments where we can have as close to boots on the ground simulations as we need in order to get actual factual results from our emulated environments.

Ryan Tseng:

And maybe just to add on to that, we also have to think about data generation and how do you do that at scale and relevant environments and engagements? And of course, there’s everything the joint force is doing around the world today, but we have to think about ways to scale that data generation, and you can use synthetic data, you can use simulation to create scenarios and engagements that become the backbone of a lot of the behaviors that can be deployed. At Shield, several of us are close to current and former members of the Tesla Autopilot team, and if you track over the last two years there’s been a step function increase in the performance of those systems. Basically, finger in the wind, 50 to 100X improvement in time between critical interventions. And so I was asking the team, “What was the unlock? What made that suddenly possible?”

And they had finally gotten the SIM to real correspondence right, they were able to dispatch very large scale simulation to augment everything that they were doing with real-world data to enable more modern methodologies, to enable very high levels of performance. And so when I think about, again, getting AI pilots deployed across a fleet to be very capable, high assurance, all the things that these gentlemen talked about, we need to be thinking about how to scale the simulation environments that have the right physics, the realistic threats, the right BlueForce models, and are manned by the greatest pilots in the world in the United States Air Force and those across the joint force.

Matt George:

For us, it’s getting first downs. I think we could sit here and talk about interoperability all we want, sort of the 50 yard line and try to go create a magic football and hug it into the end zone, but if we can work together, whether it’s the A-GRA and some of the other stuff that our awesome partners on the government side are working on, and start to go integrate these onto particular airframes and start to go actually go fly, and increase the cadence of getting stuff out there. So, just give kudos to some of you guys in the room. For those of you who are aware, the U.S. has an awesome asset out at Edwards Air Force Base called the Vista, the F-16 Vista operated with Calspan, and the Air Force is now expanding that with Project Venom.

It’s a step in the right direction, but one of the things that we’re working on is trying to make sure that we have a multitude of aircraft of different performance profiles that are all A-GRA compatible, then we can start to go test a variety of things and actually get some sets and reps up in the air.

To Ryan’s point earlier around the Tesla Autopilot team and thinking through that, simulation doesn’t matter unless you have at least a good level of base data. So for us, actually going out and getting thousands of hours of flying in weather, when somebody cuts you off in the pattern, when somebody does something stupid around you, that’s tremendously important. So, to answer your question about interoperability, I think it’s trying to get those iterative first downs versus trying to go contrive a perfect solution 50 yard line.

Col. Toni Gray:

Right. And so Ryan, when we’re talking about coordination, so how much of the problems that you see are determined by the quality of the information that’s coming up to you?

Ryan Tseng:

I would say at a tactical level, so just getting systems combat fielded, we’ve never really been constrained by data because we can go out and we can get it. When you start talking about large scale coordination, it becomes more difficult. So just to give you an example, Shield AI, I think is the only defense tech company that’s had a full-time presence in Ukraine till today, all the way from the FLOT to back, some of our best engineers are out there working alongside some amazing people to produce mission outcomes. And one of the things that we did with one of our partners, we integrated advanced AI and autonomy on a one-way attack drone. So, they were trying to make a push to operational depth fires, think 100 to 200 kilometers of probability of kill on the weapon systems where it’s in the 2.5%, maybe 5% or lower range.

And so, we put advanced autonomy on these systems we’re producing with our partner 1,000 a month, and we’ve seen the probability of kill for that weapon system increase to 70% because of advanced autonomous behaviors. And one of the things that it does is it takes off-board queuing and image chips, it’s provided basically a kill box. And if it finds a match to a stack of images against the target, these systems queued by another system can go in and kill the target. And so in that example, we needed data for the ATR system, we needed to understand how to coordinate systems together. Some people in Ukraine had to make decisions about rules of engagement, but the net income was a dramatic uplift in the lethality of a particular weapon system. Now, that is a system that’s not connected to a large JADC2 system, so it’s like people picking up the phone and calling each other and just trying to get shit done in a very difficult circumstance.

When we think about a future that’s going to have very large scale deployments of systems that need to interoperate with each other, it’s just, like I said, it’s a much larger decision space. A lot more things can happen, and then therefore the amount of data that you need becomes larger. I think for any tactical deployment, anybody here in the room, there’s probably no, I’ll say weapon that would necessarily be limited, aircraft or another story, but weapon that would be limited by data availability because a good vendor working closely with the government can try to sort of figure out how to get enough data to get the job done. But if you start talking about more complex operations, like the United States fights as a joint force, the packages, they can be big and complicated, and that’s a situation where having the right data augmented by simulation starts to become very important.

Col. Toni Gray:

So, I’m going to open up to the group. So over the next two years, so we’re building all these systems and we’re starting to integrate them into each other, what do you think is the things that we need to get right to actually create operational effects on the battlefield? Peter, we’ll start with you.

Peter Guerra:

Okay, yeah. I mean, I could say data, that would be … Integrate all the data. I think that’s obviously extremely important. At least make it able to connect together, I think is super important. We’re working with the Air Force right now on a project to do that, to connect many, I think it’s like 15 separate systems that runs your operational supply chain, just as an example. It’s like 15, it’s not 1, so it’s a lot. And so, across the entire ecosystem, and then how does that plug in with partners? That is such a foundational part that for a lot of the operations it can’t be done until you have a better way of connecting all that. It doesn’t mean everything has to be in one giant data lake or swamp, or whatever you want to call it, but it at least has to be able to see each other or be accessible by some sort of model.

And so, I think that’s going to be critical for the next few years.

Ryan Tseng:

So, having systems deployed in combat around the world today to include in the Middle East and Ukraine, I think that there’s nothing … Number one, I think people just need to be bold and make decisions and move out and decide that, hey, this technology is here. Yes, there’s engineering problems but the real problem is at the intersection of engineering and force integration to figure out how to get these things combat effective and deployed down range. And so I would just encourage, I think if you can shrink down the problem into something smaller and start to get points on the board, it starts to get the entire team moving toward that bigger vision. So number one, getting the right projects kicked off, doing it in collaboration with industry, I think that there’s some low-hanging fruit and you can start to make impact quickly. Number two, it’s clarity of vision and what needs to be done.

And one of the conversations that I’ve been having, one of the things that I’ve been trying to figure out is who in the joint force is going to take the mantle and drive the very large scale simulation, data logging, training and validation environment that can be used to train AI pilots. So, if we take the joint simulation environment, which by the way, I know the Air Force and the Navy did a lot of work on this a decade ago, I think it’s genius. It’s the only, people here probably know, but it’s the only multi-service, multi-vendor, multinational, multi-domain simulator that’s out there. And it has the mandate to enable pilots to become very combat effective, but who is going to create that same environment for the AI pilots of the future? And who in the joint force can take that flag and run with it? Because industry is not going to be able to produce that independently.

The inputs to that model are intelligence-based threat models, the physics comes from vendors all over the United States. And so, the government has unique leadership role. And I think that figuring out who can go take the flag and create that system is something important for the joint force to think through.

William Duhe:

And very importantly to enforce a contract on how AI is to be communicated between vendors and partners and companies, because as we move towards agent forward architectures, it’s not just communications between humans anymore that matter in active systems, it’s agent to agent communication is becoming way more prevalent. And when my agents don’t know how to talk to your agents, it becomes a very big issue and an issue that compounds at scale, because they are capable and willing to communicate much faster than humans. And when there’s friction, that friction builds up much faster than with human communication.

Matt George:

I just want to call out, everyone always likes to say, “Hey, well, the government needs to move faster,” sort of things like that. But I do think that there are pockets that are actually doing really great work. So in our world, folks at SOCOM, General Conley, Ms. Johnson and others I think have shown a real desire and willingness to be able to go move very, very quickly but do so in a responsible way. We brought our autonomy system all the way up through CDR, PDR, nothing more we can announce there yet, but that’s big. That’s going to be one of the first times a true takeoff touchdown autonomy system flies an operational aircraft. And that type of responsible team building is going to be really important to get points in the board. And the second is industry partnerships. So, we announced a team up with Textron to go develop a pretty cool new concept based off their sky courier aircraft, and things like that where we’re not going to be a great manufacturer of a 12,000 pound airplane, they’re not going to be a great manufacturer of full takeoff touchdown autonomy.

So, those two things together are things that we can use and leverage in order to go increase the speed of play.

Col. Toni Gray:

So, we have a couple extra minutes. Is there anything that I missed that somebody wants to address when we’re talking about AI for combat effects? Okay. So, we’ve spent the last 30 minutes or so taking apart a term that we use as though it describes one thing. And so, the point of this panel and what the conversation I hope it shows is that in order to create a real operational effect with AI, it takes far more than just one single algorithm, a large language model, a platform or a system. Everything has to support each other toward the end. So data has to support perception, perception has to support action, and individual action has to scale into coordination, and that’s where the real value for artificial intelligence shows up and it’s really when all of these pieces come together. So in the not so distance past, which might be all the AFA AI videos I audited from last year prior to this, a lot of the conversations we had really leaned more toward the theoretical.

They were questions for later. Well, today it’s not later anymore. So, every layer that we’ve talked about here is already real and it’s already deployed someplace on the battlefield. And so, my prediction by the time we all return here next year and then we’re back in this room is that the gap between what we’ve discussed and what is possible, and what is actually fielded will have closed even further. And so, I think it’s definitely something worth watching between now and then. So, thank you very much everyone. Have a great afternoon.