Project Convergence Data Cloud Pilot

A modern data engineering pipeline at the edge

I led the PC21 Data Cloud Pilot team that intended to increase our use of instrumented data to drive decisions in a data-centric way with edge and cloud technologies. Our focus was to find ways to automate our ability to stitch together the tactical data link (TDL) messages within a sensor-to-shooter (S2S) thread. We define a S2S thread as the time when a sensor classifies a target and ends with an engagement of an effect on that target.

Goal. To automate the identification of S2S thread information flows in order to enable rigorous analytical evaluations of the complex PC use cases.

Value. To effectively understand how to evaluate AI algorithms at sensors and decision nodes in order to route decisions that will employ effects in a Denied, Disrupted, Intermittent, and Limited-Bandwidth (DDIL) environment.

What we did (lines of effort):
1. Configuration Management. We manually developed new data products to join the existing disparate architectural products, providing a better understanding of the information domain.
2. Data Engineering Environment. We built edge and cloud data engineering environments that included the data format software decoders we needed, enabling us to fully stitch together a single S2S thread. We connected the first ever Azure Stack Edge devise to the Secret Internet Protocol Router Network (SIPRNet).
3. Network Observability. We instrumented key nodes to capture the specific TDLs and built network monitoring tools to observe the network, enabling near-real time understanding.
4. S2S Thread Forensic Analysis. We stitched together one S2S thread by finding patterns inside the TDL message data elements, demonstrating how to implement and automate the analysis.

The below figure depicts the architecture we deploy on an Azure Stack Edge, the first ever to connect to the SIPR network.