How Good Is Your Model?

Model-based systems engineering (MBSE) in on the rise across industry and the government. Models are essential to achieving the DOD Digital Engineering (DE) vision, enabling them to be digitally linked to each other to inform technical decisions and linked to acquisition and program data to make acquisition and program decisions. The possible wide-range of MBSE-based activities requires focus to ensure that the models are built for purpose. Currently there are no standard means to express the value, depth, breadth, and quality of a MBSE model. Thus, it can be difficult for those without detailed knowledge of MBSE to understand numerous acquisition questions about a model, such as

• Are the models adding enough value and risk reduction to the program?
• Are they of high quality so I can be confident in the information they are providing?
• Are the models progressing as needed to meet their intended purpose and risk reduction?
• Are the models mature enough for the acquisition or development phase?
• How much technical depth is in my model?

To address these concerns, we developed a rating scale for models, called Model Assurance Levels (MALs). A MAL level is used to concisely express the value, depth, breath, and quality of the model, as well as risks associated with the model. MALs are based on a scale from one to three, with three being the highest to reflect increasing value and risk reduction of the model, as conceptually depicted in Figure 1. Within each MAL level there are also sublevels to show incremental growth. Each level also has a set of associated risks, which help aid in the selection of the MAL level. As Figure 1 depicts, a lower MAL level equates to a model that has less depth and breadth, reducing risk in the program. As a MAL level increases, the risk is further reduced. When applying MALs, not every program needs to achieve the highest MAL level. Instead, a MAL goal should be selected based on the amount of risk a program is willing to accept.

A quantitative and repeatable assessment approach was developed to consistently determine a MAL level regardless of who is performing the assessment. We are in the process of automating the assessment process to improve the practical application of MALs.

MALs can be used on a program several different ways. First, they can be used to set goals to determine the potential model investment. The MAL scale  can be used to set goals and expectations for a model early in a program. Second, MALs can be used to monitor the progress of a customer’s MBE effort. The MAL score can be determined multiple times throughout the development or acquisition lifecycle to determine if the model development is on-track. Finally, MALs can be used to understand and manage risks resulting from the model—to determine if the model is suitable for use. The risks associated with a MAL level can be identified and a strategy for mitigating risks can be developed.

Published information includes:

Conference Proceedings

https://dl.acm.org/doi/abs/10.5220/0007697505420549

https://www.computer.org/csdl/proceedings-articleisor/2019/015100a069/1…

SEF Forum 2021 Charts and Presentation:

https://www.dropboxcom/sh/lnlgtixlyqo3wjh/AAd1tt8PbQg2wdPh6QLjxrja?dl=0

https://www.youtube.comv=bfslhsehnJM&list=PLFEXcXJhNDQ5hx9VWtkESCWINhxK…