By Florent Mathieu, CEO of EikoSim
After several occasions where I’ve been given advice to consult specific documents in the field of validation testing and VVUQ (Verification, Validation, and Uncertainty Quantification), I’ve decided to write a blog about this.
Here is a more detailed list of important papers, books or websites that I found along the way and are useful when learning about VVUQ approaches in the field of solid mechanics or aerospace, which is our core business at EikoSim.
Although some documents are specific to this domain, most documents cited here apply to others.
I should also add that this field is intended for fast-growing data and that this list should be updated regularly to take new documents into account.
VVUQ Standards
VVUQ standards are probably the documents that will come up first if you research compliance for VVUQ compliance.
Here are a few of the most important documents that are around.
Let’s start with simulation-oriented documents:
- ASME V&V 10: Standard for Verification and Validation in Computational Solid Mechanics: This standard focuses on verification and validation practices specific to computational solid mechanics simulations. It covers topics such as mathematical modeling, mesh convergence, code verification, and validation. Note that an aerospace-specific document is underway at ASME and should be available in 2024.
Note: this standard tells you what to do (i.e., compare a strain gauge measurement with simulation results for strain) but does not tell you how. Struggling with this step?
Check out how it’s done in EikoTwin Lite.
- NASA-STD-7009: Standard for Models and Simulations: This NASA standard provides guidelines for developing, verifying, and validating models and simulations used in aerospace systems. It covers various aspects of the modeling and simulation process, including requirements, verification, validation, and documentation.
On the test side, lots of documents can be found to learn about measurement errors, including:
- ISO 5725: Accuracy (trueness and precision) of measurement methods and results: This ISO standard provides general principles and guidelines for assessing the accuracy of measurement methods and results. While not specific to aerospace engineering, it can be useful for understanding uncertainty analysis and measurement system validation.
- There is no universally accepted standard for uncertainty and bias determination in DIC measurements, but the DIC community at iDICs has been working on a Good Practices Guide for DIC that helps DIC users navigate error sources and evaluate random errors of their DIC measurements. The “noise floor” approach is the one we use in EikoTwin DIC. EikoSim is a proud member of iDICs!
These documents and procedures can serve as references to establish VVUQ practices and testing procedures within an aerospace mechanical engineering department.
Always make sure you’re consulting the latest versions of these standards, regulations, procedures, protocols, specifications, regulations, and guidelines and exploring additional resources and services to ensure you have the most up-to-date and comprehensive compliance information.
VVUQ Frameworks
VVUQ standards are useful user tools that are used by industry leaders to build user confidence in their simulation efforts and the user will.
But how to evaluate the current level of confidence in a modeling and development process, validation process, and validation and implementation process, on which engineers are increasingly relying to make software systems make important decisions?
This is the reason specific testing and validation frameworks have been developed to evaluate simulation maturity within an organization or program.
Some service industries have adapted maturity scales (such as TRL) to measure operational maturity at service or production service levels, for example, but these two standards have been developed specifically with simulation in mind:
- The Credibility Assessment Scale (CAS) is a quality framework that has been developed by NASA to incorporate Verification, Validation, and Uncertainty Quantification (VVUQ) activities and ensure the reliability and accuracy of computational simulations. CAS emphasizes rigorous verification of simulation codes, validation against experimental data, and uncertainty quantification to assess and manage uncertainties in the simulation results.
- The Predictive Capability Maturity Model (PCMM) is a comprehensive quality framework developed by Sandia National Labs. PCMM focuses on advancing the predictive capabilities of computational models and simulations. It provides a structured approach for organizations to assess and improve their simulation capabilities. PCMM encompasses VVUQ activities as key components to ensure the reliability and credibility of simulations. The original document (2007) has been followed by several updates, and we recommend going through the authors’ production to ensure you’re updated on this topic. By the same authors, a very comprehensive book about VVUQ was issued in 2013.
VVUQ Communities
Getting involved in communities related to Verification, Validation, and Uncertainty Quantification (VVUQ) can provide valuable networking and development opportunities, access to resources, and opportunities for further development and knowledge sharing.
Here are a few notable communities where you can engage with experts and practitioners in the field of development of VVUQ:
- Engaging with professional societies related to your specific domain can offer opportunities to connect with VVUQ experts within your field. In our industry, the American Society of Mechanical Engineers (ASME) is a world leader in the field of VVUQ. Participating in their events, conferences, and working groups can help you connect with professionals who have a strong interest in VVUQ within your specific industry.
- NAFEMS is a worldwide community dedicated to developing Simulation and Finite Elements, amongst other topics. The NAFEMS Simulation Governance & Management Working Group (SGMWG) is one of the committees at NAFEMS that is interested in VVUQ. The SGMWG has issued a number of very useful documents, for instance:
Additional Resources for Validation Requirements
- Simulation Verification and Validation for Managers: If you’ve read this article up to this point, you’re probably wondering how to justify additional efforts to your manager. This book has been written for you (and for them)!
- Engineers at Sandia have been producing a “Credibility Framework,” which contains software for PCMM and PIRT evaluation. I’ve never tried it, but it could be a good entry point for the practical execution of VVUQ techniques.
