Effective Continuous Verification
Towards ML-driven Software staging automation and testing
Recently, I came across a post (ref 1) that deeply resonated with me as it highlighted a common challenge I observe daily: the promise of DevOps is stagnating. Consider these statistics from the post and reflect whether they also apply to you:
- Only 11% of DevOps users reported being able to restore service in under an hour.
- … a mere 14% can get code into production in a single day.
- … these days, 41% of users report taking more than a week to restore service.
In this post, I will share my observations and identify key areas that need to be addressed to overcome these challenges.
While most of us intuitively agree on the need for more agility and speed, traditional decision-making methods like Change Advisory Boards (CABs) and ticketing systems often become bottlenecks. These human-centric steps, valuable in the past, now hinder the DevOps flow. This reliance on human assessment typically stems from a historical lack of confidence in automated quality checks within CI/CD pipelines (code reviews, unit tests, integration tests).
We need to invest into the transition towards smart quality gates that operate based on insights generated from machine learning (ML) and artificial intelligence (AI). This requires a shift in quality assurance practices. At Swiss Digital Network, we call this new approach “Effective Continuous Verification.”
Nora Jones and Casey Rosenthal introduced the term Continuous Verification in the book “Chaos Engineering” as “a discipline of proactive experimentation, implemented as tooling that verifies system behaviours”.
Established DevOps titans define continuous verification in a slightly different way such as:
- GitLab: “Continuous verification is the process of querying external system(s) and using information from the response to make decision(s) to improve the development and deployment process.
- Harness: “Continuous Verification (CV) approach simplifies verification. Harness uses machine learning to identify normal behavior for your applications. This allows Harness to identify and flag anomalies in future deployments, and to perform automatic rollbacks”.
- OpsMx: “It verifies software updates across deployment stages, ensuring their safety and reliability in production”.
Driven by our vision to automate staging in DevOps and improve the overall testing and reliability engineering with the use of ML and AI, we at Swiss Digital Network , defined Effective Continuous Verification as:
“Systematic, comprehensive, and disciplined methodology to design and implement holistic DevOps observability and insight. The approach establishes quality gates incl. resiliency throughout the delivery pipelines and automates their verification, ultimately enhancing the productivity of testing and Reliability Engineering”.
Therefore, we focus on the following key elements within the scope of Effective Continuous Verification, on top of established and proven DevOps practices:
- Generating insights from DevOps signals to enable Smart Quality Gates
- Advancing overall testing practices and productivity
- Resiliency verification automation
- Extending observability into the Dev portion of DevOps
Let us now delve deeper into some of these aspects to illustrate our thought process. Future posts will explore each of these dimensions in more detail.
Overall framework
Smart QG fed with DevOps signals going through ML enhancement to drive staging decision
Insights from DevOps signals to enable Smart Quality Gates
Dynamic smart quality gates are automated CI/CD pipeline checkpoints that assess the code’s readiness to advance to the next stage. The term dynamic reflects the ability of these checkpoints to be coupled with various tools and generate varied data. The term smart reflects the ability of these checkpoints to be interpreted not only in a punctual and isolated manner, but also to be aggregated and combined according to rules of causality or interdependence. This holistic approach is more aligned with current DevOps practices.
There are 3 common challenges with this concept:
- Trust in our automated verification must be improved by investing into advanced testing methodologies
- Tapping into a wider set of information, than just static analysis and automated test results is necessary to build more fine-tuned models and getting higher quality in assessing quality, maturity and risk of change
- Manual human-based analysis and interpretation of all eventualities, makes this a challenging task. We believe that this can be sustained and supported by ML and AI.
Here’s where AI/ML can support building smart Quality Gates:
- Anomaly Detection: Machine learning can identify deviations from normal application behavior, enabling automatic rollbacks during deployments (Harness).
- Anomaly Causes Explanation: Combining multiple techniques such as Machine Learning, Generative AI and Knowledge Graphs can help into building Root Cause Analyses and add explainability.
- Holistic Quality Checking: AI/ML techniques are capable of aggregating multiple checkpoints and connecting them semantically and technically, so that we can provide holistic quality assessment, based on a global horizontal view of the pipelines.
- Risk weighting: Machine learning can correlate data across many DevOps events to correlate with a risk of the current change to be ready for go-live.
These are just a few examples, and we’ll delve deeper into AI/ML for decision-making in a future blog post on AI-SQUARE
Advancing overall testing practices and productivity
To build effective dynamic quality gates, it is essential to enhance testing practices, incorporating both shift-left and shift-right methodologies:
- Shift-Left Testing: Involves integrating testing early in the development cycle with the goal of fast feedback to change.
- Shift-Right Testing: Focuses on testing in production (or pre-production) environments to validate performance, reliability, and user experience under real-world conditions.
Typical challenges we have observed:
- Overemphasis on a limited number of test types (e.g., unit tests and end-to-end tests).
- Inadequate decoupling of components, which hinders effective component testing—a cornerstone of fast feedback.
- Poor quality of test data for integration tests, leading to flaky test results.
- Insufficient focus on non-functional aspects such as system resilience and scalability.
To tackle these issues, it’s crucial to adopt supporting test techniques such as virtualization, mocking, and test data management (e.g., AI-supported test data generation or test data anonymization).
To maintain overall productivity while emphasizing these key areas, generative AI can be leveraged to support test authoring.
This point is very important. I think we can split it into two aspects : wider the set of structured and tabular data. And also involving unstructured data to enhance exploitability
Seamless continuous delivery requires shifting decision-making from humans to codified quality gates. For such decision-making to be effective, testing needs to be improved, and the understanding of the system expanded through continuous verification and practices such as chaos engineering. The amount and velocity of data will require ML and AI assistance to create smart quality gates instead of static definitions. This lays the foundation to release with confidence at speed.
Stay tuned for a deeper exploration of these topics in our upcoming blog posts