Validation Lifecycle
It’s important to remember that, while testing is a critical piece, validation is more than just testing and test documentation. Validation involves activities such as user training, validation, deployment, continuous monitoring, re-training, upgrading, and eventually retirement. The level of detail of these activities will depend on whether you’re using an internally developed or publicly available LLM.
Another critical piece when it comes to AI validation is data governance. This includes pre-deployment evaluations of data provenance (lineage, traceability, and data origin), bias evaluation, data quality, data storage and privacy protection, labeling controls, and versioning of datasets. Without documented control over training and evaluation data, the validation record is incomplete regardless of how well the testing is structured. Because internal LLMs (and some public LLMs) are often refined by training on newer data, the data used may not meet security requirements of the regulatory body or the user. This must all be documented before deployment.
Above all, it’s critical to remember that the implementing company remains accountable for the AI system’s performance, even when using third-party models. AI validation must address both the model’s behavior and the surrounding system controls.
CSV vs CSA
At this point, it’s prudent to explain a little more about the history of software validation. With traditional software validation, the most commonly accepted structure was Computer Systems Validation (CSV). This involved extensive documentation, including a description of test methodology, comprehensive documentation of software requirements (design, functional, and usage), robust test scripts, thorough documentation of test results, and final validation reporting.
As vendor documentation and software test practices became more robust, it became clear to regulatory bodies that companies were spending too much time on non-value-add test activities (functional testing that had already been performed by the vendor), which took time and resources away from their own products and safety. Because of this, regulatory bodies started issuing guidelines on risk-based testing. The FDA’s risk-based testing initiative is referred to as Computer Software Assurance (CSA).
The CSA approach to regulatory testing is a risk-based test methodology. Documentation includes defining and capturing the areas of the software that have a direct impact on patient safety, data integrity, or product quality. Then testing only those areas as you intend to use them. This allows you to take a more targeted approach to ensure your product and patient safety are assured. As it turns out, CSA is an excellent approach for testing AI systems as well.
Planning Your Validation
It’s important to remember that validation is more than just the test activities. Validation is the full documentation of intended approach, controls, actual testing and evidence, and summary of activities. Below is a high-level framework that can be used to validate AI systems.
Testing LLMs Using a Risk-based Approach
It can be especially tempting for those coming from a traditional testing background to try to test every button, link, field, possible input, and possible outputs of an AI system. While this was often difficult for traditional software suites, it is almost impossible with AI. Because the natural language inputs are as varied as the language itself (and the outputs even more so), attempting to define and document all possible inputs and outputs is not within the realm of human possibility.
State the Intended Use and Workflows
Validation documentation should include a description of how the system is intended to be used. For example:
- Will the AI be used to assist with making decisions, or will it be making decisions itself?
- Will there be a human review? If so, when (always, at critical decision points, etc.)?
- Will the AI be advisory or autonomous?
- What harm can occur if the AI is wrong?
Companies using AI should also document how critical systems will be used in their various workflows. Answer questions such as:
- When will AI be used?
- For what purpose will it be used?
- Are humans in or out of the loop?
- Is there an escalation point?
- Is it possible to override the system?
- How are AI responses used?
Identify the Risks
For the above, identify the risks posed by the various processes and touchpoints and the level of risk that must be tested. Then organize in a logical way that test scripts and artifacts can be easily documented and traced to the risks. Typical risks include (but are not limited to):
- Patient safety
- Product quality
- Data integrity
- Regulatory reporting
- Privacy
- Ethical risk
Perform the Testing
Whereas traditional software test results could be documented with a simple Pass/Fail, AI testing may not be as straightforward. The overarching question is likely not “Does it work?”, but rather “What is the probability that it will perform within acceptable risk tolerance?” It may be necessary to identify the statistical results, including:
- Accuracy
- Precision/recall
- False results
- Confidence thresholds
- Acceptance criteria
Another important aspect of testing is to incorporate “adversarial” testing and continuous monitoring. Adversarial testing can include any of the following:
- Edge cases
- Adversarial prompts
- Ambiguous inputs
- Incomplete data
- Incorrect inputs
- Long-thread prompts (if allowed)
Continuous monitoring involves watching the LLM for known events such as drift, performance degradation, error rates, or deviation from baseline validation benchmarks. It also involves documenting mitigation and recovery options should a model display these behaviors.
Document Change Control
One constant of modern software is that it regularly changes with updates for new features, security fixes, and so on. Public LLMs can have a much more frequent release cycle than other modern software, whereas private LLMs may have a more measured release cadence. Regardless, as with traditional software, companies implementing AI systems should document their change control processes. This includes subjects and questions such as:
- How will we manage model versioning?
- How will we manage prompts version control (if any)?
- How often will updates be applied?
- What is the review process?
- Where are release notes?
- How are new features assessed against our documented risk?
- Will the system be implemented in a test or sandbox environment before being applied to Production?
It’s vital to remember that using a public LLM does not transfer any regulatory responsibility.
Documenting the Test Results
As with traditional validation, all test results must be documented. Especially with AI, it is critical to be able to explain your use of the system and provide transparency into evidence that it is being used that way and is performing as expected. Regulators increasingly expect the following documentation:
- Clear boundaries of intended use
- Traceability of decisions
- Explain ability of outputs
- Auditability through the logic path
The world of AI regulation is moving fast — and staying compliant means staying ahead. With that said, stay in tune for our next post that dives into “Following the Regulatory Landscape”.