A product team should review an AI use case after the pilot and before deployment, then continue regular reviews while the system is in operation. This applies NIST AI RMF guidance that AI systems should be tested before deployment and regularly in operation. The cited material does not establish a universal waiting period or fixed review interval.
How to check the post-pilot decision
Pilot completion should trigger an evidence review, not automatically justify deployment. A practical review should examine:
- The functionality and behavior observed during the pilot, including the behavior of individual components.
- Whether those results support the use case’s intended purpose and the acceptance criteria established before testing.
- Failures, edge cases, and conditions that were not tested.
- Changes to components, inputs, dependencies, or operating conditions that could alter the pilot findings.
- The disposition of unresolved issues: proceed, revise and retest, narrow the use case, or pause.
This is a practical way to apply the testing principle, not a review procedure prescribed by NIST.
What regular review should cover
Once the system is in production, each regular review should ask whether its observed functionality and behavior still support the intended use. NIST AI RMF specifically states that the functionality and behavior of the AI system and its components are monitored in production.
The team can organize that review around:
- Newly observed system and component behavior.
- Exceptions, failures, and changes in operating conditions.
- Updates to the system or its components.
- Evidence that may invalidate earlier pilot conclusions.
- Actions needed to revise, retest, limit, or stop the use case.
The cited material does not specify a review frequency, performance threshold, or mandatory set of production metrics.
What the team must still confirm
The appropriate cadence depends on circumstances that the cited material does not settle. Before setting an interval, the team must confirm:
- The potential consequences of failure and the team’s tolerance for unchecked behavior.
- How often the system, its components, or its operating conditions change.
- Applicable internal, contractual, legal, or regulatory requirements.
- The capacity available for monitoring, investigation, and retesting.
- Who can approve continued operation or changes to the use case.
Until those checks are complete, any proposed interval should be identified as the team’s operational decision rather than an official NIST requirement.