CWE-1426
Improper Validation of Generative AI Output
Description
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
Parent Weaknesses (ChildOf)
Common Consequences
Scope
Impact
Execute Unauthorized Code or Commands, Varies by Context
Potential Mitigations
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
CVE-2024-3402chain: GUI for ChatGPT API performs input validation but does not properly "sanitize" or validate model output data (CWE-1426), leading to XSS (CWE-79).
Applicable Platforms
Security Training
Train your team to recognize and prevent security threats with our comprehensive security awareness program.
Start TrainingVulnerability Scanning
Discover vulnerabilities in your applications and infrastructure before attackers do.
Scan Now