CWE-1427
Improper Neutralization of Input Used for LLM Prompting
Description
The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.
{"xhtml:p":["\n\t\t\t When prompts are constructed using externally controllable data, it is\noften possible to cause an LLM to ignore the original guidance provided by\nits creators (known as the \"system prompt\") by inserting malicious\ninstructions in plain human language or using bypasses such as special\ncharacters or tags. Because LLMs are designed to treat all instructions as\nlegitimate, there is often no way for the model to differentiate between\nwhat prompt language is malicious when it performs inference and returns\ndata. Many LLM systems incorporate data from other adjacent products or\nexternal data sources like Wikipedia using API calls and retrieval\naugmented generation (RAG). Any external sources in use that may contain\nuntrusted data should also be considered potentially malicious. \n\t\t\t "]}
Parent Weaknesses (ChildOf)
Common Consequences
Scope
Impact
Execute Unauthorized Code or Commands, Varies by Context
Scope
Impact
Read Application Data
Scope
Impact
Modify Application Data, Execute Unauthorized Code or Commands
Scope
Impact
Read Application Data, Modify Application Data, Gain Privileges or Assume Identity
Potential Mitigations
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.
During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.
CVE-2023-32786Chain: LLM integration framework has prompt injection (CWE-1427) that allows an attacker to force the service to retrieve data from an arbitrary URL, essentially providing SSRF (CWE-918) and potentially injecting content into downstream tasks.
CVE-2024-5184ML-based email analysis product uses an API service that allows a malicious user to inject a direct prompt and take over the service logic, forcing it to leak the standard hard-coded system prompts and/or execute unwanted prompts to leak sensitive data.
CVE-2024-5565Chain: library for generating SQL via LLMs using RAG uses a prompt function to present the user with visualized results, allowing altering of the prompt using prompt injection (CWE-1427) to run arbitrary Python code (CWE-94) instead of the intended visualization code.
CVE-2024-48746AI-based integration with business intel dashboard allows prompt injection through its natural language component, allowing execution of arbitrary code
Applicable Platforms
Security Training
Train your team to recognize and prevent security threats with our comprehensive security awareness program.
Start TrainingVulnerability Scanning
Discover vulnerabilities in your applications and infrastructure before attackers do.
Scan Now