Releases: Azure/PyRIT
v0.4.0
What's Changed
- New Advanced Attack Techniques: Expanded orchestrators with advanced attack techniques, including PAIR, tree of attacks, and crescendo strategies.
- New Targets: Crucible target, Prompt Shield Target, Azure OpenAI GPT-4o target
- New Converters: Added Tense, Emoji, image to text, and Character Space converters.
- New Scorers: Scale Scorer, Prompt Shield, and True/False Inverter Scorer
- Automatic Scoring & Memory Labels: Introduced automatic scoring in the PromptSendingOrchestrator. Added support for scoring with user-provided memory labels.
- Delegation SAS Authentication: Supported delegation SAS authentication for secure interactions with Azure Blob Storage targets.
- Improved Resiliency: Enhanced the resiliency of targets, converters, and orchestrators with robust error handling mechanisms.
- Bug Fixes & Performance: Various bug fixes, added support for Python 3.12, speedup unit tests
- Fetch functionality: Introduced functionality to fetch adversarial datasets, such as SecLists, XStest etc.,
- Updated Demo Codes: Replaced demo code examples with the GPT-4o target.
Full List of Changes
- FIX: Fixing policheck bug by @rlundeen2 in #261
- release v0.3.0 by @jbolor21 in #265
- DOC: Adding Guidance on Incorporating Research by @rlundeen2 in #268
- FEAT: Adding Tense Converter by @rlundeen2 in #273
- [FEAT] Add Scoring to PromptSendingOrchestrator by @nina-msft in #262
- FIX Fixed mypy Type Failures by @elgertam in #269
- FEAT: Adding Crucible Target by @rlundeen2 in #277
- FIX ValueError with Azure TTS Target in Single Turn Conversation Using PromptSendingOrchestrator by @nina-msft in #278
- FEAT: Converter Tokens by @rlundeen2 in #279
- [FIX] Add flake8-copyright check to pre-commit hooks by @nina-msft in #281
- FIX Exclude Morse Converter from Flake8 Precommit by @nina-msft in #284
- [DRAFT] [FIX] Replace Orchestrator ID with UUID by @nina-msft in #285
- DOC update citation for past tense paper by @romanlutz in #288
- FEAT Add scale scorer by @romanlutz in #274
- FEAT Add Delegation SAS-Based Auth, Update Storage Plugins, and Async Blob Download by @rdheekonda in #286
- FEAT add (back) Gandalf scorer by @romanlutz in #287
- MAINT clean up copyright by @romanlutz in #297
- FEAT: Add Error Handling to AML Chat Target by @nina-msft in #299
- FIX: bug with multi-modal image responses by @rlundeen2 in #301
- MAINT: Improving some LLM Converters by @rlundeen2 in #300
- [FIX][Issue #302] update language version enforcement to fix black-pre-commit installation incompatibility by @shivenchawla in #303
- FEAT return ID in conversation duplication code by @romanlutz in #296
- [FEAT] Implement PAIR by @dlmgary in #255
- FEAT add float scale threshold scorer by @romanlutz in #294
- FEAT: Add GPT4-o chat target by @shivenchawla in #293
- FEAT: Adding Emoji Converter by @rlundeen2 in #306
- DOC: Doc Reorg by @rlundeen2 in #304
- MAINT: Removing asyncio sleep by @rlundeen2 in #309
- MAINT add support for Python 3.12 and fix tests that started breaking by @romanlutz in #305
- FEAT Add print_conversation method to prompt sending orchestrator by @romanlutz in #312
- FEAT Add many-shot jailbreaking feature implementation by @KutalVolkan in #254
- FEAT: Add tree of attacks with pruning by @salmazainana in #210
- FEAT Add Space Converter by @rdheekonda in #316
- FEAT Add Flexible Memory Labels and Scoring to Orchestrators by @rdheekonda in #315
- FEAT: Crescendo Orchestrator by @SafwanA02 in #275
- Feat: Adding multi-turn promptSendingOrchestrator by @rlundeen2 in #317
- DOC Fix README.md link by @romanlutz in #319
- MAINT: Fixing data serializer ability to properly raise errors by @rlundeen2 in #318
- FEAT: Add fetch function for SecLists AI LLM Bias Testing datasets (#267) by @KutalVolkan in #280
- FEAT: Adding true_false inverter scorer by @rlundeen2 in #321
- FIX: fixing check links by @rlundeen2 in #323
- FEAT: Add Exception Handling to Azure TTS Target by @nina-msft in #322
- DOC - replacing gpt4 with gpt4o in example notebooks by @jsong468 in #313
- [MAINT] Changing Examples from stop signs by @jbolor21 in #325
- FEAT Prompt Shield by @ValbuenaVC in #271
- FEAT: add xstest dataset by @KutalVolkan in #320
- [FEAT] Created add_image_text_converter and unit tests by @jsong468 in #328
- DOC: Adding Notebook to document re-sending previous prompts by @rlundeen2 in #332
- MAINT: speeding up crescendo tests by @rlundeen2 in #333
- FIX Move pillow from dev to core dependency by @rdheekonda in #334
- FIX add sample image classifier file by @jbolor21 in #336
- FEAT: Add deterministic flag and custom substitutions to LeetspeakConverter by @KutalVolkan in #329
- MAINT Remove Duplicate Module by @rdheekonda in #337
- MAINT Restructure pyrit.models module and prune by @romanlutz in #338
- [MAINT] Speeding up unit tests by @jbolor21 in #335
- FIX Crescendo backtrack with same orchestrator ID and handling responses with markdown syntax by @romanlutz in #340
New Contributors
- @shivenchawla made their first contribution in #303
- @KutalVolkan made their first contribution in #254
- @salmazainana made their first contribution in #210
- @jsong468 made their first contribution in #313
- @ValbuenaVC made their first contribution in #271
Full Changelog: v0.3.0...v0.4.0
v0.3.0
What's Changed
- New and improved scorers! Many new scorers have been added, and scorers can now be swapped out and made generic.
- Many new attack techniques and variations have been introduced. These include skeleton key, most of GPTFuzz, adding text to images, repeated token attack, cipherchat, shorten/expand, tone, CodeChameleon, and more. A total of 13 new converters have been added!
- Framework improvements:
- Ability to duplicate conversations for reuse (this makes implementation easier for attacks like PAIR/TAP/crescendo).
- Converters can be added to LLM responses.
- All framework calls are now async and parallelizable.
- Error handling and intelligent automatic retries in targets (e.g., for network errors) and converters/scorers (e.g., for JSON deserialization).
Full list of Changes
- FEAT: Refactoring and Standardizing Scores and Scorers by @rlundeen2 in #190
- FIX: Making RESULTS_PATH be simple in pip packages by @rlundeen2 in #191
- FIX: Minor Self-Ask Scorer Improvements by @rlundeen2 in #194
- FEAT: Adding Scores to the Database by @rlundeen2 in #195
- MAINT use context manager in XPIA notebook by @romanlutz in #198
- FEAT: Update score_async to add score to database by @rlundeen2 in #200
- FEAT support duplicating memory when cloning orchestrators by @romanlutz in #177
- MAINT: Likert Scoring Tweaks to Reduce False Positives by @rlundeen2 in #201
- FEAT add CSV support by @romanlutz in #197
- FEAT: Adding Human in the Loop Scorer by @rlundeen2 in #202
- FEAT: Azure content filter scorer by @cseifert1 in #206
- FEAT Adding Image Converter: add text on image by @jbolor21 in #205
- FEAT: Score Prompts Orchestrator by @rlundeen2 in #208
- MAINT: Deprecated send_prompt methods by @mart123p in #204
- FEAT Add image generation example with red teaming orchestrator and unify existing orchestrator definitions by @romanlutz in #189
- FEAT: self ask conversation objective and verifier scorer for crescendo by @cseifert1 in #209
- FEAT: Centralize Exception Handling and Implement in GPTv Target by @rdheekonda in #207
- MAINT Making Prompt Converters Async by @jbolor21 in #211
- Update .env_example Typo "Azure Open AI"→"Azure OpenAI" by @hyoshioka0128 in #214
- MAINT: Small scoring updates by @rlundeen2 in #215
- MAINT: Adding pretty print functionality and small RTO updates by @rlundeen2 in #217
- DOC: Re-organizing documentation by @rlundeen2 in #219
- FEAT: Add Suffix Converter by @NaijingGuo in #212
- MAINT: Updating GPT-V to use new exception guide by @rlundeen2 in #220
- FEAT: Add nesting and prepend/append jailbreaks from papers by @jl8771 in #216
- MAINT Adding Error Handling to OpenAIChatInterface by @jbolor21 in #218
- MAINT Add Exception Handling to DALLE Target by @rdheekonda in #221
- FEAT: Add repeated token attack converter by @jl8771 in #224
- MAINT Resolve Install Issues and Add Multiline Text Wrapping in AddTextImageConverter by @rdheekonda in #230
- MAINT: PromptRequestPiece SHA setting update by @rlundeen2 in #231
- FEAT: Implements Crescendo-style attack based on system prompt. by @dlmgary in #237
- MAINT add notebook version disclaimer by @romanlutz in #234
- FEAT: Adding Converters to Output by @rlundeen2 in #236
- DOC: Reorganizing MemoryDocs by @rlundeen2 in #239
- Added complex code jailbreak template by @petebryan in #238
- FEAT: Add prompt converters for atbash, caesar, morse and cipherchat from paper by @jl8771 in #223
- MAINT add test instructions to release guide by @romanlutz in #232
- FIX: Fixing doc links by @rlundeen2 in #245
- FEAT: Adding Master Key Jailbreak by @SafwanA02 in #248
- MAINT Adding Error Handling Code for converters by @jbolor21 in #247
- FIX: Fixing score conversation history by @rlundeen2 in #251
- FEAT: Add shorten/expand converters by @jl8771 in #246
- FEAT: Add CodeChameleon converter by @jl8771 in #240
- FEAT: Adding Noise and Tone Converters by @rlundeen2 in #252
- FEAT: Add persuasion converter with 5 persuasion techniques by @jl8771 in #253
- FEAT Implementation of SQL Server connectivity by @elgertam in #227
- MAINT Error Handling for Scorers by @jbolor21 in #256
- FIX: Skeleton Key Orchestrator by @SafwanA02 in #260
- MAINT upgrading AOAI version by @jbolor21 in #264
New Contributors
- @mart123p made their first contribution in #204
- @hyoshioka0128 made their first contribution in #214
- @jl8771 made their first contribution in #216
- @SafwanA02 made their first contribution in #248
- @elgertam made their first contribution in #227
Full Changelog: v0.2.1...v0.3.0
v0.2.1
What's Changed
- added user authentication support for AOAI Chat Targets
- request validation in targets
- support for exporting conversations from the memory
Full list of changes
- Updating Release to 0.2.1.dev0 by @rlundeen2 in #181
- FEAT Add User AuthN Support to AOAI Chat Targets by @nina-msft in #182
- MAINT Add Request Validation for All Prompt Targets by @rdheekonda in #184
- FEAT Export Conversation by Orchestrator ID by @nina-msft in #183
Full Changelog: v0.2.0...v0.2.1
v0.2.0
What's Changed
- Multi-modal support: You can now input/output various multi-modal targets.
- XPIA support: Enabling easier second order prompt injection attacks.
- A more robust local (duckDB) database: Allowing querying and inserting previous conversations.
Full list of Changes
- Added a ChatMessageNormalizer that formats messages in the template specified by a Hugging Face tokenizer by @blakebullwinkel in #128
- PromptMemoryEntry Table Added for more Extensible Target Logic by @rlundeen2 in #125
- Added prompt softener prompt converter by @cseifert1 in #132
- Dataset Organization and Adding Public Jailbreaks by @rlundeen2 in #131
- Adding Image Target by @jbolor21 in #118
- Adding more authentication methods, add capital letters converter by @pgrek001 in #139
- Add cross-domain prompt injection orchestrator by @romanlutz in #127
- Added support to target an Ollama endpoint as a prompt chat target by @uskr in #141
- Normalizer multi modal/flexible support refactor by @rlundeen2 in #143
- Adding Identifiers to Memory by @rlundeen2 in #145
- Adding Data Type Normalizer Helpers by @rlundeen2 in #147
- Updating run_jupytext to cache notebooks that previously passed by @rlundeen2 in #148
- Gandalf through level7 by @jorisdg in #152
- Adding Multi-Modal Output Support to Converters by @rlundeen2 in #155
- Adding TTS Target by @rlundeen2 in #161
- Updating Gandalf Target to be more clear by @rlundeen2 in #153
- Support python 3.11 by @romanlutz in #168
- New Converters: Replace Whitespace and Leetspeak by @jbolor21 in #162
- Refactored SelfAskGptClassifier into SelfAskScore class and added Likert scale scoring by @blakebullwinkel in #154
- Fix mypy issues, convert Azure completion class to target, fix AOAI and OAI tests, remove clip embedding class by @romanlutz in #172
- Converter for prompt text to audio by @pgrek001 in #149
- Updating PromptSendingOrchestrator to handle multi-modal by @rlundeen2 in #174
- Generalize XPIA orchestrator by @romanlutz in #163
- Add Several Content Classifiers by @nina-msft in #175
- Add AzureOpenAIGPTVChatTarget to Support MultiModal by @rdheekonda in #160
- Refactoring Dalle Target to support database by @jbolor21 in #156
New Contributors
- @blakebullwinkel made their first contribution in #128
- @cseifert1 made their first contribution in #132
- @libera826 made their first contribution in #133
- @NaijingGuo made their first contribution in #140
- @uskr made their first contribution in #141
- @jorisdg made their first contribution in #144
Full Changelog: v0.1.2...v0.2.0
v0.1.2
What's Changed
Big changes this release include solidifying the orchestrator
, converter
, target
model for attacks, and migrating the local memory storage from a JSON file to a DuckDB instance.
The first two demos have been updated with the new architecture, and two new demos have been added; send all prompts and using prompt converters
Full List of Changes
- FEAT: Adding StringJoinConverter by @rlundeen2 in #70
- DOC: Add release instructions by @romanlutz in #57
- FEAT: Chain Prompt Converters in Normalizer by @rlundeen2 in #73
- FEAT: Adding Support for 1:N PromptConverters by @rlundeen2 in #75
- FEAT: Adding NoOpTarget by @rlundeen2 in #79
- FEAT: Added converter for ascii art by @petebryan in #81
- FEAT: Add rot13 by @pgrek001 in #80
- FEAT: Adding Batch/Async Processing to PromptTargets by @rlundeen2 in #91
- FEAT: add support for chat messages dataset by @dlmgary in #90
- DOC: Release guidelines and PR template update by @romanlutz in #92
- DOC: Adding Docs for ChatMessageNormalizer by @rlundeen2 in #93
- FEAT: Prompt Variation Converter by @jbolor21 in #86
- DOC: Adding Converter Docs and Demos by @rlundeen2 in #100
- MAINT: Add red teaming orchestrators to replace RedTeamingBot by @romanlutz in #84
- FEAT: Making prompt_nop_target into a stream target by @rlundeen2 in #99
- MAINT: Adding orchestrator abstract base class by @rlundeen2 in #102
- FIX: simplify flow in red teaming orchestrator code by @romanlutz in #105
- DOC: use google style docstrings by @romanlutz in #104
- DOC: add short guide on how to handle stale PRs & introduce standardized prefixes by @romanlutz in #101
- FEAT: Language Translation Converter by @rlundeen2 in #106
- FEAT: Add scalable and efficient memory by @rdheekonda in #97
- FEAT: add support for question answering benchmark by @dlmgary in #94
- FEAT: New prompt target: AzureBlobStorageTarget by @nina-msft in #95
- FEAT: Add UTR39 confusability converter by @yonatanzunger in #115
- MAINT: Refactoring AzureOpenAIChat to only be a promptTarget by @rlundeen2 in #114
- FEAT: Add support to OpenAI API to use official or custom endpoints by @friyin in #65
- FEAT: Migrating Azure ML to PromptTarget by @rlundeen2 in #113
- Various bug fixes and smaller documentation updates by the AI Red Team
New Contributors
- @eltociear made their first contribution in #67
- @petebryan made their first contribution in #81
- @pgrek001 made their first contribution in #80
- @yonatanzunger made their first contribution in #115
- @friyin made their first contribution in #65
Full Changelog: v0.1.1...v0.1.2
v0.1.1
What's Changed
The previous release 0.1.0 did not include the datasets used in the example notebooks. Version 0.1.1 addresses this.
Full list of changes
- Cleanup notebook outputs by @rdheekonda in #56
- Fixing bug with pypi package directory by @rlundeen2 in #60
- Add Details to Contributor Guide About Forking/PRs by @nina-msft in #59
- Fix hyperlinks and updated package version by @rdheekonda in #58
- add datasets folder to wheel by @romanlutz (#61)
Full Changelog: https://github.com/Azure/PyRIT/commits/v0.1.1
v0.1.0
What's Changed
This is the first release of PyRIT to PyPI including basic documentation and notebooks.
Full list of changes
- Update notebook dependency due to vulnerabilities by @romanlutz in #26
- Add article in SECURITY.md by @romanlutz in #27
- Remove poetry by @romanlutz in #24
- add NOTICE file by @romanlutz in #28
- Change README content to say "prompt injection attacks" by @romanlutz in #29
- Update Demo Notebook Prerequisites by @rdheekonda in #30
- Update chat client & multiturn notebooks with more information by @romanlutz in #25
- Remove unused definitions from models.py by @romanlutz in #31
- Update pyproject.toml to find packages/modules by @romanlutz in #33
- Prompt Normalizer and Refactor by @rlundeen2 in #32
- Export requirements before component governance is run by @romanlutz in #35
- fixing bug with red_teaming_bot by @rlundeen2 in #34
- Fixing bug with RedTeamingBot attack strategy by @rlundeen2 in #36
- Standardize Endpoints and API Keys for AML Chat by @rlundeen2 in #37
- Added Flag character transformer for invisible prompts. by @ChristBian1 in #38
- Add Jupyter Dependencies + aiohttp to pyproject.toml by @nina-msft in #39
- AML Endpoint Bug Fix and Refactor by @rlundeen2 in #40
- Add jailbreakchat prompt templates and modify the structure of existing prompts by @romanlutz in #42
- Fix link to How to Guide by @romanlutz in #41
- fixing bug where default was set to debug by @rlundeen2 in #44
- build: update versions of pre-commit hooks by @dlmgary in #47
- Integrate Hugging Face Models with Azure ML: Download, Register, Deploy, and Test by @rdheekonda in #43
- Standardizing environment for completions and embeddings by @rlundeen2 in #46
- Adding Chat Message Normalizers so AML can work with more Models by @rlundeen2 in #45
- Improving multi-turn strategy by @rlundeen2 in #48
- Add ipykernel dependency package by @rdheekonda in #50
- Refactoring Docs to use JupyText by @rlundeen2 in #49
- Replace old graphics with new ones by @romanlutz in #53
- Add learn link to README by @romanlutz in #55
- Convert PyRIT Framework How to Guide to jupytext by @nina-msft in #51
- Move wiki contents into doc folder, update references by @romanlutz in #54
- Update HF Azure ML deployment managed compute to allow liveness probe configuration by @rdheekonda in #52
New Contributors
- @ChristBian1 made their first contribution in #38
Full Changelog: https://github.com/Azure/PyRIT/commits/v0.1.0