leondz/garak issues and pull requests

#685 - convert module-level constant names to uppercase

Pull Request - State: closed - Opened by leondz 4 months ago

#683 - probe: persuasive jailbreak

Issue - State: open - Opened by leondz 4 months ago
Labels: probes, new plugin

#682 - Generator streamlining, docs

Pull Request - State: closed - Opened by leondz 4 months ago
Labels: documentation, generators

#681 - inaccurate detection results for FalseAssertion50

Issue - State: open - Opened by JKL98ISR 4 months ago - 1 comment
Labels: bug, detectors

#680 - documentation: make contributing doc really clear

Issue - State: open - Opened by leondz 4 months ago
Labels: documentation

#679 - documentation: describe generator program flow

Issue - State: closed - Opened by leondz 4 months ago - 1 comment
Labels: documentation, duplicate, generators

#678 - handle target model returning None in atkgen

Pull Request - State: closed - Opened by leondz 4 months ago
Labels: bug, probes

#677 - Guard cli run with invalid or incomplete config

Pull Request - State: closed - Opened by jmartin-tech 4 months ago - 1 comment

#672 - support for Metal/MPS

Issue - State: closed - Opened by leondz 5 months ago
Labels: enhancement, mac

#665 - handle extant but closed `hitlogfile` file

Pull Request - State: closed - Opened by leondz 5 months ago - 1 comment

#662 - documentation: generation process

Issue - State: closed - Opened by leondz 5 months ago
Labels: documentation, generators

#659 - OpenAIGenerator throws pickling exception after single-attempt probes when `parallel_attempts` > 1

Issue - State: closed - Opened by leondz 5 months ago - 4 comments
Labels: bug, generators

#656 - remove `generations` count param from generator constructors

Issue - State: open - Opened by leondz 5 months ago
Labels: enhancement, tests

#654 - generator list missing some HF generators

Issue - State: closed - Opened by leondz 5 months ago - 1 comment
Labels: bug, generators, cli

#646 - `_config` object contains `state` not suitable for multiprocessing

Issue - State: open - Opened by jmartin-tech 5 months ago - 1 comment
Labels: bug, enhancement

#644 - support chat modality in attempts

Pull Request - State: closed - Opened by leondz 5 months ago - 2 comments
Labels: enhancement

#638 - Feature/azure

Pull Request - State: open - Opened by Sunforger 5 months ago - 5 comments
Labels: generators, new plugin

#628 - improve new contributor journey - good first issues

Issue - State: open - Opened by leondz 5 months ago
Labels: housekeeping

#619 - skip ctrl-c capture if verbose mode is on

Issue - State: closed - Opened by leondz 5 months ago - 3 comments
Labels: enhancement, good first issue

#617 - add ArtPrompt probes

Pull Request - State: open - Opened by jmartin-tech 5 months ago - 3 comments
Labels: probes, new plugin

#610 - verify + improve mitigation detector

Issue - State: open - Opened by leondz 5 months ago - 1 comment
Labels: detectors, quality-accuracy

#595 - update the configuration machinery to take class-level info

Issue - State: closed - Opened by leondz 6 months ago - 1 comment
Labels: enhancement

#593 - tests: test garak's loading performance

Issue - State: open - Opened by leondz 6 months ago - 1 comment
Labels: enhancement, tests, quality-speed

#592 - get black stable

Issue - State: open - Opened by leondz 6 months ago
Labels: housekeeping

#589 - Configurable `Generator` parameters

Issue - State: closed - Opened by erickgalinkin 6 months ago - 13 comments
Labels: bug, enhancement, generators

#583 - probe: simple adaptive attacks

Issue - State: open - Opened by leondz 6 months ago
Labels: good first issue, probes, new plugin

#552 - `openai` generators should fail nicer when `--model_name` unspecified

Issue - State: open - Opened by leondz 6 months ago - 1 comment
Labels: enhancement, generators

#542 - generator: ollama

Issue - State: open - Opened by leondz 7 months ago
Labels: good first issue, generators, new plugin

#530 - probe: BEAST / Fast Adversarial Attacks on Language Models In One GPU Minute

Issue - State: closed - Opened by leondz 7 months ago
Labels: probes, new plugin

#516 - probe: beavertails

Issue - State: open - Opened by leondz 7 months ago
Labels: good first issue, probes, new plugin

#508 - Build on windows failing due to ecoji lib dependency

Issue - State: open - Opened by jnpushkar1507 7 months ago - 11 comments
Labels: bug, windows

#479 - put garak data all in one place

Issue - State: open - Opened by leondz 7 months ago - 6 comments
Labels: enhancement

#449 - docs: buffs

Issue - State: open - Opened by leondz 8 months ago
Labels: documentation, buffs

#361 - fail more helpfully when parallelism not supported

Issue - State: open - Opened by leondz 10 months ago
Labels: bug, ux

#352 - enable comparison of two runs

Issue - State: closed - Opened by leondz 10 months ago - 1 comment
Labels: duplicate, reporting

#338 - test `parallel_attempts`

Issue - State: open - Opened by leondz 10 months ago
Labels: tests

#312 - RTP prompt counts > number of lines in the files

Issue - State: closed - Opened by leondz 10 months ago - 1 comment
Labels: bug, probes

#311 - log a clear error message if a requested probe is not known

Issue - State: closed - Opened by leondz 10 months ago - 1 comment
Labels: duplicate, enhancement

#282 - probe: repeated char injection

Issue - State: open - Opened by leondz about 1 year ago
Labels: good first issue, probes, new plugin

#280 - probe: advbench / nlp security test

Issue - State: closed - Opened by leondz about 1 year ago - 1 comment
Labels: probes, new plugin

#278 - check what's going on with the cloze task when [MASK] is very early in the sentence

Issue - State: open - Opened by leondz about 1 year ago
Labels: enhancement, probes, quality-accuracy

#263 - generator: anthropic

Issue - State: open - Opened by leondz about 1 year ago - 3 comments
Labels: generators, new plugin

#259 - probe: ruby package hallucination

Issue - State: open - Opened by leondz about 1 year ago
Labels: good first issue, probes

#247 - test: add black test into tests

Issue - State: open - Opened by leondz about 1 year ago
Labels: tests, housekeeping

#200 - option to see `atkgen` auto red-team turns

Issue - State: open - Opened by leondz about 1 year ago - 1 comment
Labels: good first issue, probes

#199 - device-side assert with long inputs & huggingface generators

Issue - State: open - Opened by leondz about 1 year ago - 3 comments
Labels: bug

#198 - "device" as cli/config param

Issue - State: closed - Opened by leondz about 1 year ago - 1 comment
Labels: duplicate, good first issue, cli

#191 - fail if non-existing plugin is requested

Issue - State: closed - Opened by leondz about 1 year ago - 3 comments
Labels: cli, ux

#175 - generator: fastchat support

Issue - State: open - Opened by leondz over 1 year ago
Labels: generators, new plugin

#173 - HF peft support

Issue - State: open - Opened by leondz over 1 year ago - 1 comment
Labels: generators, new plugin

#161 - test: plugin documentation is present

Issue - State: open - Opened by leondz over 1 year ago - 3 comments
Labels: documentation, good first issue, tests

#120 - get `--plugin_info` working with openai (+other) plugins

Issue - State: open - Opened by leondz over 1 year ago - 2 comments
Labels: bug

#109 - Support chat modality

Issue - State: closed - Opened by leondz over 1 year ago - 6 comments
Labels: enhancement

#108 - probe: next batch of Language Model Risk Cards

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes

#100 - probe: misinformation seeding

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#99 - probe: LM_Memorization

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#98 - probe: scour giuven95 collection for material

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#97 - probes: more goodside

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#96 - fix things that stop us `yield`ing results, prompts

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement

#95 - download tqdm is missing when downloading realtoxicityprompts for the first time

Issue - State: open - Opened by leondz over 1 year ago
Labels: bug, probes

#94 - hf generator load message only prints the part of the model name after /

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: bug, good first issue, generators

#93 - fix word-boundary matching in TextProbe

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: bug, good first issue, detectors

#92 - detector: UK-only risky words

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: good first issue, detectors, new plugin

#91 - factor functions out of probes.base.*

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: probes

#90 - DAN probes need a follow-up turn where capabilities are tested

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, quality-accuracy

#89 - probe: confidential/sensitive PII

Issue - State: open - Opened by leondz over 1 year ago - 2 comments
Labels: probes, detectors, new plugin

#88 - probe: credit cards

Issue - State: open - Opened by leondz over 1 year ago
Labels: good first issue, probes, new plugin

#87 - detector: violence

Issue - State: open - Opened by leondz over 1 year ago
Labels: good first issue, detectors, new plugin

#86 - add an `active` flag to allow marking plugins as inactive

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: probes, detectors, generators

#85 - config file

Issue - State: closed - Opened by leondz over 1 year ago - 4 comments
Labels: enhancement

#84 - remove Detector keyword from detector classes

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: good first issue, detectors

#83 - set up a global "prompts-per-probe" cap

Issue - State: open - Opened by leondz over 1 year ago
Labels: cli

#82 - unify & expose the "generation length" parameter

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: generators

#81 - detector: refutation

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement, detectors

#80 - detector: misogyny

Issue - State: open - Opened by leondz over 1 year ago - 1 comment
Labels: good first issue, detectors, new plugin

#79 - probes should include a verbose "goal" attribute

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: probes

#78 - evaluators should read from attempt objects

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement

#77 - provide multiple cli output modes

Issue - State: open - Opened by leondz over 1 year ago
Labels: enhancement, cli

#76 - probe: prompt revelation attacks

Issue - State: open - Opened by leondz over 1 year ago - 1 comment
Labels: probes, new plugin

#75 - probe: obfuscation, injection, virtualisation

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#74 - probe: injection with tags

Issue - State: open - Opened by leondz over 1 year ago
Labels: good first issue

#73 - probe: injection in whois

Issue - State: open - Opened by leondz over 1 year ago
Labels: good first issue, probes, new plugin

#72 - probe: swyx techniques

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#71 - probe: lmrc

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement, probes

#70 - probe: fake references

Issue - State: open - Opened by leondz over 1 year ago
Labels: probes, new plugin

#69 - probe: misleading questions

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: probes, new plugin

#68 - throw exception if plugin invoked where key is missing

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement

#67 - detector: mitigations

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement, good first issue, detectors, new plugin

#66 - print (probe|detector|generator) description at the side of the class name when listing

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement, cli

#65 - sync testing expectations with what --list* delivers

Issue - State: open - Opened by leondz over 1 year ago
Labels: enhancement

#64 - api call parallelism

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: duplicate, enhancement

#63 - add --detectors CLI option

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: enhancement, detectors, cli

#62 - hit log

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: enhancement

#61 - add typing hints around core functions

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement

#60 - generator: llama.cpp integration

Issue - State: closed - Opened by leondz over 1 year ago - 1 comment
Labels: enhancement, generators

#59 - log probe params at probe invocation

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: enhancement, probes, reporting

#58 - log generator params at invocation

Issue - State: closed - Opened by leondz over 1 year ago - 2 comments
Labels: enhancement, generators, reporting

#57 - add Replicate generator

Pull Request - State: closed - Opened by leondz over 1 year ago

#56 - add n-gram approx matching, default 4-gram

Pull Request - State: closed - Opened by leondz over 1 year ago

#55 - Dir layout

Pull Request - State: closed - Opened by leondz over 1 year ago

GitHub / leondz/garak issues and pull requests