Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / aws/fmeval issues and pull requests

#98 - test: implement integration tests for toxicity eval algorithm

Pull Request - State: closed - Opened by danielezhu 11 months ago

#97 - test: Add integration tests for General Semantic Robustness

Pull Request - State: closed - Opened by rvasahu-amazon 11 months ago

#96 - fix: fix jmespaths with spaces

Pull Request - State: closed - Opened by oyangz 11 months ago

#95 - fix: make BertscoreHelperModel a ray actor

Pull Request - State: closed - Opened by danielezhu 11 months ago

#94 - test: add sanity check for example notebooks

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago

#93 - docs: updating readme

Pull Request - State: closed - Opened by malhotra18 11 months ago

#92 - test: implement integration tests for summarization accuracy

Pull Request - State: closed - Opened by danielezhu 11 months ago

#90 - fix: address science report comments

Pull Request - State: closed - Opened by oyangz 11 months ago

#89 - feat: adding example notebooks

Pull Request - State: closed - Opened by RamVegiraju 11 months ago

#88 - fix: increase ABS_TOL to account for non-deterministic models

Pull Request - State: closed - Opened by danielezhu 11 months ago

#87 - fix: update notebooks to accommodate code changes

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago - 1 comment

#86 - fix: add original accuracy scores to semantic robustness output

Pull Request - State: closed - Opened by oyangz 11 months ago

#84 - revert: remove imdb reviews dataset and cnn daily mail dataset

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago

#83 - fix: add qa accuracy semantic robustness to eval_algo_mapping.py

Pull Request - State: closed - Opened by danielezhu 11 months ago

#82 - Example with pre-populated model outputs

Pull Request - State: closed - Opened by RamVegiraju 11 months ago

#81 - fix: set default evaluation_algorithm_config

Pull Request - State: closed - Opened by oyangz 11 months ago

#80 - fix: revert changes; remove endpoint urls

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago

#79 - fix: determine endpoint urls based on constants

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago

#78 - fix: import model_runner_util module

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#77 - test: update __eq__ method of EvalScore to use absolute tolerance

Pull Request - State: closed - Opened by danielezhu 12 months ago

#76 - feat: update sm/js model runner to optionally take session

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#75 - fix: fix category score sorting to take top 10 scores in plot

Pull Request - State: closed - Opened by oyangz 12 months ago

#74 - fix: update notebooks for latest code changes

Pull Request - State: closed - Opened by pinaraws 12 months ago

#73 - fix: use values in algo name map

Pull Request - State: closed - Opened by pinaraws 12 months ago

#72 - fix: fix pipeline: set MAX_ROWS_TO_TAKE to 100000

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#71 - fix(get_dataset): dataset sampling for large datasets

Pull Request - State: closed - Opened by keerthanvasist 12 months ago - 1 comment

#69 - feat: update output path in EvalOutput

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#68 - fix: update reporting utils to reflect new report design

Pull Request - State: closed - Opened by oyangz 12 months ago

#65 - test: fix the tolerance value used by QA accuracy integ test

Pull Request - State: closed - Opened by danielezhu 12 months ago

#64 - test: fix the tolerance value used by QA accuracy integ test

Pull Request - State: closed - Opened by danielezhu 12 months ago

#63 - feat: adding default prompt templates

Pull Request - State: closed - Opened by malhotra18 12 months ago

#62 - feat: update data configs with real s3 paths

Pull Request - State: closed - Opened by keerthanvasist 12 months ago - 1 comment

#59 - refactor: refactor code in composers.py

Pull Request - State: closed - Opened by danielezhu 12 months ago

#58 - feat: implement StringContentComposer for x-text content type

Pull Request - State: closed - Opened by danielezhu 12 months ago

#56 - feat: implement summarization toxicity

Pull Request - State: closed - Opened by malhotra18 12 months ago

#55 - test: add integration test for qa accuracy algorithm

Pull Request - State: closed - Opened by danielezhu 12 months ago

#54 - feat: implement qa toxicity eval algo

Pull Request - State: closed - Opened by malhotra18 12 months ago

#53 - feat: add bedrock example notebook

Pull Request - State: closed - Opened by pinaraws 12 months ago - 1 comment

#52 - fix: removing incorrect optionals from eval configs

Pull Request - State: closed - Opened by malhotra18 12 months ago

#51 - refactor: rename semantic_preserving_perturbations to perturbation_ut…

Pull Request - State: closed - Opened by malhotra18 12 months ago - 1 comment

#50 - feat: add error parameter to EvalOutput

Pull Request - State: closed - Opened by oyangz 12 months ago

#49 - ci: run integration tests using codebuild on main branch(test)

Pull Request - State: closed - Opened by Satish615 12 months ago

#48 - feat: implement toxicity eval algo

Pull Request - State: closed - Opened by malhotra18 12 months ago

#47 - feat: add classification accuracy semantic robustness eval algo

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#46 - feat: add qa accuracy semantic robustness eval algo

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#45 - fix: raise error when get empty target_output_delimiter

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#43 - test: implement factual knowledge integration test

Pull Request - State: closed - Opened by danielezhu 12 months ago

#42 - feat: update ContentComposer to work well with json

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#41 - build(deps): bump urllib3 from 1.26.17 to 1.26.18

Pull Request - State: closed - Opened by dependabot[bot] 12 months ago
Labels: dependencies

#39 - feat: add num_records as a parameter to evaluate method

Pull Request - State: closed - Opened by keerthanvasist 12 months ago - 1 comment

#37 - feat: allow setting parallelization factor as env

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#36 - feat: add support for vLLM API Server, for model runner inference

Pull Request - State: closed - Opened by lorenzwl 12 months ago

#35 - feat: update prompt stereotyping to aggregate on boolean score

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#33 - feat: implement general semantic robustness

Pull Request - State: closed - Opened by malhotra18 12 months ago - 1 comment

#31 - fix: jumpstart models support more content types

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#27 - build: change install_poetry call in devtool functions

Pull Request - State: closed - Opened by Satish615 12 months ago - 2 comments

#23 - docs: add example notebook for jumpstart model runner

Pull Request - State: closed - Opened by danielezhu 12 months ago

#22 - fix: update bedrock model runner to work with empty jmespaths

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#20 - ci: add github worflow to run unit tests

Pull Request - State: closed - Opened by Satish615 12 months ago

#19 - refactor: summarization accuracy and unit tests

Pull Request - State: closed - Opened by malhotra18 12 months ago

#18 - feat: add evaluation with custom model notebook

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#17 - ci: add github workflow for lint_checks

Pull Request - State: closed - Opened by Satish615 12 months ago

#14 - Add Bedrock model runner

Pull Request - State: closed - Opened by pinaraws 12 months ago

#13 - feat: add Classification Accuracy Eval Algorithm

Pull Request - State: closed - Opened by xiaoyi-cheng 12 months ago

#12 - style: fix for ./devtool lint

Pull Request - State: closed - Opened by malhotra18 12 months ago

#11 - feat: update data config for built-in datasets

Pull Request - State: closed - Opened by keerthanvasist 12 months ago

#10 - feat: implemented bertscore for summarrization accuracy

Pull Request - State: closed - Opened by malhotra18 12 months ago

#8 - Update urllib

Pull Request - State: closed - Opened by pinaraws 12 months ago

#6 - build(deps): bump urllib3 from 1.26.16 to 1.26.17

Pull Request - State: closed - Opened by dependabot[bot] 12 months ago - 1 comment
Labels: dependencies

#4 - Add notebook

Pull Request - State: closed - Opened by pinaraws about 1 year ago

#3 - Clean up

Pull Request - State: closed - Opened by pinaraws about 1 year ago

#2 - Clean up

Pull Request - State: closed - Opened by pinaraws about 1 year ago

#1 - Initial commit

Pull Request - State: closed - Opened by pinaraws about 1 year ago