Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / aws/fmeval issues and pull requests
#99 - test: Add integration tests for prompt stereotyping evaluation algorithm
Pull Request -
State: closed - Opened by rvasahu-amazon 11 months ago
#98 - test: implement integration tests for toxicity eval algorithm
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#97 - test: Add integration tests for General Semantic Robustness
Pull Request -
State: closed - Opened by rvasahu-amazon 11 months ago
#96 - fix: fix jmespaths with spaces
Pull Request -
State: closed - Opened by oyangz 11 months ago
#95 - fix: make BertscoreHelperModel a ray actor
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#94 - test: add sanity check for example notebooks
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
#93 - docs: updating readme
Pull Request -
State: closed - Opened by malhotra18 11 months ago
#92 - test: implement integration tests for summarization accuracy
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#91 - feat: update get_eval_algorithm to return instantiated algorithm
Pull Request -
State: closed - Opened by keerthanvasist 11 months ago
#90 - fix: address science report comments
Pull Request -
State: closed - Opened by oyangz 11 months ago
#89 - feat: adding example notebooks
Pull Request -
State: closed - Opened by RamVegiraju 11 months ago
#88 - fix: increase ABS_TOL to account for non-deterministic models
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#87 - fix: update notebooks to accommodate code changes
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
- 1 comment
#86 - fix: add original accuracy scores to semantic robustness output
Pull Request -
State: closed - Opened by oyangz 11 months ago
#85 - refactor: create fixtures for model runners used by integration tests
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#84 - revert: remove imdb reviews dataset and cnn daily mail dataset
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
#83 - fix: add qa accuracy semantic robustness to eval_algo_mapping.py
Pull Request -
State: closed - Opened by danielezhu 11 months ago
#82 - Example with pre-populated model outputs
Pull Request -
State: closed - Opened by RamVegiraju 11 months ago
#81 - fix: set default evaluation_algorithm_config
Pull Request -
State: closed - Opened by oyangz 11 months ago
#80 - fix: revert changes; remove endpoint urls
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
#79 - fix: determine endpoint urls based on constants
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
#78 - fix: import model_runner_util module
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#77 - test: update __eq__ method of EvalScore to use absolute tolerance
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#76 - feat: update sm/js model runner to optionally take session
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#75 - fix: fix category score sorting to take top 10 scores in plot
Pull Request -
State: closed - Opened by oyangz 12 months ago
#74 - fix: update notebooks for latest code changes
Pull Request -
State: closed - Opened by pinaraws 12 months ago
#73 - fix: use values in algo name map
Pull Request -
State: closed - Opened by pinaraws 12 months ago
#72 - fix: fix pipeline: set MAX_ROWS_TO_TAKE to 100000
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#71 - fix(get_dataset): dataset sampling for large datasets
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
- 1 comment
#70 - feat: update jumpstart model runner - create and use jumpstart composer and extractor
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
- 1 comment
#69 - feat: update output path in EvalOutput
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#68 - fix: update reporting utils to reflect new report design
Pull Request -
State: closed - Opened by oyangz 12 months ago
#67 - feat: removing seed as a parameter for semantic robustness eval configs
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#66 - test(TestQAAccuracySemanticRobustness): updating tolerance and expected results
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#65 - test: fix the tolerance value used by QA accuracy integ test
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#64 - test: fix the tolerance value used by QA accuracy integ test
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#63 - feat: adding default prompt templates
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#62 - feat: update data configs with real s3 paths
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
- 1 comment
#61 - test: implement integration test for QA accuracy semantic robustness
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#60 - fix: retry logic by set same default value for sagemaker/bedrock session
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#59 - refactor: refactor code in composers.py
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#58 - feat: implement StringContentComposer for x-text content type
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#57 - fix: add model_output to dataset for semantic robustness algos evaluate
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#56 - feat: implement summarization toxicity
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#55 - test: add integration test for qa accuracy algorithm
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#54 - feat: implement qa toxicity eval algo
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#53 - feat: add bedrock example notebook
Pull Request -
State: closed - Opened by pinaraws 12 months ago
- 1 comment
#52 - fix: removing incorrect optionals from eval configs
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#51 - refactor: rename semantic_preserving_perturbations to perturbation_ut…
Pull Request -
State: closed - Opened by malhotra18 12 months ago
- 1 comment
#50 - feat: add error parameter to EvalOutput
Pull Request -
State: closed - Opened by oyangz 12 months ago
#49 - ci: run integration tests using codebuild on main branch(test)
Pull Request -
State: closed - Opened by Satish615 12 months ago
#48 - feat: implement toxicity eval algo
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#47 - feat: add classification accuracy semantic robustness eval algo
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#46 - feat: add qa accuracy semantic robustness eval algo
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#45 - fix: raise error when get empty target_output_delimiter
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#44 - fix: update devtool all command so that it works when run from a freshly-cloned repo
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#43 - test: implement factual knowledge integration test
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#42 - feat: update ContentComposer to work well with json
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#41 - build(deps): bump urllib3 from 1.26.17 to 1.26.18
Pull Request -
State: closed - Opened by dependabot[bot] 12 months ago
Labels: dependencies
#40 - feat: implement summarization accuracy semantic robustness eval algo
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#39 - feat: add num_records as a parameter to evaluate method
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
- 1 comment
#38 - refactor: move refactor: move get_eval_algorithm under amazon_fmeval
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#37 - feat: allow setting parallelization factor as env
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#36 - feat: add support for vLLM API Server, for model runner inference
Pull Request -
State: closed - Opened by lorenzwl 12 months ago
#35 - feat: update prompt stereotyping to aggregate on boolean score
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#34 - perf: materialize code at the end of each component to avoid duplicat…
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#33 - feat: implement general semantic robustness
Pull Request -
State: closed - Opened by malhotra18 12 months ago
- 1 comment
#32 - fix: implement __reduce__ method for SageMakerModelRunner and JumpstartModelRunner
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#31 - fix: jumpstart models support more content types
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#30 - fix: update notebook to correct installation issues, correct model errors, improve usability
Pull Request -
State: closed - Opened by lorenzwl 12 months ago
#29 - fix: update notebook to correct installation issues, correct model errors, improve usability
Pull Request -
State: closed - Opened by lorenzwl 12 months ago
#28 - fix: update jumpstart notebook with correct content_type and output in model runner
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#27 - build: change install_poetry call in devtool functions
Pull Request -
State: closed - Opened by Satish615 12 months ago
- 2 comments
#26 - fix: remove womens clothing review dataset from supported datasets te…
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#25 - feat: re-set the model predictor and session after Ray actor is done
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#24 - fix: update generate_model_predict_response_for_dataset to use ray actors
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#23 - docs: add example notebook for jumpstart model runner
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#22 - fix: update bedrock model runner to work with empty jmespaths
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#21 - docs: added custom model evaluation notebook with chat gpt model runner
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#20 - ci: add github worflow to run unit tests
Pull Request -
State: closed - Opened by Satish615 12 months ago
#19 - refactor: summarization accuracy and unit tests
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#18 - feat: add evaluation with custom model notebook
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#17 - ci: add github workflow for lint_checks
Pull Request -
State: closed - Opened by Satish615 12 months ago
#16 - fix: update predict method of model runners to not extract when no jmespath is present
Pull Request -
State: closed - Opened by danielezhu 12 months ago
#15 - fix: update assert for eval score to be float or int (instead of just…
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#14 - Add Bedrock model runner
Pull Request -
State: closed - Opened by pinaraws 12 months ago
#13 - feat: add Classification Accuracy Eval Algorithm
Pull Request -
State: closed - Opened by xiaoyi-cheng 12 months ago
#12 - style: fix for ./devtool lint
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#11 - feat: update data config for built-in datasets
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#10 - feat: implemented bertscore for summarrization accuracy
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#9 - refactor: move all the project modules and files to amazon-fmeval dir
Pull Request -
State: closed - Opened by Satish615 12 months ago
#8 - Update urllib
Pull Request -
State: closed - Opened by pinaraws 12 months ago
#7 - refactor: reorganize all mappings, complete implementation get_eval_algorithm method
Pull Request -
State: closed - Opened by malhotra18 12 months ago
#6 - build(deps): bump urllib3 from 1.26.16 to 1.26.17
Pull Request -
State: closed - Opened by dependabot[bot] 12 months ago
- 1 comment
Labels: dependencies
#5 - feat: initial commit - amazon foundation model evaluations library
Pull Request -
State: closed - Opened by keerthanvasist 12 months ago
#4 - Add notebook
Pull Request -
State: closed - Opened by pinaraws about 1 year ago
#3 - Clean up
Pull Request -
State: closed - Opened by pinaraws about 1 year ago
#2 - Clean up
Pull Request -
State: closed - Opened by pinaraws about 1 year ago
#1 - Initial commit
Pull Request -
State: closed - Opened by pinaraws about 1 year ago