Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / aws/fmeval issues and pull requests
#328 - doc: update README.md to include set up challenges from virtual env
Pull Request -
State: open - Opened by kirupang-code about 1 month ago
#327 - build(deps) bump nltk from 3.8.1 to 3.9.1
Pull Request -
State: closed - Opened by danielezhu about 1 month ago
#326 - build: bump fmeval version to 1.2.0
Pull Request -
State: closed - Opened by kirupang-code about 1 month ago
#325 - doc: delete ROUGE/METEOR score from QAAccuracy documentation
Pull Request -
State: closed - Opened by kirupang-code about 2 months ago
#324 - build(deps): bump nltk from 3.8.1 to 3.9
Pull Request -
State: closed - Opened by dependabot[bot] about 2 months ago
- 2 comments
Labels: dependencies
#323 - build(deps): pinned nltk version to address build failure
Pull Request -
State: closed - Opened by kirupang-code about 2 months ago
#322 - Python 3.11 and 3.12 compatibility?
Issue -
State: open - Opened by OlivierBinette about 2 months ago
- 2 comments
#321 - build(deps): bump aiohttp from 3.9.5 to 3.10.2
Pull Request -
State: closed - Opened by dependabot[bot] about 2 months ago
- 1 comment
Labels: dependencies
#320 - [Fix] Updated notebook rendering as suggested in issue 316
Pull Request -
State: closed - Opened by polaschwoebel about 2 months ago
- 3 comments
#319 - fix: radar plot generation
Pull Request -
State: closed - Opened by xiaoyi-cheng about 2 months ago
- 1 comment
#318 - chore: update urllib3 version
Pull Request -
State: closed - Opened by kirupang-code 2 months ago
#317 - refactor: `SummarizationAccuracyMetrics` transform to handle multiple target outputs more efficiently
Pull Request -
State: closed - Opened by kirupang-code 2 months ago
#316 - [Bug] Radar plot using plotly not displaying
Issue -
State: closed - Opened by jcassiojr 2 months ago
#315 - feat: add `BERT_SCORE` to `QAAccuracySemanticRobustness`
Pull Request -
State: closed - Opened by kirupang-code 2 months ago
#314 - feat: Add `BERT_SCORE` to `QAAccuracy` and update unit/integration tests
Pull Request -
State: closed - Opened by kirupang-code 3 months ago
- 1 comment
#313 - build: bump fmeval version to 1.1.0
Pull Request -
State: closed - Opened by danielezhu 3 months ago
#312 - chore: rename factual knowledge scores
Pull Request -
State: closed - Opened by danielezhu 3 months ago
#311 - feat: update s3 data source for us-isof partition
Pull Request -
State: closed - Opened by oyangz 3 months ago
- 1 comment
#310 - fix: rename target/retrieved context to `context`, make it a list
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
#309 - fix: update how default payloads get extracted from model spec
Pull Request -
State: closed - Opened by danielezhu 3 months ago
#308 - fix: update how default payloads get extracted from model spec
Pull Request -
State: closed - Opened by danielezhu 3 months ago
#307 - feat: add configurable param logical_operator (OR/AND) to factual knowledge
Pull Request -
State: closed - Opened by kirupang-code 3 months ago
#306 - WIP: source code changes to fix default payloads issue.
Pull Request -
State: closed - Opened by danielezhu 3 months ago
#305 - feat: update context to take lists and rename context field
Pull Request -
State: closed - Opened by oyangz 3 months ago
- 1 comment
#304 - build(deps): bump zipp from 3.19.0 to 3.19.1
Pull Request -
State: closed - Opened by dependabot[bot] 3 months ago
- 4 comments
Labels: dependencies
#303 - build(deps): bump certifi from 2024.2.2 to 2024.7.4
Pull Request -
State: closed - Opened by dependabot[bot] 3 months ago
Labels: dependencies
#302 - feat: add `quasi_exact_inclusion` metric to factual knowledge; change `factual_knowledge` score name to `exact_inclusion`
Pull Request -
State: closed - Opened by kirupang-code 3 months ago
- 1 comment
#301 - fix: register placeholder_to_record_key in GeneratePrompt transform
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
- 1 comment
#300 - feat: add validate_prompt_template util
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
#299 - fix: Context precision score computation
Pull Request -
State: closed - Opened by awsvmaringa 3 months ago
- 1 comment
#298 - feat: faithfulness handle no statements edge case
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
#297 - feat: add error field in EvalScore
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
#296 - build(deps-dev): bump pdoc from 14.5.0 to 14.5.1
Pull Request -
State: closed - Opened by dependabot[bot] 3 months ago
Labels: dependencies
#295 - feat: add answer relevance algo
Pull Request -
State: closed - Opened by xiaoyi-cheng 3 months ago
- 2 comments
#294 - feat: add col_to_validate to evaluate_dataset
Pull Request -
State: closed - Opened by oyangz 4 months ago
- 1 comment
#293 - feat: support embedding model runner
Pull Request -
State: closed - Opened by xiaoyi-cheng 4 months ago
#292 - build(deps): bump urllib3 from 1.26.18 to 1.26.19
Pull Request -
State: closed - Opened by dependabot[bot] 4 months ago
- 2 comments
Labels: dependencies
#291 - feat: add faithfulness eval algo
Pull Request -
State: closed - Opened by xiaoyi-cheng 4 months ago
#290 - merge main into rageval branch
Pull Request -
State: closed - Opened by oyangz 4 months ago
- 1 comment
#289 - feat: add context precision for context quality eval algo
Pull Request -
State: closed - Opened by oyangz 4 months ago
- 2 comments
#288 - feat: modify GeneratePrompt transform to take placeholder_dict
Pull Request -
State: closed - Opened by xiaoyi-cheng 4 months ago
#287 - build(deps): bump tornado from 6.4 to 6.4.1
Pull Request -
State: closed - Opened by dependabot[bot] 4 months ago
- 1 comment
Labels: dependencies
#286 - ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead.
Issue -
State: closed - Opened by aakash086 4 months ago
- 4 comments
#285 - build: bump fmeval version to 1.0.4
Pull Request -
State: closed - Opened by danielezhu 4 months ago
#284 - [Feature] Add callback mechanism to evaluation
Issue -
State: open - Opened by acere 4 months ago
- 1 comment
#283 - Support multiple data configs in evaluate
Pull Request -
State: closed - Opened by athewsey 4 months ago
- 1 comment
#282 - Fix comparisons for string enumerations
Pull Request -
State: closed - Opened by athewsey 4 months ago
- 2 comments
#281 - feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs
Pull Request -
State: closed - Opened by keerthanvasist 4 months ago
- 1 comment
#280 - [Feature] Add system metrics collected during evaluation to eval_output
Issue -
State: open - Opened by acere 4 months ago
- 4 comments
#279 - build(deps): bump requests from 2.31.0 to 2.32.0
Pull Request -
State: closed - Opened by dependabot[bot] 5 months ago
- 4 comments
Labels: dependencies
#278 - [Feature] Streaming results/progress and summary metrics for faster feedback
Issue -
State: open - Opened by athewsey 5 months ago
- 1 comment
#277 - chore: update BarPlotCell unit test
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#276 - chore: add support for proprietary models
Pull Request -
State: closed - Opened by shrestha-bikash 5 months ago
#275 - build: bump fmeval version to 1.0.3
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#274 - fix: update pinned sagemaker python sdk version and get_user_agent_extra util function
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#273 - feat: allow placeholder dict for prompt composer
Pull Request -
State: closed - Opened by xiaoyi-cheng 5 months ago
#272 - build(deps): bump jinja2 from 3.1.3 to 3.1.4
Pull Request -
State: closed - Opened by dependabot[bot] 5 months ago
Labels: dependencies
#271 - build(deps): bump tqdm from 4.66.2 to 4.66.3
Pull Request -
State: closed - Opened by dependabot[bot] 5 months ago
Labels: dependencies
#270 - build(deps): bump sagemaker from 2.216.1 to 2.218.0
Pull Request -
State: closed - Opened by dependabot[bot] 5 months ago
- 2 comments
Labels: dependencies
#269 - [Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency
Issue -
State: open - Opened by athewsey 5 months ago
- 1 comment
#268 - [Feature] JSON export/import for EvalOutput classes
Issue -
State: open - Opened by athewsey 5 months ago
- 3 comments
#267 - feat: fetch log probability jmespath from JS metadata
Pull Request -
State: closed - Opened by keerthanvasist 5 months ago
#266 - feat: add target_context to dataset columns
Pull Request -
State: closed - Opened by oyangz 5 months ago
- 2 comments
#265 - docs: update telemetry-related info in README and docstrings
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#264 - fix: fix patching in unit tests
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#263 - build: bump fmeval version to 1.0.2
Pull Request -
State: closed - Opened by danielezhu 5 months ago
- 1 comment
#262 - feat: add fmeval-specific user agent header to botocore config for telemetry purposes
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#261 - docs: add syntax highlighting
Pull Request -
State: closed - Opened by connorads 5 months ago
- 1 comment
#260 - docs: create Github Actions workflow for generating docs via pdoc
Pull Request -
State: closed - Opened by danielezhu 5 months ago
#259 - test: update matplotlib version and figure cell init test
Pull Request -
State: closed - Opened by oyangz 5 months ago
#258 - chore: update lib versions based on dependabot recommendation
Pull Request -
State: closed - Opened by keerthanvasist 6 months ago
#257 - build(deps): bump aiohttp from 3.9.3 to 3.9.4
Pull Request -
State: closed - Opened by dependabot[bot] 6 months ago
- 2 comments
Labels: dependencies
#256 - chore: simplify botocore/boto3-related util code
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#255 - build: bump fmeval version to 1.0.1
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#254 - fix: update output record key validation logic in validate_call
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#253 - Revert "build: bump fmeval version to 1.0.1"
Pull Request -
State: closed - Opened by oyangz 6 months ago
#252 - fix: fix logic in evaluate_dataset to handle BYO inference outputs use case
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#251 - build: bump fmeval version to 1.0.1
Pull Request -
State: closed - Opened by oyangz 6 months ago
#250 - build(deps): bump idna from 3.6 to 3.7
Pull Request -
State: closed - Opened by dependabot[bot] 6 months ago
- 2 comments
Labels: dependencies
#249 - build(deps): bump transformers from 4.37.2 to 4.38.0
Pull Request -
State: closed - Opened by dependabot[bot] 6 months ago
- 2 comments
Labels: dependencies
#248 - docs: update README with details on contributing new eval algos
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#247 - [Feature] Image fields for multi-modal models
Issue -
State: open - Opened by athewsey 6 months ago
- 1 comment
#246 - [Feature] Multi-variable prompt templates
Issue -
State: open - Opened by athewsey 6 months ago
- 2 comments
#245 - fix: fix s3 uri validation for built-in datasets
Pull Request -
State: closed - Opened by oyangz 6 months ago
- 1 comment
#244 - build(deps): bump pillow from 10.2.0 to 10.3.0
Pull Request -
State: closed - Opened by dependabot[bot] 6 months ago
- 2 comments
Labels: dependencies
#243 - build: bump fmeval version to 1.0.0
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#242 - fix: set default region for boto3 client to access built-in datasets
Pull Request -
State: closed - Opened by oyangz 6 months ago
#241 - feat: update implementation of Toxicity to use Transform-based approach
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#240 - feat: update implementation of PromptStereotyping to use Transform-based approach
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#238 - feat: update implementation of FactualKnowledge to use Transform-based approach
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#236 - feat: update implementation of ClassificationAccuracy to use Transform-based approach
Pull Request -
State: closed - Opened by danielezhu 6 months ago
#227 - Toxicity transforms
Pull Request -
State: closed - Opened by franluca 7 months ago
- 1 comment
#221 - build(deps-dev): bump black from 23.7.0 to 24.3.0
Pull Request -
State: closed - Opened by dependabot[bot] 7 months ago
- 2 comments
Labels: dependencies
#212 - fix: update the default prompt templates for the built-in datasets
Pull Request -
State: closed - Opened by jmikko 7 months ago
#199 - Evaluation Algorithm for Recommendations
Issue -
State: closed - Opened by mccartni-aws 7 months ago
- 4 comments
#186 - String enums not comparing as expected
Issue -
State: closed - Opened by athewsey 8 months ago
- 3 comments
#167 - feat: conversion of targets to strings
Pull Request -
State: closed - Opened by franluca 9 months ago
- 2 comments
#165 - [Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")`
Issue -
State: open - Opened by athewsey 9 months ago
- 3 comments
#135 - feat(scientific correctness tests): test suite for scientific correctness
Pull Request -
State: closed - Opened by polaschwoebel 11 months ago
- 1 comment
#100 - refactor: rename package name to fmeval
Pull Request -
State: closed - Opened by xiaoyi-cheng 11 months ago
- 2 comments