Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / aws/fmeval issues and pull requests

#327 - build(deps) bump nltk from 3.8.1 to 3.9.1

Pull Request - State: closed - Opened by danielezhu about 1 month ago

#326 - build: bump fmeval version to 1.2.0

Pull Request - State: closed - Opened by kirupang-code about 1 month ago

#325 - doc: delete ROUGE/METEOR score from QAAccuracy documentation

Pull Request - State: closed - Opened by kirupang-code about 2 months ago

#324 - build(deps): bump nltk from 3.8.1 to 3.9

Pull Request - State: closed - Opened by dependabot[bot] about 2 months ago - 2 comments
Labels: dependencies

#323 - build(deps): pinned nltk version to address build failure

Pull Request - State: closed - Opened by kirupang-code about 2 months ago

#322 - Python 3.11 and 3.12 compatibility?

Issue - State: open - Opened by OlivierBinette about 2 months ago - 2 comments

#321 - build(deps): bump aiohttp from 3.9.5 to 3.10.2

Pull Request - State: closed - Opened by dependabot[bot] about 2 months ago - 1 comment
Labels: dependencies

#320 - [Fix] Updated notebook rendering as suggested in issue 316

Pull Request - State: closed - Opened by polaschwoebel about 2 months ago - 3 comments

#319 - fix: radar plot generation

Pull Request - State: closed - Opened by xiaoyi-cheng about 2 months ago - 1 comment

#318 - chore: update urllib3 version

Pull Request - State: closed - Opened by kirupang-code 2 months ago

#316 - [Bug] Radar plot using plotly not displaying

Issue - State: closed - Opened by jcassiojr 2 months ago

#315 - feat: add `BERT_SCORE` to `QAAccuracySemanticRobustness`

Pull Request - State: closed - Opened by kirupang-code 2 months ago

#314 - feat: Add `BERT_SCORE` to `QAAccuracy` and update unit/integration tests

Pull Request - State: closed - Opened by kirupang-code 3 months ago - 1 comment

#313 - build: bump fmeval version to 1.1.0

Pull Request - State: closed - Opened by danielezhu 3 months ago

#312 - chore: rename factual knowledge scores

Pull Request - State: closed - Opened by danielezhu 3 months ago

#311 - feat: update s3 data source for us-isof partition

Pull Request - State: closed - Opened by oyangz 3 months ago - 1 comment

#309 - fix: update how default payloads get extracted from model spec

Pull Request - State: closed - Opened by danielezhu 3 months ago

#308 - fix: update how default payloads get extracted from model spec

Pull Request - State: closed - Opened by danielezhu 3 months ago

#306 - WIP: source code changes to fix default payloads issue.

Pull Request - State: closed - Opened by danielezhu 3 months ago

#305 - feat: update context to take lists and rename context field

Pull Request - State: closed - Opened by oyangz 3 months ago - 1 comment

#304 - build(deps): bump zipp from 3.19.0 to 3.19.1

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago - 4 comments
Labels: dependencies

#303 - build(deps): bump certifi from 2024.2.2 to 2024.7.4

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago
Labels: dependencies

#301 - fix: register placeholder_to_record_key in GeneratePrompt transform

Pull Request - State: closed - Opened by xiaoyi-cheng 3 months ago - 1 comment

#300 - feat: add validate_prompt_template util

Pull Request - State: closed - Opened by xiaoyi-cheng 3 months ago

#299 - fix: Context precision score computation

Pull Request - State: closed - Opened by awsvmaringa 3 months ago - 1 comment

#298 - feat: faithfulness handle no statements edge case

Pull Request - State: closed - Opened by xiaoyi-cheng 3 months ago

#297 - feat: add error field in EvalScore

Pull Request - State: closed - Opened by xiaoyi-cheng 3 months ago

#296 - build(deps-dev): bump pdoc from 14.5.0 to 14.5.1

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago
Labels: dependencies

#295 - feat: add answer relevance algo

Pull Request - State: closed - Opened by xiaoyi-cheng 3 months ago - 2 comments

#294 - feat: add col_to_validate to evaluate_dataset

Pull Request - State: closed - Opened by oyangz 4 months ago - 1 comment

#293 - feat: support embedding model runner

Pull Request - State: closed - Opened by xiaoyi-cheng 4 months ago

#292 - build(deps): bump urllib3 from 1.26.18 to 1.26.19

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 2 comments
Labels: dependencies

#291 - feat: add faithfulness eval algo

Pull Request - State: closed - Opened by xiaoyi-cheng 4 months ago

#290 - merge main into rageval branch

Pull Request - State: closed - Opened by oyangz 4 months ago - 1 comment

#289 - feat: add context precision for context quality eval algo

Pull Request - State: closed - Opened by oyangz 4 months ago - 2 comments

#287 - build(deps): bump tornado from 6.4 to 6.4.1

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 1 comment
Labels: dependencies

#285 - build: bump fmeval version to 1.0.4

Pull Request - State: closed - Opened by danielezhu 4 months ago

#284 - [Feature] Add callback mechanism to evaluation

Issue - State: open - Opened by acere 4 months ago - 1 comment

#283 - Support multiple data configs in evaluate

Pull Request - State: closed - Opened by athewsey 4 months ago - 1 comment

#282 - Fix comparisons for string enumerations

Pull Request - State: closed - Opened by athewsey 4 months ago - 2 comments

#280 - [Feature] Add system metrics collected during evaluation to eval_output

Issue - State: open - Opened by acere 4 months ago - 4 comments

#279 - build(deps): bump requests from 2.31.0 to 2.32.0

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 4 comments
Labels: dependencies

#277 - chore: update BarPlotCell unit test

Pull Request - State: closed - Opened by danielezhu 5 months ago

#276 - chore: add support for proprietary models

Pull Request - State: closed - Opened by shrestha-bikash 5 months ago

#275 - build: bump fmeval version to 1.0.3

Pull Request - State: closed - Opened by danielezhu 5 months ago

#273 - feat: allow placeholder dict for prompt composer

Pull Request - State: closed - Opened by xiaoyi-cheng 5 months ago

#272 - build(deps): bump jinja2 from 3.1.3 to 3.1.4

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago
Labels: dependencies

#271 - build(deps): bump tqdm from 4.66.2 to 4.66.3

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago
Labels: dependencies

#270 - build(deps): bump sagemaker from 2.216.1 to 2.218.0

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 2 comments
Labels: dependencies

#268 - [Feature] JSON export/import for EvalOutput classes

Issue - State: open - Opened by athewsey 5 months ago - 3 comments

#267 - feat: fetch log probability jmespath from JS metadata

Pull Request - State: closed - Opened by keerthanvasist 5 months ago

#266 - feat: add target_context to dataset columns

Pull Request - State: closed - Opened by oyangz 5 months ago - 2 comments

#265 - docs: update telemetry-related info in README and docstrings

Pull Request - State: closed - Opened by danielezhu 5 months ago

#264 - fix: fix patching in unit tests

Pull Request - State: closed - Opened by danielezhu 5 months ago

#263 - build: bump fmeval version to 1.0.2

Pull Request - State: closed - Opened by danielezhu 5 months ago - 1 comment

#261 - docs: add syntax highlighting

Pull Request - State: closed - Opened by connorads 5 months ago - 1 comment

#259 - test: update matplotlib version and figure cell init test

Pull Request - State: closed - Opened by oyangz 5 months ago

#257 - build(deps): bump aiohttp from 3.9.3 to 3.9.4

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 2 comments
Labels: dependencies

#256 - chore: simplify botocore/boto3-related util code

Pull Request - State: closed - Opened by danielezhu 6 months ago

#255 - build: bump fmeval version to 1.0.1

Pull Request - State: closed - Opened by danielezhu 6 months ago

#254 - fix: update output record key validation logic in validate_call

Pull Request - State: closed - Opened by danielezhu 6 months ago

#253 - Revert "build: bump fmeval version to 1.0.1"

Pull Request - State: closed - Opened by oyangz 6 months ago

#251 - build: bump fmeval version to 1.0.1

Pull Request - State: closed - Opened by oyangz 6 months ago

#250 - build(deps): bump idna from 3.6 to 3.7

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 2 comments
Labels: dependencies

#249 - build(deps): bump transformers from 4.37.2 to 4.38.0

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 2 comments
Labels: dependencies

#248 - docs: update README with details on contributing new eval algos

Pull Request - State: closed - Opened by danielezhu 6 months ago

#247 - [Feature] Image fields for multi-modal models

Issue - State: open - Opened by athewsey 6 months ago - 1 comment

#246 - [Feature] Multi-variable prompt templates

Issue - State: open - Opened by athewsey 6 months ago - 2 comments

#245 - fix: fix s3 uri validation for built-in datasets

Pull Request - State: closed - Opened by oyangz 6 months ago - 1 comment

#244 - build(deps): bump pillow from 10.2.0 to 10.3.0

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 2 comments
Labels: dependencies

#243 - build: bump fmeval version to 1.0.0

Pull Request - State: closed - Opened by danielezhu 6 months ago

#227 - Toxicity transforms

Pull Request - State: closed - Opened by franluca 7 months ago - 1 comment

#221 - build(deps-dev): bump black from 23.7.0 to 24.3.0

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 2 comments
Labels: dependencies

#212 - fix: update the default prompt templates for the built-in datasets

Pull Request - State: closed - Opened by jmikko 7 months ago

#199 - Evaluation Algorithm for Recommendations

Issue - State: closed - Opened by mccartni-aws 7 months ago - 4 comments

#186 - String enums not comparing as expected

Issue - State: closed - Opened by athewsey 8 months ago - 3 comments

#167 - feat: conversion of targets to strings

Pull Request - State: closed - Opened by franluca 9 months ago - 2 comments

#135 - feat(scientific correctness tests): test suite for scientific correctness

Pull Request - State: closed - Opened by polaschwoebel 11 months ago - 1 comment

#100 - refactor: rename package name to fmeval

Pull Request - State: closed - Opened by xiaoyi-cheng 11 months ago - 2 comments