confident-ai/deepeval issues and pull requests

#1348 - finish dag

Pull Request - State: closed - Opened by penguine-ip about 18 hours ago - 1 comment

#1347 - fix: "Use a Custom LLM" guide

Pull Request - State: open - Opened by kaikaidai 1 day ago - 1 comment

#1346 - recommend metrics telemetry

Pull Request - State: closed - Opened by kritinv 2 days ago - 1 comment

#1345 - fix-logged-in-with

Pull Request - State: closed - Opened by kritinv 2 days ago - 1 comment

#1344 - fix: typos in metrics templates

Pull Request - State: open - Opened by FilippoPaganelli 3 days ago - 1 comment

#1343 - Change to a version of gRPC >= 1.70.0 as the current verison (1.67.1) has a known vulnerability.

Issue - State: open - Opened by harshapraneeth 4 days ago

#1342 - deepeval recommend metrics

Pull Request - State: closed - Opened by kritinv 4 days ago - 1 comment

#1341 - logged in with

Pull Request - State: closed - Opened by kritinv 4 days ago - 1 comment

#1340 - New Relic License key and endpoint are exposed in source

Issue - State: open - Opened by paul-abb 4 days ago - 1 comment

#1339 - Add Gemini models native support as LLM judge

Pull Request - State: open - Opened by rarsan 4 days ago - 3 comments

#1338 - auto_evaluate bug fixes

Pull Request - State: closed - Opened by kritinv 4 days ago - 1 comment

#1337 - Creating Custom EvaluationDataset test_case but can't use 'Evaluate' methods

Issue - State: closed - Opened by HandsRemember 4 days ago - 1 comment

#1336 - Fix broken link to GEval paper in metrics-conversational-g-eval.mdx

Pull Request - State: closed - Opened by j-mesnil 5 days ago - 1 comment

#1335 - AttributeError: 'NoneType' object has no attribute 'score'

Issue - State: closed - Opened by Sara-Hossny 5 days ago - 2 comments

#1334 - new release

Pull Request - State: closed - Opened by penguine-ip 5 days ago - 1 comment

#1333 - query about trying to add test cases from csv file

Issue - State: open - Opened by Sara-Hossny 6 days ago - 2 comments

#1332 - CybersecurityGuard.name property returns None causing Pydantic validation error

Issue - State: closed - Opened by KISLAYA-SRI 6 days ago

#1331 - add o3-mini models

Pull Request - State: closed - Opened by luarss 7 days ago - 2 comments

#1330 - embedder bug fix

Pull Request - State: closed - Opened by kritinv 7 days ago - 1 comment

#1329 - chore: Fix typo in telemetry

Pull Request - State: closed - Opened by PLNech 8 days ago - 3 comments

#1328 - new release

Pull Request - State: closed - Opened by penguine-ip 8 days ago - 1 comment

#1327 - revise the logic of deciding whether should_use_batch

Pull Request - State: closed - Opened by ruiqizhu-ricky 8 days ago - 2 comments

#1326 - qa agent tutorial

Pull Request - State: closed - Opened by kritinv 8 days ago - 1 comment

#1325 - Use Case Tutorials QA agent

Pull Request - State: closed - Opened by kritinv 9 days ago - 1 comment

#1324 - Need more clarity around retrieval_context vs context

Issue - State: closed - Opened by ParikhKadam 10 days ago - 2 comments

#1323 - Use Case Tutorials Legal Doc Summarizer

Pull Request - State: closed - Opened by kritinv 10 days ago - 1 comment

#1322 - ollama fix

Pull Request - State: closed - Opened by kritinv 10 days ago - 1 comment

#1321 - Generate from scratch fixes

Pull Request - State: closed - Opened by kritinv 11 days ago - 1 comment

#1320 - delete .vector_db folder if chunking is unsuccesful

Pull Request - State: closed - Opened by kritinv 11 days ago - 1 comment

#1319 - Update synthesizer-introduction.mdx

Pull Request - State: closed - Opened by sobs0 11 days ago - 4 comments

#1318 - async fixes for tracing

Pull Request - State: closed - Opened by kritinv 12 days ago - 1 comment

#1317 - tool correctness

Pull Request - State: closed - Opened by kritinv 13 days ago - 1 comment

#1316 - Context precision docfix

Pull Request - State: closed - Opened by luarss 14 days ago - 2 comments

#1315 - Fix circular import in Scorer

Pull Request - State: closed - Opened by luarss 14 days ago - 2 comments

#1314 - test

Pull Request - State: closed - Opened by kritinv 14 days ago - 1 comment

#1313 - Update benchmarks-MMLU.mdx with MMLUTask

Pull Request - State: closed - Opened by AMindToThink 14 days ago - 2 comments

#1312 - Azure function returns tuple

Pull Request - State: open - Opened by JaroslawMargasDev 15 days ago - 3 comments

#1311 - Jailbreaking Crescendo Schematic Change

Pull Request - State: closed - Opened by kritinv 15 days ago - 1 comment

#1310 - updated task-completion doc

Pull Request - State: closed - Opened by kritinv 15 days ago - 1 comment

#1309 - make red teamer surface errors

Pull Request - State: closed - Opened by penguine-ip 15 days ago - 1 comment

#1308 - AttributeError: 'str' object has no attribute 'statements'

Issue - State: closed - Opened by romai 15 days ago - 2 comments

#1307 - Typo fix. "Confidnet" -> "Confident"

Pull Request - State: closed - Opened by r-sniper 16 days ago - 2 comments

#1306 - trying to use deepseek as custom llm

Issue - State: closed - Opened by Sara-Hossny 16 days ago - 8 comments

#1304 - local model/azure fix

Pull Request - State: closed - Opened by kritinv 16 days ago - 1 comment

#1303 - update docs g-eval

Pull Request - State: closed - Opened by kritinv 16 days ago - 1 comment

#1302 - new release

Pull Request - State: closed - Opened by penguine-ip 16 days ago - 1 comment

#1301 - add conditional displaying

Pull Request - State: closed - Opened by penguine-ip 16 days ago - 1 comment

#1300 - Exclude tests folder during installation

Pull Request - State: closed - Opened by fj11 16 days ago - 3 comments

#1299 - add validated openai models

Pull Request - State: closed - Opened by luarss 16 days ago - 2 comments

#1298 - GPT model testing

Issue - State: closed - Opened by luarss 17 days ago - 1 comment

#1297 - TypeError: Synthesizer.generate_goldens_from_docs() got an unexpected keyword argument 'chunk_size'

Issue - State: closed - Opened by Sara-Hossny 17 days ago - 2 comments

#1296 - feature status telemetry

Pull Request - State: closed - Opened by kritinv 17 days ago - 1 comment

#1295 - task completion metric

Pull Request - State: closed - Opened by kritinv 17 days ago - 1 comment

#1294 - Fix typo in Update guides-using-custom-llms.mdx

Pull Request - State: closed - Opened by ChristianBernhard 18 days ago - 2 comments

#1293 - tool correctness + input parameters and output

Pull Request - State: closed - Opened by kritinv 18 days ago - 1 comment

#1292 - .

Pull Request - State: closed - Opened by penguine-ip 19 days ago - 1 comment

#1291 - Release v2.1.9

Pull Request - State: closed - Opened by penguine-ip 19 days ago - 1 comment

#1290 - ToolCall support

Pull Request - State: closed - Opened by kritinv 19 days ago - 1 comment

#1289 - specify synthesizer context length

Pull Request - State: closed - Opened by kritinv 19 days ago - 1 comment

#1288 - update link

Pull Request - State: closed - Opened by kritinv 19 days ago - 1 comment

#1287 - OpenAI model updates

Pull Request - State: closed - Opened by luarss 19 days ago - 2 comments

#1286 - OpenAI Batch API

Issue - State: open - Opened by luarss 19 days ago - 1 comment

#1285 - structured outputs using OpenAI API

Pull Request - State: closed - Opened by kritinv 20 days ago - 1 comment

#1284 - Benchmarks introduction does setup wrong

Issue - State: open - Opened by AMindToThink 21 days ago - 3 comments

#1283 - auto-eval

Pull Request - State: closed - Opened by kritinv 21 days ago - 1 comment

#1282 - autologin

Pull Request - State: open - Opened by kritinv 21 days ago - 1 comment

#1281 - Add base attack in the returned results in redteam scan

Issue - State: open - Opened by AmineDjeghri 22 days ago - 3 comments

#1280 - updated telemetry

Pull Request - State: closed - Opened by kritinv 22 days ago - 1 comment

#1279 - new release

Pull Request - State: closed - Opened by penguine-ip 22 days ago - 1 comment

#1278 - Add update warning opt out

Pull Request - State: closed - Opened by luarss 23 days ago - 1 comment

#1277 - Expand openai models support

Pull Request - State: closed - Opened by luarss 23 days ago - 2 comments

#1276 - telemetry

Pull Request - State: closed - Opened by kritinv 23 days ago - 1 comment

#1275 - Remove Python <3.13 version restriction from setup.py

Issue - State: closed - Opened by 21m1n 23 days ago - 3 comments

#1274 - fix batched evaluate() behavior

Pull Request - State: closed - Opened by BjarniHaukur 23 days ago - 2 comments

#1273 - docs: update README.md

Pull Request - State: closed - Opened by eltociear 24 days ago - 2 comments

#1272 - OpenAI model support

Issue - State: closed - Opened by luarss 24 days ago - 1 comment

#1271 - fix

Pull Request - State: closed - Opened by penguine-ip 24 days ago - 1 comment

#1270 - new release

Pull Request - State: closed - Opened by penguine-ip 24 days ago - 1 comment

#1269 - Synthesizer cost

Issue - State: closed - Opened by luarss 24 days ago - 1 comment

#1268 - docs posthog

Pull Request - State: closed - Opened by kritinv 24 days ago - 1 comment

#1267 - fix task reloading of MMLU

Pull Request - State: closed - Opened by exhyy 25 days ago - 1 comment

#1266 - Synthesizer cost tracker breaks golden generation pipeline for custom llms

Issue - State: open - Opened by pazevedo-hyland 26 days ago - 4 comments

#1265 - testcase score fluctuate significantly

Issue - State: closed - Opened by enx-github 26 days ago - 5 comments

#1264 - ImportError: cannot import name 'Scorer' from partially initialized module 'deepeval.scorer' (most likely due to a circular import) (/Users/macbookpro/Documents/Codes/ML/Athena_2_LLM_Evaluation/venv/lib/python3.11/site-packages/deepeval/scorer/init.py)

Issue - State: closed - Opened by shuvo881 26 days ago - 1 comment

#1263 - Set default value of evaluation_cost to 0 in BaseMetric class

Pull Request - State: closed - Opened by rawheel 29 days ago - 3 comments

#1263 - Set default value of evaluation_cost to 0 in BaseMetric class

Pull Request - State: open - Opened by rawheel 29 days ago - 1 comment

#1262 - metric.evaluate() raise TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'

Issue - State: open - Opened by rawheel 29 days ago

GitHub / confident-ai/deepeval issues and pull requests