swe-bench/SWE-bench issues and pull requests

#77 - Error when running evaluation - ModuleNotFoundError: No module named 'conda'

Issue - State: closed - Opened by brombaut 10 months ago - 11 comments
Labels: evaluation

#77 - Error when running evaluation - ModuleNotFoundError: No module named 'conda'

Issue - State: closed - Opened by brombaut 10 months ago - 11 comments
Labels: evaluation

#76 - Error when running evaluation script - ModuleNotFoundError: No module named 'swebench'

Issue - State: closed - Opened by brombaut 10 months ago - 4 comments
Labels: evaluation

#76 - Error when running evaluation script - ModuleNotFoundError: No module named 'swebench'

Issue - State: closed - Opened by brombaut 10 months ago - 4 comments
Labels: evaluation

#75 - Sharing my gpt-4-bm25-27k results

Issue - State: closed - Opened by roywei 10 months ago - 1 comment

#75 - Sharing my gpt-4-bm25-27k results

Issue - State: closed - Opened by roywei 10 months ago - 1 comment

#74 - Unable to replicate basic results

Issue - State: closed - Opened by bytesuji 10 months ago - 3 comments
Labels: evaluation

#74 - Unable to replicate basic results

Issue - State: closed - Opened by bytesuji 10 months ago - 3 comments
Labels: evaluation

#73 - How to run evaluation with swe-bench_lite?

Issue - State: closed - Opened by pi-null-mezon 10 months ago - 5 comments
Labels: evaluation

#73 - How to run evaluation with swe-bench_lite?

Issue - State: closed - Opened by pi-null-mezon 10 months ago - 5 comments
Labels: evaluation

#72 - Upper bound score by skilled human?

Issue - State: closed - Opened by paul-gauthier 10 months ago - 13 comments
Labels: future, in progress

#72 - Upper bound score by skilled human?

Issue - State: closed - Opened by paul-gauthier 10 months ago - 13 comments
Labels: future, in progress

#71 - Failed to install low versions of Python with Conda

Issue - State: closed - Opened by abmfy 10 months ago - 2 comments
Labels: evaluation

#71 - Failed to install low versions of Python with Conda

Issue - State: closed - Opened by abmfy 10 months ago - 2 comments
Labels: evaluation

#70 - Carlos

Pull Request - State: closed - Opened by carlosejimenez 10 months ago

#70 - Carlos

Pull Request - State: closed - Opened by carlosejimenez 10 months ago

#69 - Update README.md

Pull Request - State: closed - Opened by ssh-randy 10 months ago - 1 comment

#69 - Update README.md

Pull Request - State: closed - Opened by ssh-randy 10 months ago - 1 comment

#68 - PASS_TO_PASS tests failing on original program

Issue - State: closed - Opened by yuntongzhang 10 months ago - 2 comments
Labels: evaluation

#68 - PASS_TO_PASS tests failing on original program

Issue - State: closed - Opened by yuntongzhang 10 months ago - 2 comments
Labels: evaluation

#67 - Handle the case when a conda environment is selected

Pull Request - State: closed - Opened by bstee615 10 months ago - 1 comment

#67 - Handle the case when a conda environment is selected

Pull Request - State: closed - Opened by bstee615 10 months ago - 1 comment

#66 - UnboundLocalError: local variable 'env_name' referenced before assignment in harness/utils.py line 56

Issue - State: closed - Opened by bstee615 10 months ago - 1 comment

#66 - UnboundLocalError: local variable 'env_name' referenced before assignment in harness/utils.py line 56

Issue - State: closed - Opened by bstee615 10 months ago - 1 comment

#65 - Add PyPI Library Build Logic to Main Repo

Pull Request - State: closed - Opened by john-b-yang 10 months ago

#65 - Add PyPI Library Build Logic to Main Repo

Pull Request - State: closed - Opened by john-b-yang 10 months ago

#64 - Update run_live.py

Pull Request - State: closed - Opened by moresearch 10 months ago

#64 - Update run_live.py

Pull Request - State: closed - Opened by moresearch 10 months ago

#63 - Update README.md

Pull Request - State: closed - Opened by moresearch 10 months ago - 1 comment

#63 - Update README.md

Pull Request - State: closed - Opened by moresearch 10 months ago - 1 comment

#62 - Print full stacktrace on error in get_tasks_pipeline.py; add docstring

Pull Request - State: closed - Opened by klieret 10 months ago

#62 - Print full stacktrace on error in get_tasks_pipeline.py; add docstring

Pull Request - State: closed - Opened by klieret 10 months ago

#61 - Abort make_repo.sh script on error; add docstring

Pull Request - State: closed - Opened by klieret 10 months ago - 1 comment

#61 - Abort make_repo.sh script on error; add docstring

Pull Request - State: closed - Opened by klieret 10 months ago - 1 comment

#60 - Update environment.yml

Pull Request - State: closed - Opened by moresearch 11 months ago - 1 comment

#60 - Update environment.yml

Pull Request - State: closed - Opened by moresearch 11 months ago - 1 comment

#59 - How can I run_evalution? What is the args swe_bench_tasks,I need to create by myself?

Issue - State: closed - Opened by Ottomachine1 11 months ago - 4 comments
Labels: evaluation

#59 - How can I run_evalution? What is the args swe_bench_tasks,I need to create by myself?

Issue - State: closed - Opened by Ottomachine1 11 months ago - 4 comments
Labels: evaluation

#58 - Difficulty Reproducing BM25 Results

Issue - State: open - Opened by VecherVhatuX 11 months ago - 4 comments
Labels: inference, in progress

#58 - Difficulty Reproducing BM25 Results

Issue - State: open - Opened by VecherVhatuX 11 months ago - 4 comments
Labels: inference, in progress

#57 - Unable to test for Scikit-Learn

Issue - State: closed - Opened by xingyaoww 11 months ago - 7 comments
Labels: evaluation

#57 - Unable to test for Scikit-Learn

Issue - State: closed - Opened by xingyaoww 11 months ago - 7 comments
Labels: evaluation

#56 - Add Dockerfile and fix the pytest for matplotlib

Pull Request - State: closed - Opened by xingyaoww 11 months ago - 1 comment

#56 - Add Dockerfile and fix the pytest for matplotlib

Pull Request - State: closed - Opened by xingyaoww 11 months ago - 1 comment

#55 - Unable to run evaluation. Testbed is having trouble creating a specified conda environment

Issue - State: closed - Opened by KevinCL16 11 months ago - 16 comments
Labels: evaluation

#55 - Unable to run evaluation. Testbed is having trouble creating a specified conda environment

Issue - State: closed - Opened by KevinCL16 11 months ago - 16 comments
Labels: evaluation

#54 - Fix get top pypi repos

Pull Request - State: closed - Opened by klieret 11 months ago

#54 - Fix get top pypi repos

Pull Request - State: closed - Opened by klieret 11 months ago

#53 - Make scripts executable and add `description` field to `ArgumentParser`

Pull Request - State: closed - Opened by klieret 11 months ago - 1 comment

#53 - Make scripts executable and add `description` field to `ArgumentParser`

Pull Request - State: closed - Opened by klieret 11 months ago - 1 comment

#52 - Pyupgrade autofixes

Pull Request - State: closed - Opened by klieret 11 months ago - 1 comment

#52 - Pyupgrade autofixes

Pull Request - State: closed - Opened by klieret 11 months ago - 1 comment

#51 - Clarification Needed on Evaluation Code for Retrieval with BM25 and Context Length-Specific Recall Metrics

Issue - State: closed - Opened by Adefful 11 months ago - 3 comments

#51 - Clarification Needed on Evaluation Code for Retrieval with BM25 and Context Length-Specific Recall Metrics

Issue - State: closed - Opened by Adefful 11 months ago - 3 comments

#50 - bug fixes for swe-bench

Pull Request - State: closed - Opened by wwzeng1 11 months ago - 1 comment

#50 - bug fixes for swe-bench

Pull Request - State: closed - Opened by wwzeng1 11 months ago - 1 comment

#49 - Allow conda directory to be specified in run_evaluation.py

Pull Request - State: closed - Opened by Jonty800 11 months ago - 1 comment

#49 - Allow conda directory to be specified in run_evaluation.py

Pull Request - State: closed - Opened by Jonty800 11 months ago - 1 comment

#48 - add JP URL & JP Docs

Pull Request - State: closed - Opened by Sunwood-ai-labs 11 months ago - 1 comment

#48 - add JP URL & JP Docs

Pull Request - State: closed - Opened by Sunwood-ai-labs 11 months ago - 1 comment

#47 - Request for Baseline Model Testing Log Files for Research Purposes

Issue - State: closed - Opened by yczhou001 11 months ago - 1 comment

#47 - Request for Baseline Model Testing Log Files for Research Purposes

Issue - State: closed - Opened by yczhou001 11 months ago - 1 comment

#46 - What are expected to submit for the leaderboard integration?

Issue - State: closed - Opened by zhimin-z 11 months ago - 4 comments
Labels: documentation

#46 - What are expected to submit for the leaderboard integration?

Issue - State: closed - Opened by zhimin-z 11 months ago - 4 comments
Labels: documentation

#45 - Reproducible Experiments via MLFlow or Similar [FeatureRequest]

Issue - State: closed - Opened by moresearch 11 months ago - 1 comment

#45 - Reproducible Experiments via MLFlow or Similar [FeatureRequest]

Issue - State: closed - Opened by moresearch 11 months ago - 1 comment

#44 - Running SWE-Llama(13B/7B) on HuggingFace Endpoints [FeatureRequest]

Issue - State: closed - Opened by moresearch 11 months ago - 1 comment

#44 - Running SWE-Llama(13B/7B) on HuggingFace Endpoints [FeatureRequest]

Issue - State: closed - Opened by moresearch 11 months ago - 1 comment

#43 - Issues in the test case parsing logic from the logs

Issue - State: closed - Opened by anmolagarwal999 about 1 year ago - 5 comments
Labels: evaluation

#43 - Issues in the test case parsing logic from the logs

Issue - State: closed - Opened by anmolagarwal999 about 1 year ago - 5 comments
Labels: evaluation

#42 - Updated Docker Container

Issue - State: closed - Opened by sorendunn about 1 year ago - 3 comments

#42 - Updated Docker Container

Issue - State: closed - Opened by sorendunn about 1 year ago - 3 comments

#41 - fix(get_conda_env_names): handle lines with only path in `conda env l…

Pull Request - State: closed - Opened by itaowei about 1 year ago - 1 comment

#41 - fix(get_conda_env_names): handle lines with only path in `conda env l…

Pull Request - State: closed - Opened by itaowei about 1 year ago - 1 comment

#40 - Unreliability when generating patches with `diff` format?

Issue - State: closed - Opened by ErikBjare about 1 year ago - 2 comments

#40 - Unreliability when generating patches with `diff` format?

Issue - State: closed - Opened by ErikBjare about 1 year ago - 2 comments

#39 - Argument `--swe_bench_tasks` is unclear

Issue - State: closed - Opened by LinxinS97 about 1 year ago - 4 comments

#39 - Argument `--swe_bench_tasks` is unclear

Issue - State: closed - Opened by LinxinS97 about 1 year ago - 4 comments

#38 - Unable to create a mirror of an repository

Issue - State: closed - Opened by ZiyueWang25 about 1 year ago - 2 comments

#38 - Unable to create a mirror of an repository

Issue - State: closed - Opened by ZiyueWang25 about 1 year ago - 2 comments

#37 - some wrong in build_dataset.py

Issue - State: closed - Opened by lyc3202 about 1 year ago - 2 comments

#37 - some wrong in build_dataset.py

Issue - State: closed - Opened by lyc3202 about 1 year ago - 2 comments

#36 - Inference on CPU

Issue - State: closed - Opened by moresearch about 1 year ago - 1 comment
Labels: wontfix

#36 - Inference on CPU

Issue - State: closed - Opened by moresearch about 1 year ago - 1 comment
Labels: wontfix

#35 - Update README.md

Pull Request - State: closed - Opened by moresearch about 1 year ago - 1 comment

#35 - Update README.md

Pull Request - State: closed - Opened by moresearch about 1 year ago - 1 comment

#34 - logs are unusable with multiple test instances

Issue - State: closed - Opened by JasonGross about 1 year ago - 5 comments

#34 - logs are unusable with multiple test instances

Issue - State: closed - Opened by JasonGross about 1 year ago - 5 comments

#33 - errors related to subprocesses in run_evaluation.py

Issue - State: closed - Opened by sorendunn about 1 year ago - 4 comments

#33 - errors related to subprocesses in run_evaluation.py

Issue - State: closed - Opened by sorendunn about 1 year ago - 4 comments

#32 - Also clean and checkout submodules

Pull Request - State: closed - Opened by JasonGross about 1 year ago - 1 comment
Labels: future