allenai/open-instruct issues and pull requests

#286 - Add safety evals via `submit_eval`

Pull Request - State: closed - Opened by hamishivi 3 months ago - 2 comments

#285 - Patch checkpointing step

Pull Request - State: closed - Opened by hamishivi 3 months ago

#284 - Upload to hf models to revision

Pull Request - State: closed - Opened by vwxyzjn 3 months ago

#283 - Add a `train` field to training data

Issue - State: open - Opened by hamishivi 3 months ago - 2 comments
Labels: enhancement

#282 - Add new eval Benchmarks for evaluating long context

Pull Request - State: open - Opened by nouhadziri 3 months ago

#281 - Try uv for package management (245ms project installation)

Pull Request - State: open - Opened by vwxyzjn 3 months ago

#280 - Make faster docker build

Issue - State: open - Opened by vwxyzjn 3 months ago

#279 - RS: save config and raw scores

Pull Request - State: closed - Opened by vwxyzjn 3 months ago

#278 - Propagate failure/preemptation and set timeout for multinode training

Pull Request - State: closed - Opened by yizhongw 3 months ago - 1 comment

#277 - Auto evaluation via `--launch_beaker_eval_jobs`

Pull Request - State: closed - Opened by vwxyzjn 3 months ago - 6 comments

#276 - Make open-instruct pip installable

Issue - State: open - Opened by yakazimir 3 months ago - 1 comment
Labels: enhancement

#275 - Fix path in train data script

Pull Request - State: closed - Opened by hamishivi 3 months ago

#274 - Log beaker experiment URL

Pull Request - State: closed - Opened by vwxyzjn 3 months ago

#272 - missing sharegpt_html_cleaned_and_split_2048.json

Issue - State: closed - Opened by ppalantir 3 months ago - 1 comment

#265 - small optimization to avoid overhead

Pull Request - State: closed - Opened by vwxyzjn 3 months ago

#259 - Use updated Tulu config in oe-eval and optionally run multiturn fewshot evals

Pull Request - State: closed - Opened by pdasigi 3 months ago - 4 comments

#257 - Add v3.3 config

Pull Request - State: closed - Opened by natolambert 3 months ago

#255 - Commands for model evaluation

Issue - State: closed - Opened by mutiann 3 months ago - 1 comment

#238 - Add safety evaluation as a submodule

Pull Request - State: closed - Opened by nouhadziri 3 months ago - 1 comment

#210 - Any strategies for writing function convert_(dataset_name)_data for a custom dataset on hugging face?

Issue - State: closed - Opened by AnnyShen55 4 months ago - 3 comments

#204 - Add online trainers

Pull Request - State: closed - Opened by vwxyzjn 4 months ago - 1 comment

#195 - Revamping the human feedback interface frontend with Next.js

Pull Request - State: open - Opened by darrensapalo 4 months ago

#190 - size mismatch error when running merge_lora.py

Issue - State: closed - Opened by jl3676 4 months ago - 3 comments

#171 - ./scripts/prepare_train_data.sh--throws an error

Issue - State: closed - Opened by dumeixiang 5 months ago - 11 comments

#165 - flash-attn needs data type of bfloat16 or float16

Pull Request - State: closed - Opened by notoookay 6 months ago - 5 comments

#155 - Migrate to tokenizer + fastchat conv templates

Issue - State: open - Opened by natolambert 7 months ago - 2 comments

#100 - Question about the decoding settings for alpaca_eval

Issue - State: closed - Opened by huchinlp 10 months ago - 1 comment

#99 - Better HumanEval instructions

Pull Request - State: closed - Opened by Muennighoff 10 months ago - 4 comments

#98 - Seeking for Help: how to work deepspeed zero stage 3 with quantized reference model?

Issue - State: closed - Opened by grayground 10 months ago - 1 comment

#97 - [Issue with solution] Merging qlora error: open_instruct/merge_lora.py:25 TypeError: 'QuantState' object does not support item assignment

Issue - State: closed - Opened by ZucksLiu 10 months ago - 1 comment

#96 - Release human evaluation interface and tulu 1 human eval data.

Pull Request - State: closed - Opened by yizhongw 11 months ago

#95 - Embedding resizing logic may be flawed

Issue - State: closed - Opened by prajdabre 11 months ago - 1 comment

#94 - File Not Found error while running finetune_lora_with_accelerate.sh

Issue - State: closed - Opened by palash04 11 months ago - 1 comment

#93 - Support for tuning on chat data

Issue - State: closed - Opened by xufangzhi 11 months ago - 1 comment

#92 - Some various fixes and changes for olmo

Pull Request - State: closed - Opened by hamishivi 11 months ago

#91 - Tulu v2 Sanky Diagram

Issue - State: closed - Opened by nuoma 11 months ago - 4 comments

#90 - model weights after finetune can not load by vllm or huggingface

Issue - State: closed - Opened by ZiyiLiubird 11 months ago - 1 comment

#89 - balanced_low_0 Related Issue During Inference

Issue - State: closed - Opened by mshen2 11 months ago - 1 comment

#88 - about tulu v2 conversion

Issue - State: closed - Opened by nuoma 12 months ago - 2 comments

#87 - Possible unreliability of MMLU estimates

Issue - State: closed - Opened by Kong-Aobo 12 months ago - 1 comment

#86 - DPO scripts + docker update + readme update

Pull Request - State: closed - Opened by hamishivi 12 months ago - 1 comment

#85 - replace `:` in eval/mmlu/run_eval.py

Pull Request - State: closed - Opened by Forence1999 12 months ago - 3 comments

#84 - Improving evaluation scripts

Pull Request - State: closed - Opened by yizhongw 12 months ago

#83 - The alpaca eval is not working can't reproduce

Issue - State: closed - Opened by sanyalsunny111 12 months ago - 3 comments

#82 - mmlu run_eval.py: difference in evaluation results between run_eval.py(average_acc: 27.4) and normal results: (llama2-hf: 41.8)

Issue - State: closed - Opened by Stycoo 12 months ago

#81 - How to pull the image

Issue - State: closed - Opened by Forence1999 12 months ago - 1 comment

#80 - Use hf tokenizer for portability when preparing data

Pull Request - State: closed - Opened by hamishivi 12 months ago - 1 comment

#79 - Fix several issues in the evals.

Pull Request - State: closed - Opened by yizhongw 12 months ago

#78 - Update requirements.txt

Pull Request - State: closed - Opened by pminervini about 1 year ago - 1 comment

#77 - Rotary position embedding scaling 8k for Llama2?

Issue - State: closed - Opened by shi-kejian about 1 year ago - 1 comment

#76 - about SFT dataset and dataset ratio

Issue - State: closed - Opened by huliang2016 about 1 year ago - 2 comments

#75 - Cannot reproduce llama2-7b MMLU score

Issue - State: closed - Opened by harryhan618 about 1 year ago - 2 comments

#74 - Not loading cached datasets for preprocessing

Issue - State: closed - Opened by tangzhy about 1 year ago - 1 comment

#73 - Switch to using the built-in flash attention 2 support in Transformers library.

Pull Request - State: closed - Opened by yizhongw about 1 year ago - 1 comment

#72 - could you please report your training cost?

Issue - State: closed - Opened by lucywang720 about 1 year ago - 2 comments

#71 - This script does not work with cerebras gpt and gpt neox.

Issue - State: closed - Opened by kevinscaria about 1 year ago - 1 comment

#70 - minor difference between results in paper and my reproducing results

Issue - State: closed - Opened by lucywang720 about 1 year ago - 3 comments

#69 - Refine prompting and answer parsing for evals

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#68 - Refine training data preparation and add Tulu V2 mixture.

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#67 - Refine the lora merging code.

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#66 - Fine-tuning Reproducibility

Issue - State: closed - Opened by dongmean about 1 year ago - 4 comments

#65 - CUDA OOM for qlora

Issue - State: closed - Opened by tangzhy about 1 year ago - 12 comments

#64 - Support using vllm for generative evaluations

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#63 - Use latest alpaca_eval library to evaluate performance on alpaca farm.

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#62 - Fix MMLU answer token and probability calculation

Pull Request - State: closed - Opened by OyvindTafjord about 1 year ago - 3 comments

#61 - Support multi-node training on beaker.

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#60 - Qlora Documtation + fixes

Pull Request - State: closed - Opened by hamishivi about 1 year ago

#59 - QLora support

Pull Request - State: closed - Opened by hamishivi about 1 year ago

#58 - Revert "merge in lora"

Pull Request - State: closed - Opened by jacob-morrison about 1 year ago

#57 - merge in lora

Pull Request - State: closed - Opened by jacob-morrison about 1 year ago

#56 - Update merge_lora.py

Pull Request - State: closed - Opened by TJKlein about 1 year ago - 3 comments

#55 - [Refactor] Update evaluation scripts to support customized prompting format for specific models (e.g., LLaMa2-chat)

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#54 - It seems that the tokens <|user|> and <|assistant|> are not treated as special tokens.

Issue - State: closed - Opened by Zcchill about 1 year ago - 2 comments

#53 - Estimated timeline for human data release

Issue - State: closed - Opened by martiansideofthemoon about 1 year ago - 3 comments

#52 - [Bug Fix] Fix step misalignment when resuming from a checkpoint

Pull Request - State: closed - Opened by tongyx361 about 1 year ago - 3 comments

#51 - Does LoRa finetuning give weight diff or actual weight?

Issue - State: closed - Opened by macabdul9 about 1 year ago - 2 comments

#50 - Any plan to support mt-bench evaluation?

Issue - State: closed - Opened by HuihuiChyan about 1 year ago - 1 comment

#49 - Add demo scripts

Pull Request - State: closed - Opened by hamishivi about 1 year ago

#48 - Add TruthfulQA Eval

Pull Request - State: closed - Opened by yizhongw about 1 year ago

#47 - Minor toxigen fix, dummy length evaluator

Pull Request - State: closed - Opened by hamishivi about 1 year ago

#46 - When doing LORA fine tuning with grouped optimizers we get an error:

Pull Request - State: closed - Opened by prajdabre about 1 year ago - 1 comment

#45 - Hyperparameter details of different sizes

Issue - State: closed - Opened by hills-code about 1 year ago - 1 comment

#44 - Toxigen Eval

Pull Request - State: closed - Opened by hamishivi about 1 year ago

#43 - Mismatch MMLU results with arxiv paper

Issue - State: closed - Opened by constanzafierro about 1 year ago - 4 comments