tunib-ai/oslo issues and pull requests

#150 - fix local_world_size for slurm

Pull Request - State: closed - Opened by singleheart over 2 years ago - 1 comment

#149 - Tp 2p5d / 2d / 1d deparallelize

Pull Request - State: closed - Opened by jason9693 over 2 years ago

#147 - Tensor sequence parallel

Pull Request - State: closed - Opened by bzantium over 2 years ago

#146 - Merge from master

Pull Request - State: closed - Opened by jinok2im over 2 years ago

#145 - Trainer

Pull Request - State: closed - Opened by tree-park over 2 years ago

#144 - Add OPT Model mapping

Pull Request - State: closed - Opened by dongs0104 over 2 years ago - 1 comment

#143 - Update data collators and Add models

Pull Request - State: closed - Opened by gimmaru over 2 years ago - 3 comments

#142 - Add splitting bias for head

Pull Request - State: closed - Opened by bzantium over 2 years ago - 2 comments

#141 - Fix tp 2p5d deparallelize

Pull Request - State: closed - Opened by bzantium over 2 years ago

#140 - Add & Test Novograd, Lamb, MixedPrecisionLamb

Pull Request - State: closed - Opened by cozytk over 2 years ago

#139 - Init PP comm with RPC

Pull Request - State: closed - Opened by hyunwoongko over 2 years ago - 1 comment

#138 - Update data binarization

Pull Request - State: closed - Opened by gimmaru over 2 years ago

#137 - Add modeling transformers

Pull Request - State: closed - Opened by gimmaru over 2 years ago - 1 comment

#136 - fix: :bug: Fix bug to fail fused layer norm cuda file build

Pull Request - State: closed - Opened by loopinf over 2 years ago

#135 - Add fused optimizers

Pull Request - State: closed - Opened by cozytk over 2 years ago

#134 - Fixes softmax

Pull Request - State: closed - Opened by cozytk over 2 years ago

#133 - implement tp-3d layers, wrapper, test codes and refactor all tp test codes and layers

Pull Request - State: closed - Opened by bzantium over 2 years ago - 2 comments

#132 - Add mappings for head (LM, Classification, QA...) and _slice_head in wrappers.

Pull Request - State: closed - Opened by bzantium over 2 years ago - 1 comment

#131 - Remove Trainer import bug

Pull Request - State: closed - Opened by tree-park over 2 years ago - 1 comment

#130 - is_lm_head -> is_head

Pull Request - State: closed - Opened by bzantium over 2 years ago

#129 - Add LMHead mappings

Pull Request - State: closed - Opened by bzantium over 2 years ago - 4 comments

#128 - remove: deprecated fused functions

Pull Request - State: closed - Opened by cozytk over 2 years ago - 1 comment

#127 - Data binarization for Pre-training

Pull Request - State: closed - Opened by gimmaru over 2 years ago - 1 comment

#126 - Tp 2p5d wrapper

Pull Request - State: closed - Opened by jason9693 over 2 years ago - 2 comments

#125 - First Pull Request for Trainer

Pull Request - State: closed - Opened by tree-park over 2 years ago - 1 comment

#124 - fix Linear2D gather_output operation

Pull Request - State: closed - Opened by bzantium over 2 years ago

#123 - Fix tp2d

Pull Request - State: closed - Opened by bzantium over 2 years ago - 2 comments

#122 - [hotfix] multinode rank error

Pull Request - State: closed - Opened by jason9693 over 2 years ago

#121 - Merge pp communication work

Pull Request - State: closed - Opened by jinok2im over 2 years ago - 1 comment

#120 - FusedAdam & CPUAdam

Pull Request - State: closed - Opened by cozytk over 2 years ago - 6 comments

#119 - Implement Expert Parallel and Test for Initialization and Forward Pass

Pull Request - State: closed - Opened by scsc0511 over 2 years ago

#118 - Implement Expert Parallel and Test for Initialization and Forward Pass

Pull Request - State: closed - Opened by scsc0511 over 2 years ago - 3 comments

#117 - Kernel no repeat

Pull Request - State: closed - Opened by loopinf over 2 years ago

#116 - Implement Expert Parallel and Test for Forward Pass

Pull Request - State: closed - Opened by scsc0511 over 2 years ago

#115 - remove lazy modules and move to experimental

Pull Request - State: closed - Opened by reniew over 2 years ago - 2 comments

#114 - fix: Circular Import ZeroRedundancyOptimizer

Pull Request - State: closed - Opened by cozytk over 2 years ago

#113 - remove lazy at wrapper-1,2d

Pull Request - State: closed - Opened by bzantium over 2 years ago

#112 - Merge pp_partition into pp_communication_jinok

Pull Request - State: closed - Opened by jinok2im over 2 years ago

#111 - [WIP] Add tp-2d-wrapper

Pull Request - State: closed - Opened by bzantium over 2 years ago - 8 comments

#110 - [WIP] init: fusedAdam

Pull Request - State: closed - Opened by cozytk over 2 years ago - 1 comment

#109 - [WIP] Implement PP communication

Pull Request - State: closed - Opened by hyunwoongko over 2 years ago

#108 - Fix data collator

Pull Request - State: closed - Opened by hyunwoongko over 2 years ago

#107 - Fix Softmax API

Pull Request - State: closed - Opened by hyunwoongko over 2 years ago

#106 - Fused layer norm

Pull Request - State: closed - Opened by loopinf over 2 years ago - 1 comment

#105 - Remove scheduler from PP class

Pull Request - State: closed - Opened by Quentin-Anthony almost 3 years ago

#104 - Fix import error

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#103 - remove schedulers

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#102 - Fix PP partition bug

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#101 - Refactoring MultiheadAttention with todo anchors

Pull Request - State: closed - Opened by l-yohai almost 3 years ago - 2 comments

#100 - minor update in DDP

Pull Request - State: closed - Opened by jinok2im almost 3 years ago

#99 - remove useless file

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#98 - Add P2P communicator

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#97 - modify parallel context

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago - 2 comments

#96 - Implementation of Sequential Parallelism

Pull Request - State: closed - Opened by ohwi almost 3 years ago - 5 comments

#95 - fix rank transpose error for tp-2d

Pull Request - State: closed - Opened by bzantium almost 3 years ago

#94 - Tp 2p5d

Pull Request - State: closed - Opened by jason9693 almost 3 years ago

#92 - [WIP] Add tp-2d wrapper

Pull Request - State: closed - Opened by bzantium almost 3 years ago

#91 - Scale mask softmax

Pull Request - State: closed - Opened by cozytk almost 3 years ago

#90 - Add backward testing for tp-1d, 2d

Pull Request - State: closed - Opened by bzantium almost 3 years ago - 1 comment

#89 - Make parallel_context optional

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#88 - Add Expert Parallel Context

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#87 - Add tp-1d layers testing

Pull Request - State: closed - Opened by bzantium almost 3 years ago - 2 comments

#86 - Add contributors

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#85 - Fix Pre-commit CI

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#84 - Revert "[WIP] Add data processing modules referring to the lassl"

Pull Request - State: closed - Opened by gimmaru almost 3 years ago

#83 - Add test code for ZeRO stage 3

Pull Request - State: closed - Opened by jinok2im almost 3 years ago - 6 comments

#82 - Integrate Sequence Parallelism branches

Pull Request - State: closed - Opened by l-yohai almost 3 years ago - 3 comments

#81 - add ZeRO with TP test

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#80 - Pipeline parallelism point-to-point communication wrapper setup

Pull Request - State: closed - Opened by MicPie almost 3 years ago

#79 - [WIP] ZeRO optimizer

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#78 - Fix TP 1D

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#77 - Kernel fusion rms norm layer

Pull Request - State: closed - Opened by loopinf almost 3 years ago

#76 - [DDP] Process_group -> Parallel_Context

Pull Request - State: closed - Opened by hmy831004 almost 3 years ago - 1 comment

#75 - Branch ddp

Pull Request - State: closed - Opened by hmy831004 almost 3 years ago

#74 - DDP Branch pull request

Pull Request - State: closed - Opened by hmy831004 almost 3 years ago - 1 comment

#73 - [Draft] Implement ZeRO Stage 3

Pull Request - State: closed - Opened by jinok2im almost 3 years ago - 5 comments

#71 - [WIP] Implement TP 2D

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago - 12 comments

#70 - [WIP] Add colossal ops

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#69 - Tp 1d

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago - 1 comment

#68 - [WIP] Implement MultiHeadAttention for Sequence Parallel

Pull Request - State: closed - Opened by l-yohai almost 3 years ago

#67 - [WIP] ADD Tensor Parallel 1D

Pull Request - State: closed - Opened by bzantium almost 3 years ago

#65 - make zero branch for collaboration

Pull Request - State: closed - Opened by jinok2im almost 3 years ago

#64 - [WIP] Create base trainer.py train.py

Pull Request - State: closed - Opened by donggrii almost 3 years ago - 1 comment

#62 - [WIP] Implement ZeRO Stage 2

Pull Request - State: closed - Opened by jayten-jeon almost 3 years ago

#61 - rename files

Pull Request - State: closed - Opened by loopinf almost 3 years ago - 15 comments

#60 - [WIP] add test code of sp training

Pull Request - State: closed - Opened by l-yohai almost 3 years ago - 2 comments

#59 - Add OSS

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#58 - [WIP] Add data processing modules referring to the lassl

Pull Request - State: closed - Opened by gimmaru almost 3 years ago - 6 comments

#57 - [WIP] Implement ZeRO Stage 3 (FSDP)

Pull Request - State: closed - Opened by jinok2im almost 3 years ago - 9 comments

#56 - Fix aib fusion

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#55 - [WIP] Add FusedScaledMaskSoftmax

Pull Request - State: closed - Opened by cozytk almost 3 years ago - 11 comments

#54 - [WIP] remove forward and load_state_dict pre hook when initialization

Pull Request - State: closed - Opened by bzantium almost 3 years ago - 1 comment

#53 - fix docstring

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#52 - change tp mappings

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#51 - fix import

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#50 - implementing gpt2

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#49 - rename conv_1d to conv

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#48 - Add Conv1D, LazyConv1D layers

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#47 - Add LazyLinear class

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

#45 - modify REAMDE

Pull Request - State: closed - Opened by hyunwoongko almost 3 years ago

GitHub / tunib-ai/oslo issues and pull requests