HomebrewNLP/Olmax issues and pull requests

#101 - Better shampoo splitting

Pull Request - State: closed - Opened by ClashLuke about 2 years ago

#100 - Don't decay mixer

Pull Request - State: closed - Opened by ClashLuke about 2 years ago

#99 - Adam square

Pull Request - State: closed - Opened by ClashLuke about 2 years ago - 1 comment

#98 - No flatten depth

Pull Request - State: closed - Opened by ClashLuke about 2 years ago - 1 comment

#97 - Improve Shampoo

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#96 - Flatten Conv for Shampoo, FP64 inverse

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#95 - fix(shampoo): don't debias stat

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#94 - fix(optimizer): debias in correct direction

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#93 - Mean teacher

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#92 - More normv2

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#91 - Looped pool

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 2 comments

#90 - Tests for ctx.parameters

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#89 - feat(model): single branch revnet

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#88 - Weight-Tie MoE

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#87 - Dense2

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#86 - Moe tree

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 2 comments

#85 - Moe2

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 4 comments

#84 - log stats + fix first checkpoint/resume

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 2 comments

#83 - Cleanup backend

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#82 - Looks linear

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#81 - Multiple forward per backward

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, engineering, core

#80 - Staged batchsize training

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, engineering, core

#79 - Compact Loss

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, core

#78 - Causality Test

Issue - State: open - Opened by ClashLuke over 2 years ago - 1 comment
Labels: engineering, core

#77 - LpNorm + ScaleNorm

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#76 - Fix checkpoint

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#75 - Hierarchical mixer

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#74 - test(model): no qrnn

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#73 - Scan

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#72 - L1 LayerNorm

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#71 - Scale

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 19 comments

#70 - Square LR-Schedule

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: engineering, core

#69 - SM3 instead of Adam in Adam#Shampoo

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#68 - add managed training, use tpucare for sweep

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#67 - high-performance multi-gpu video2tfrecord

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#66 - reduce epsilon yet improve stability

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#65 - perf(optimizer/shampoo): remove multi-preconditioning

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#64 - feat(optimizer): use normalized rmsprop

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#63 - faet(model): use more stable l2norm

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#62 - perf(optimizer): use adam+shampoo

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#61 - Learning-rate schedule as beta schedule

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment
Labels: research, core

#60 - MuP Normalization

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment
Labels: research, core

#59 - MESA/SAM

Pull Request - State: closed - Opened by ClashLuke over 2 years ago

#58 - Self-Convolution

Issue - State: closed - Opened by ClashLuke over 2 years ago - 2 comments
Labels: research, engineering, core

#57 - Add configurable layer scales

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#56 - feat(model): add alibi conv

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#55 - Complex Momentum

Issue - State: closed - Opened by ClashLuke over 2 years ago - 2 comments
Labels: research, ML, core

#54 - Alternative Losses

Issue - State: open - Opened by ClashLuke over 2 years ago - 2 comments
Labels: research, ML, core

#53 - Balance update weights of depthwise vs. pointwise convolution

Issue - State: closed - Opened by ClashLuke over 2 years ago - 2 comments
Labels: research, ML, core

#52 - Hierarchical network

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 3 comments

#51 - Transfer weights across size

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 2 comments

#50 - Hierarchical Network

Issue - State: closed - Opened by ClashLuke over 2 years ago - 5 comments
Labels: research, ML, core

#49 - Long-Range-Arena Evaluation

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: engineering, ML, downstream

#48 - ALiBi Convolution

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment
Labels: research, ML, core

#47 - Rmsprop grafting

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#46 - Gradient Noise

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, ML, core

#45 - Retrieval Augmented Causal Generation

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, ML, core

#44 - Encoder-Decoder Architecture

Issue - State: open - Opened by ClashLuke over 2 years ago
Labels: research, ML, core

#43 - Initialize deep model from shallow model

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment
Labels: research, engineering, ML, core

#42 - Alternative Sampling Methods

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 2 comments

#41 - Allow broken TPUs + Fix inference

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#40 - Multi-Host Scaling

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#39 - Typical Sampling

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#38 - Reuse ("donate") Buffers

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#37 - Reuse Parameter-Buffers

Issue - State: closed - Opened by ClashLuke over 2 years ago - 1 comment
Labels: engineering, ML, core

#36 - Shampoo Refactor

Pull Request - State: closed - Opened by ClashLuke over 2 years ago - 1 comment

#35 - Optimizer Grafting

Issue - State: closed - Opened by ClashLuke over 2 years ago - 2 comments
Labels: research, ML, core

#34 - Shampoo Optimizer

Pull Request - State: closed - Opened by ClashLuke almost 3 years ago - 4 comments

#33 - Long-Context Model

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 2 comments
Labels: engineering, ML, core

#32 - Automated Eval-Demo Update

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, mlops

#31 - Automated Long-Running Experiments

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, mlops

#30 - Automated Integration Tests

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, mlops

#29 - Long-Context Experiments

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, ML

#28 - Pretrained Embeddings, Stop at EOS, Untied Embeddings

Pull Request - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment

#27 - Checkpoint, Restore and Inference

Pull Request - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment

#26 - Reduce Compile-Time

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 3 comments
Labels: research, engineering, ML, core

#25 - Chunked Cross-Entropy

Pull Request - State: closed - Opened by ClashLuke almost 3 years ago

#23 - "Resume" option for tokenizers

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, downstream

#22 - Release pretrained weights

Issue - State: open - Opened by ClashLuke almost 3 years ago

#21 - Language-Model Evaluation

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, ML, downstream

#20 - Frontend

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering

#19 - Web API

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: engineering

#18 - Inference CLI

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: engineering

#17 - Finalize checkpoint/restore

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: engineering

#16 - Stabilize MoE

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: engineering, ML, core

#15 - Shampoo Optimizer

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: research, engineering, ML, core

#14 - Momentum Quantization

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: research, engineering, ML, core

#13 - Scaling

Issue - State: closed - Opened by ClashLuke almost 3 years ago
Labels: research, engineering, ML, core

#12 - Non-Autoregressive Generation

Issue - State: open - Opened by ClashLuke almost 3 years ago - 2 comments
Labels: research, ML, downstream

#11 - Image Classification

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: research, ML, downstream

#10 - Tokenizing Phonetics

Issue - State: open - Opened by ClashLuke almost 3 years ago - 7 comments
Labels: research, ML, downstream

#9 - Audio Modelling

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: research, ML

#8 - Explicit Memory

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: research, ML, core

#7 - Faster QRNN

Issue - State: closed - Opened by ClashLuke almost 3 years ago - 11 comments
Labels: research, ML, core

#6 - MoE + Weight Sharing

Issue - State: open - Opened by ClashLuke almost 3 years ago
Labels: research, ML, core

#5 - Video Generation via Tokens

Issue - State: open - Opened by ClashLuke almost 3 years ago - 1 comment
Labels: research, ML, downstream

#4 - perf(model): replace pjit with pmap

Pull Request - State: closed - Opened by ClashLuke almost 3 years ago - 2 comments

#3 - Weights and Biases

Pull Request - State: closed - Opened by JackMcCoy over 3 years ago - 1 comment

#2 - Infrence

Pull Request - State: closed - Opened by XMaster96 over 3 years ago

#1 - todo list

Issue - State: closed - Opened by ClashLuke over 3 years ago

GitHub / HomebrewNLP/Olmax issues and pull requests