Metadata-Version: 2.4
Name: gbra
Version: 0.1.1
Summary: GBRA (Gember-Berke Rank Adaptation): DoRA-style fine-tuning with a buffered, lazily refreshed weight norm.
Author: Berke T. Adanc
License: MIT
Keywords: lora,dora,peft,fine-tuning,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: safetensors>=0.4
Provides-Extra: hf
Requires-Dist: transformers>=4.40; extra == "hf"
Provides-Extra: 4bit
Requires-Dist: bitsandbytes>=0.43; extra == "4bit"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: transformers>=4.40; extra == "test"
Dynamic: license-file

# GBRA: Gember-Berke Rank Adaptation

*Pronounced "Gember RA". A GEMBER && Apex Logiç Studio project by Berke T. Adanç.*

GBRA is a DoRA-style parameter-efficient fine-tuning method. It keeps DoRA's split of each weight into a learned **magnitude** and a low-rank-updated **direction**, but **buffers the weight norm** instead of recomputing it in every forward pass.

```
DoRA:  W' = m · (W + s·BA) / ‖W + s·BA‖     ← norm recomputed every forward (and again during gradient-checkpoint recompute)
GBRA:  W' = m · (W + s·BA) / n               ← n is a stored buffer, refreshed every K optimizer steps
```

**Why it helps:** With gradient accumulation, the weights only change once per optimizer step. Recomputing the norm for every micro-batch repeats the same work many times. GBRA computes it once and reuses it.

## Install

```bash
pip install gbra             # core
pip install "gbra[hf]"       # + Hugging Face Trainer callback
pip install "gbra[4bit]"     # + 4-bit (bitsandbytes) base models
```

## Usage

```python
from transformers import AutoModelForCausalLM, Trainer
from gbra import apply_gbra, GBRACallback, save_gbra, merge_gbra

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = apply_gbra(model, r=16, alpha=16)          # wraps q/k/v/o/gate/up/down projections

trainer = Trainer(model=model, ..., callbacks=[GBRACallback(refresh_every=10)])
trainer.train()

save_gbra(model, "my_adapter")                     # small adapter (A, B, m, n)
merge_gbra(model)                                  # fold into plain nn.Linear for export
```

Turkish aliases are also available: `gbra_ekle`, `tum_normlari_yenile`, `gbra_kaydet`, `gbra_yukle`, `gbra_birlestir`.

## Preliminary results

Qwen2.5-0.5B-Instruct, 3,000 Turkish chat/tool-use examples, 375 optimizer steps (batch 1 × 8 accumulation), r=16, single Kaggle T4, fp16 autocast. Same data, seed and hyper-parameters for every method. DoRA and LoRA via 🤗 PEFT.

| Method | s / step | vs LoRA | Peak memory | Val. loss |
|---|---|---|---|---|
| LoRA | **5.77** | 1.00× | 8.11 GB | 1.6376 |
| DoRA (PEFT) | 9.10 | 1.58× | 8.11 GB | 1.6285 |
| **GBRA K=10** | 6.47 | 1.12× | 8.11 GB | **1.6249** |
| GBRA K=100 | 6.47 | 1.12× | 8.11 GB | 1.6253 |
| GBRA K=never | 6.46 | 1.12× | 8.11 GB | 1.6263 |

**In this run, GBRA was ~29% faster than DoRA per step, with equal or slightly lower validation loss.** A shorter 125-step run gave the same speed ratio.

### Limitations (please read)

- **Single seed, single small model, single dataset.** The loss differences between GBRA variants and DoRA are small (≈0.001–0.004) and may be noise; repeated runs with more seeds are in progress.
- Much of the speed-up comes from **not recomputing the norm per micro-batch**. With no gradient accumulation and no gradient checkpointing, the gap to DoRA should be smaller.
- The comparison is against PEFT's DoRA implementation at the time of testing, not against every possible DoRA implementation.
- 4-bit (bitsandbytes) support is implemented but **has not yet been benchmarked**.
- Unlike PEFT LoRA, GBRA does not currently use fused kernels such as Unsloth's.

## Credits

GBRA builds directly on **DoRA** (Liu et al., 2024, *"DoRA: Weight-Decomposed Low-Rank Adaptation"*) and **LoRA** (Hu et al., 2021). Related ideas on learned scaling vectors appear in (IA)³ and VeRA. The contribution here is the buffered / lazily refreshed norm and its evaluation.

The core `forward` of `GBRALinear` was written by Berke T. Adanç. Parts of the surrounding library and tests were developed with the help of an AI assistant.

## Changelog

- **0.1.1**: Python 3.9 compatibility fix in `GBRACallback`; new `refresh_on_end` option (refreshes the norm once more when training ends); README install instructions for PyPI.
- **0.1.0**: First release.

## License

MIT

---

## Türkçe özet

GBRA, DoRA'nın "büyüklük + yön" ayrımını korur, ama ağırlık normunu her adımda yeniden hesaplamak yerine **saklar** ve sadece **K optimizer adımında bir** günceller. Gradient birikimi kullanılan eğitimde ağırlıklar mini-batch'ler arasında değişmediği için DoRA aynı normu tekrar tekrar hesaplar. GBRA bu tekrarı ortadan kaldırır.

İlk deneyde (Qwen2.5-0.5B, 375 adım, tek seed) GBRA, DoRA'dan **~%29 daha hızlı** ve doğrulama loss'u aynı ya da biraz daha iyiydi. Sonuçlar tek seed'e dayandığı için **ön sonuç** olarak değerlendirilmelidir.
