Fine-tuning Gemma 4 MoE — Customizing Arena #6 with 3.8B Active Parameters
Apply QLoRA to Gemma 4 26B MoE. Expert layer LoRA strategies, Dense vs MoE comparison, MoE-specific training tips, and Ollama deployment. LoRA Series Part 4.
Series: Part 1: LoRA Theory | Part 2: QLoRA + Custom Data | Part 3: Eval + Deploy | Part 4 (this post)
Parts 1-3 covered LoRA fundamentals through deployment using Qwen 2.5 7B. Part 4 levels up — we apply LoRA to a Gemma 4 MoE model.
Why Gemma 4? Three reasons:
- MoE architecture: 26B total params, only 3.8B active. Inference cost is 4B-class, but performance is Arena #6

The rest of this series is paid
- ✓ The whole “LoRA fine-tuning” series: 3 posts in English, 3 in Korean
- ✓ One payment, no expiry. Opens when you sign in with the email you pay with.
The series pass needs no account to pay. The subscription needs a free account.