From LLaMA to DeepSeek Multi-head Latent Attention with Knowledge Distillation: Making InkubaLM Fast and Efficient
Redesigning InkubaLM's attention architecture (LLaMA → DeepSeek MLA) for fast on-device inference while preserving its low-resource language knowledge through distillation.
Read more
InkubaLM-0.4B is a small language model developed by Lelapa AI's Fundamental Research Team for low-resource African languages. It is built on a LLaMA-based architecture with approximately 0.4B parameters, designed to make useful language technology accessible under limited compute and data constraints. This is independent research I do because I care about low-resource languages.
My current research direction is to investigate whether InkubaLM's attention architecture can be redesigned around DeepSeek's Multi-head Latent Attention (MLA) to reduce inference overhead and make the model more practical for on-device deployment. DeepSeek introduced MLA as a mechanism for compressing the key-value representation, reducing the memory cost of the KV cache during inference.
Rather than simply fine-tuning the existing model, I am exploring rebuilding the modified architecture from scratch and studying how to transfer the capabilities of the original InkubaLM into the new architecture through knowledge distillation. The objective is to preserve as much of the original model's linguistic knowledge as possible while delivering a more inference-efficient architecture that can run efficiently on phones and other local, resource-constrained hardware.
At this stage, this is early research: I am primarily reading the relevant literature, studying the InkubaLM architecture and DeepSeek's attention mechanism, and experimenting with how the two approaches could be combined. The long-term goal is to contribute an African-language model that can run efficiently on phones and other local, resource-constrained hardware.
