The Challenge of Low-Resource Languages in AI
What happens when you need to build an AI language model for a language that most foundation models barely understand? This was the exact challenge facing Azercell Telecom LLC, Azerbaijan's leading telecommunications provider, when they set out to create an Azerbaijani large language model (LLM) for telecom use cases and customer support.
The problem with Azerbaijani—and many other morphologically rich languages—is that standard English-optimized tokenizers fragment complex word forms inefficiently. For example, the Azerbaijani word "kitablardan" (meaning "from the books") gets split into multiple subword tokens, dramatically reducing how much actual content fits within a model's context window.
A Three-Stage Solution Framework
In a six-week collaboration with the AWS Generative AI Innovation Center, Azercell developed a production-ready framework on Amazon SageMaker AI that delivered remarkable results:
- 23% higher training throughput
- 58% lower peak GPU memory usage
- 2x improvement in tokenization efficiency
Their approach consisted of three sequential stages, each building on the previous one:
Stage 1: Custom Tokenizer Development
The team evaluated three different tokenization approaches and discovered that building a custom monolingual tokenizer specifically for Azerbaijani was game-changing. Using Byte-Level Byte-Pair Encoding (BBPE), they created a tokenizer that reduced tokens per word from 3.22 (baseline) to 1.59—effectively doubling the amount of Azerbaijani text that could fit within Llama 3.2's 128k-token context window.
This means approximately 80,000 words versus 40,000 with standard tokenizers—a massive practical advantage for real-world applications.
Stage 2: Continued Pre-Training with Advanced Optimizations
The team adapted Llama 3.2 1B to understand Azerbaijani using two key optimization techniques:
Distributed Training with FSDP: By implementing Fully Sharded Data Parallel training, they reduced per-GPU model state memory from 9.23 GB to just 1.17 GB, freeing up space for larger batch sizes and better hardware utilization.
Liger Kernel Integration: These memory-efficient, Triton-based implementations fuse multiple operations into single GPU kernel launches, dramatically reducing intermediate memory allocations while maintaining numerical accuracy.
Stage 3: Conversational Fine-Tuning with LoRA
The final stage transformed the pre-trained model into a conversational assistant using Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that significantly reduces the number of trainable parameters needed.
Key Insights for Prompt Engineers and AI Practitioners
Tokenization Matters More Than You Think
This project demonstrates that custom tokenization can be as important as model architecture for non-English languages. The 2x improvement in tokenization efficiency directly translates to more effective prompts and better model performance.
Memory Optimization Enables Scaling
The 58% reduction in GPU memory usage through kernel-level optimizations shows how low-level improvements can have massive practical impacts. This approach makes it feasible to train larger models or use smaller, more cost-effective hardware.
Modular Architecture for Flexibility
The three-stage approach creates a modular pipeline where improvements to tokenization benefit all subsequent training stages, and continued pre-training configurations transfer across different fine-tuning tasks.
Practical Applications Beyond Azerbaijani
While this project focused on Azerbaijani, the techniques apply to many low-resource and morphologically complex languages including:
- Turkish, Finnish, and other agglutinative languages
- Arabic and Hebrew with their rich morphological systems
- Indigenous languages with limited training data
The Broader Impact
This collaboration between Azercell and AWS proves that specialized language models for underserved languages are not just possible—they're practical and economically viable. The framework's cost-effectiveness comes from using Amazon SageMaker's pay-per-use model, where training jobs provision fresh instances and terminate after completion, eliminating idle cluster costs.
For the global AI community, this work provides a blueprint for democratizing language AI beyond English and other high-resource languages, opening up new possibilities for millions of speakers worldwide.
Source: Training Azerbaijani language models on Amazon SageMaker AI by Aleksei Iancheruk