AllenAI Open Instruct Tulu 3 Post-Training with SFT, DPO, RLVR, GRPO, and Verifier-Based Evaluation

In this tutorial, we build an end-to-end post-training pipeline for a compact instruction-tuned language model using AllenAI’s Open Instruct framework. We move through three major training stages: Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning with Verifiable Rewards using GRPO, while adapting the original multi-GPU Tulu 3 stack to fit within a 16 GB runtime. We clone the Open Instruct repository, selectively load its native loss and utility functions, configure LoRA adapters, prepare GSM8K data for each training stage, and use deterministic verifiers to evaluate generated mathematical answers. Throughout the workflow, we preserve the core optimization logic of Open Instruct while replacing distributed components such as vLLM, Ray actors, DeepSpeed, and asynchronous rollout queues with lightweight Hugging Face and PyTorch implementations suitable for Colab.

We install the required lightweight dependencies, clone the Open Instruct repository, and configure the Colab environment for stable execution. We detect the available GPU precision mode and select either FP16 or BF16 autocasting based on the hardware capabilities. We also extract the original DPO, GRPO, masking, and log-probability functions directly from the repository without importing its full distributed training stack.

We define a centralized configuration class that controls the model, dataset sizes, learning rates, batch settings, and optimization parameters for every training stage. We initialize the Open Instruct tokenizer while preserving the model’s chat template and ensuring that padding and end-of-sequence tokens remain correctly separated. We then tokenize a sample conversation and visualize which assistant tokens contribute to the supervised training loss.

We load GSM8K and transform its questions and solutions into a consistent conversational format for SFT, DPO, and RLVR training. We create supervised examples, preference pairs with deliberately incorrect final answers, and verifier-ready prompts with structured ground-truth labels. We also initialize Open Instruct’s GSM8K, mathematical, and instruction-following verifiers and use them to score generated responses deterministically.

We load the Qwen instruction model, apply LoRA adapters to its attention and feed-forward projection layers, and restrict optimization to the trainable adapter parameters. We configure mixed-precision execution, gradient scaling, gradient clipping, learning-rate scheduling, and temporary KV-cache activation for generation. We then evaluate the untrained baseline on GSM8K using greedy decoding and verifier-based answer accuracy.

We construct a padded SFT DataLoader and train the LoRA adapters on tokenized GSM8K conversations using gradient accumulation. We optimize the model with cross-entropy loss calculated only over the unmasked assistant response tokens. We track the training loss and learning rate throughout the stage and evaluate the updated model after supervised fine-tuning.