How to Fine-Tune LFM2 Using QLoRA and DPO: A Complete Step-by-Step Coding Tutorial on Google Colab

["In this tutorial, we embark on a comprehensive journey to fine-tune Liquid AI's LFM2 model through a complete open-source workflow. The process begins with loading the base LFM2 checkpoint with QLoRA, followed by preparing a chat-style supervised fine-tuning dataset. We then train a lightweight LoRA adapter using TRL and PEFT, and merge the adapter back into the model.

To further enhance the workflow, we extend it with DPO to demonstrate how response preference can be improved using chosen and rejected answers. The end result is a practical pipeline that transforms a base LFM2 model into an SFT-tuned, preference-aligned checkpoint, ready for further testing or deployment.", 'To initiate the fine-tuning process, we install all the required libraries for fine-tuning LFM2 inside Google Colab. This includes importing core tools from Transformers, TRL, PEFT, datasets, bitsandbytes, and PyTorch.

We also define the main training settings, detect available GPUs, and select the appropriate precision for efficient training.', "The next step involves loading the LFM2 base model with optional 4-bit quantization to reduce GPU memory usage. We prepare the tokenizer, set the padding token, and define a chat function for testing model responses. A baseline prompt is run to compare the model's behavior before and after fine-tuning.", 'We then load a chat-formatted supervised fine-tuning dataset, keeping only the messages column.

LoRA is configured for lightweight adapter-based training, and the SFT training settings are defined. The model is trained with SFT, the LoRA adapter is saved, and the improved model response is tested.', 'After clearing earlier training objects from memory to free GPU resources, we reload the base LFM2 model in fp16 or bf16 and attach the trained SFT LoRA adapter. The adapter is then merged into the base model, and the merged SFT checkpoint is saved for the next stage.

Optionally, we run DPO using prompt-chosen-and-rejected response pairs, configure another LoRA adapter for preference tuning, and train the SFT-merged model with DPO. Finally, the DPO adapter is merged, the final model checkpoint is saved, and the result is compared against earlier outputs.', "In conclusion, we have built a full fine-tuning pipeline for LFM2 using only open-source tools, including Transformers, TRL, PEFT, datasets, and bitsandbytes. By utilizing QLoRA to make training efficient on Colab GPUs, applying supervised fine-tuning to chat-formatted data, merging the trained adapter into the base model, and optionally further improving the model through DPO, we gain a clear understanding of how modern LLM fine-tuning works in practice.

The process takes us from loading the model to producing a final checkpoint that can be compared against the original baseline and prepared for deployment. Check out the Codes with Notebook here. Also, feel free to follow us on Twitter and don't forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter.