Adaptive Experimentation with Meta’s Ax: A Practical Coding Guide

A Practical Coding Guide">

In this tutorial, we explore adaptive experimentation using Meta’s Ax with the modern Client API. We work through a complete workflow where we tune a RandomForest model on a synthetic classification dataset while balancing predictive accuracy against model footprint. We begin by defining a mixed search space with integer, float, log-scaled, and categorical parameters, then use Ax’s ask-tell optimization loop to run constrained Bayesian optimization, multi-objective optimization, and parameter-constrained experimentation. Along the way, we visualize convergence, inspect the Pareto frontier, use Ax’s built-in analysis tools, and persist the experiment for future reuse.

We begin by preparing the Colab environment and installing the required packages for Ax and scikit-learn. We import the core libraries for optimization, machine learning, plotting, logging, and reproducibility. We also configure warnings and Ax logging to keep the notebook output clean and focused on the experimental results.

We create a synthetic multi-class classification dataset and define a cross-validation strategy to evaluate Random Forest models. We build an evaluation function that returns both accuracy and model size, allowing us to measure performance and cost together. We then define a mixed search space with integer, float, log-scaled, and categorical parameters, along with a reusable ask-tell study runner.

We run a constrained single-objective Bayesian optimization study where we maximize accuracy while keeping model size below a fixed threshold. We use Ax to suggest hyperparameter configurations, evaluate them, and report both accuracy and model size back to the optimizer. We then extract the best feasible configuration and plot the best accuracy achieved over feasible trials.

We move from single-objective optimization to multi-objective optimization by jointly maximizing accuracy and minimizing model size. We use Ax to search for configurations that represent strong trade-offs between predictive performance and computational footprint. We then calculate and visualize the empirical Pareto frontier to understand how accuracy varies with model size.

We demonstrate parameter constraints using a simple two-dimensional synthetic optimization problem. We ask Ax to minimize the distance to a target point while enforcing the input constraint that the sum of the two variables remains below a boundary. We observe that the optimizer respects the constraint and finds the best feasible point near the constrained optimum.

We use Ax’s built-in analysis tools to generate diagnostic cards, such as sensitivity, cross-validation, and other experiment insights, when the environment supports them. We then save the completed experiment to a JSON file and reload it to verify that the optimization state is preserved. We finish by confirming that the tutorial covers constrained optimization, multi-objective trade-offs, analysis, and experiment persistence.