Bayesian optimization

Bayesian optimization relies on a probabilistic surrogate model, most commonly a Gaussian process (GP), to approximate the unknown objective function. The GP provides a posterior distribution over functions given observed data, yielding both a mean prediction and an uncertainty estimate at any candidate point. An acquisition function then quantifies the utility of evaluating a new point, balancing exploitation (sampling where the mean is low) and exploration (sampling where uncertainty is high). Popular acquisition functions include expected improvement (EI), probability of improvement, and upper confidence bound (UCB). The method iteratively selects the point that maximizes the acquisition function, evaluates the true objective, updates the surrogate, and repeats. This approach is particularly effective when evaluations are costly, such as in drug design or aerodynamic simulations, where the number of evaluations is limited to a few hundred or less.

Bayesian optimization has been widely adopted for hyperparameter tuning of machine learning models, where it often outperforms grid search and random search in terms of sample efficiency. It is also used in experimental design, robotics for controller tuning, and materials science for discovering new compounds. Variants extend the basic framework to handle constraints, multi-objective problems, and high-dimensional spaces. For example, constrained Bayesian optimization incorporates feasibility constraints into the acquisition function, while multi-objective versions use Pareto front approaches. Parallel and asynchronous versions allow multiple evaluations to be run concurrently, reducing wall-clock time. Recent developments include the use of neural networks as surrogates, such as Bayesian neural networks and deep kernel learning, to scale to higher dimensions and larger datasets.

The theoretical underpinnings of Bayesian optimization stem from the regret bounds of multi-armed bandit problems. For Gaussian process surrogates, algorithms like GP-UCB achieve sublinear cumulative regret under certain smoothness assumptions, providing a principled guarantee of convergence to the global optimum. Expected improvement is known to be consistent under mild conditions, meaning it converges to the optimum as the number of evaluations grows. The choice of kernel and its hyperparameters significantly affects performance; automatic relevance determination (ARD) kernels allow the method to learn which dimensions matter most. Recent theoretical work has also analyzed the effect of noisy observations and misspecified priors, showing that robust acquisition functions can mitigate these issues. These results provide a solid foundation for the practical success of the method.

Beyond the standard GP-based approach, Bayesian optimization has been applied to niche problems such as tuning the parameters of particle accelerators and optimizing the design of photonic devices. A lesser-known variant, entropy search, directly seeks to reduce the uncertainty about the location of the optimum, rather than improving the surrogate globally. Another edge case is the use of Bayesian optimization in human-in-the-loop settings, where the objective is a subjective human preference, as in preference-based optimization. The method also appears in the context of automatic machine learning (AutoML) frameworks like Hyperopt and Spearmint, which have their roots in early academic prototypes. A notable historical detail is that the concept of using a probabilistic model to guide optimization dates back to the 1960s, with work by Kushner on the 'efficient global optimization' idea, predating the modern Gaussian process formulation by decades.

Acquisition function

A function that determines the next point to evaluate by balancing exploration and exploitation.

Gaussian process

A stochastic process used as a surrogate model, providing a distribution over functions.

Expected improvement

An acquisition function that measures the expected amount of improvement over the current best observation.

Black-box function

A function whose analytical form is unknown and can only be evaluated at specific points.