Optimization research at LBNL focuses on the development of efficient optimization algorithms for computationally expensive black box problems. These problems arise in many science areas. For example, we are collaborating with domain scientists who develop complex simulation models to study physical phenomena such as climate, cosmology, combustion, etc . These simulation models have parameters whose adjustment significantly influences how well the simulation represents reality. The goal is to find the best parameters that minimize the error between the simulation and the observations. In other science areas, we encounter design problems in which we want to maximize (or minimize) a quantity, for example, the number of electrons that are emitted from laser-plasma accelerators, or in the design of an airfoil where the drag should be minimized and the lift maximized simultaneously.

Optimization problem characteristics

The optimization problems we are working on all have the following characteristics :

  • Objective function values are computed by a time consuming black-box simulation model
  • Derivative information is not available
  • Problems are multimodal, i.e., there are several local optima
  • Large parameter space; the parameters can be continuous, mixed-integer, integer, binary

Additional challenges are posed by the presence of the following characteristics

  • Constraints may be present; they may be computationally cheap or computed by an expensive black-box simulation model
  • The simulation model may not return a function value (not a bug in the simulation model, e.g., may be some exception case in the simulation model)
  • Multiple conflicting expensive black box objective functions have to be optimized simultaneously
  • Multiple levels of simulation model fidelity are available
  • Uncertainty, for example, in observation data that is used to calibrate simulations, in simulation outputs, in simulation parameters, in model choice (model discrepancy)

Challenge: simulation models are becoming increasingly complex

Goal: find a near-optimal solution by doing only very few expensive simulation model evaluations in order to keep the optimization time acceptable

  • Traditional algorithms such as evolutionary algorithms or gradient-based methods require too many expensive function evaluations to find near-optimal solutions
  • Function evaluations are becoming even more expensive as simulation models become increasingly complex and of higher fidelity, requiring even more computational resources

Sophisticated optimization algorithms are needed to efficiently and effectively find the optimal simulation model parameters

Our Approach: use surrogate models to approximate the expensive functions

We use computationally cheap approximation models (surrogate models, s(x)) of the expensive functions, f(x), in order to predict function values at unsampled points and to guide the optimization search for improved solutions. In general, surrogate model optimization works as follows:

  • Step 1: Create an initial experimental design. Evaluate f(x) at the selected points. Fit the surrogate model s(x) to the data: f(x) = s(x) + e(x), where e(x) is the difference between the true simulation model output and the surrogate model prediction
  • Step 2: Use s(x) to select a new evaluation point xnew and compute f(xnew).
  • Step 3: If the stopping criterion is not satisfied, update s(x) with the new data and go to Step 2. Otherwise, stop.