Default backend and performance in ADNLPModels

As illustrated in the tutorial on backends, ADNLPModels.jl use different backend for each method from the NLPModel API that are implemented. By default, it uses the following:

using ADNLPModels, NLPModels

f(x) = 100 * (x[2] - x[1]^2)^2 + (x[1] - 1)^2
T = Float64
x0 = T[-1.2; 1.0]
lvar, uvar = zeros(T, 2), ones(T, 2) # must be of same type than `x0`
lcon, ucon = -T[0.5], T[0.5]
c!(cx, x) = begin
  cx[1] = x[1] + x[2]
  return cx
end
nlp = ADNLPModel!(f, x0, lvar, uvar, c!, lcon, ucon)
get_adbackend(nlp)

ADModelBackend{
  ForwardDiffADGradient,
  ForwardDiffADHvprod,
  ForwardDiffADJprod,
  ForwardDiffADJtprod,
  SparseADJacobian,
  SparseADHessian,
  ForwardDiffADGHjvprod,
}

Note that ForwardDiff.jl is mainly used as it is efficient and stable.

Predefined backends

Another way to know the default backends used is to check the constant ADNLPModels.default_backend.

ADNLPModels.default_backend

Dict{Symbol, Type} with 12 entries:
  :hprod_residual_backend    => ForwardDiffADHvprod
  :jprod_backend             => ForwardDiffADJprod
  :jtprod_backend            => ForwardDiffADJtprod
  :gradient_backend          => ForwardDiffADGradient
  :hprod_backend             => ForwardDiffADHvprod
  :hessian_residual_backend  => ForwardDiffADHessian
  :ghjvprod_backend          => ForwardDiffADGHjvprod
  :hessian_backend           => SparseADHessian
  :jprod_residual_backend    => ForwardDiffADJprod
  :jtprod_residual_backend   => ForwardDiffADJtprod
  :jacobian_residual_backend => SparseADJacobian
  :jacobian_backend          => SparseADJacobian

More generally, the package anticipates more uses

ADNLPModels.predefined_backend

Dict{Symbol, Dict{Symbol}} with 3 entries:
  :default   => Dict{Symbol, Type}(:hprod_residual_backend=>ForwardDiffADHvprod…
  :generic   => Dict{Symbol, DataType}(:hprod_residual_backend=>GenericForwardD…
  :optimized => Dict{Symbol, Type}(:hprod_residual_backend=>ReverseDiffADHvprod…

The backend :optimized will mainly focus on the most efficient approaches, for instance using ReverseDiff to compute the gradient instead of ForwardDiff.

ADNLPModels.predefined_backend[:optimized]

Dict{Symbol, Type} with 12 entries:
  :hprod_residual_backend    => ReverseDiffADHvprod
  :jprod_backend             => ForwardDiffADJprod
  :jtprod_backend            => ReverseDiffADJtprod
  :gradient_backend          => ReverseDiffADGradient
  :hprod_backend             => ReverseDiffADHvprod
  :hessian_residual_backend  => ForwardDiffADHessian
  :ghjvprod_backend          => ForwardDiffADGHjvprod
  :hessian_backend           => SparseReverseADHessian
  :jprod_residual_backend    => ForwardDiffADJprod
  :jtprod_residual_backend   => ReverseDiffADJtprod
  :jacobian_residual_backend => SparseADJacobian
  :jacobian_backend          => SparseADJacobian

The backend :generic focuses on backend that make no assumptions on the element type, see Creating an ADNLPModels backend that supports multiple precisions.

It is possible to use these pre-defined backends using the keyword argument backend when instantiating the model.

nlp = ADNLPModel!(f, x0, lvar, uvar, c!, lcon, ucon, backend = :optimized)
get_adbackend(nlp)

ADModelBackend{
  ReverseDiffADGradient,
  ReverseDiffADHvprod,
  ForwardDiffADJprod,
  ReverseDiffADJtprod,
  SparseADJacobian,
  SparseReverseADHessian,
  ForwardDiffADGHjvprod,
}

Hessian and Jacobian computations

It is to be noted that by default the Jacobian and Hessian matrices are dense.

(get_nnzj(nlp), get_nnzh(nlp)) # number of nonzeros elements in the Jacobian and Hessian

(2, 3)

To enable sparse computations of these entries, one needs to first load the package Symbolics.jl

using Symbolics

and now

ADNLPModels.predefined_backend[:optimized][:jacobian_backend]

ADNLPModels.SparseADJacobian

ADNLPModels.predefined_backend[:optimized][:hessian_backend]

ADNLPModels.SparseReverseADHessian

Choosing another optimization problem with the optimized backend will compute sparse Jacobian and Hessian matrices.

f(x) = (x[1] - 1)^2
T = Float64
x0 = T[-1.2; 1.0]
lvar, uvar = zeros(T, 2), ones(T, 2) # must be of same type than `x0`
lcon, ucon = -T[0.5], T[0.5]
c!(cx, x) = begin
  cx[1] = x[2]
  return cx
end
nlp = ADNLPModel!(f, x0, lvar, uvar, c!, lcon, ucon, backend = :optimized)
(get_nnzj(nlp), get_nnzh(nlp))

(1, 1)

x = rand(T, 2)
jac(nlp, x)

1×2 SparseArrays.SparseMatrixCSC{Float64, Int64} with 1 stored entry:
  ⋅   1.0

The package Symbolics.jl is used to compute the sparsity pattern of the sparse matrix. The evaluation of the number of directional derivatives needed to evaluate the matrix is done by ColPack.jl.