A Dynamic Programming Heuristic for Dense Hessian Chain Bracketing

Second derivatives of mathematical models for real-world phenomena are fundamental ingredients of a wide range of numerical simulation methods including parameter sensitivity analysis, uncertainty quantification, nonlinear optimization and model calibration. The evaluation of such Hessians often dominates the overall computational effort. Various combinatorial optimization problems can be formulated based on the highly desirable exploitation of the associativity of the chain rule of differential calculus. The fundamental Hessian Accumulation problem aiming to minimize the number of floating-point operations required for the computation of a Hessian turns out to be NP-complete. The restriction to suitable subspaces of the exponential search space proposed in this paper ensures computational tractability while yielding improvements by factors of ten and higher over standard approaches based on second-order tangent and adjoint algorithmic differentiation. Motivated by second-order parameter sensitivity analysis of surrogate numerical models obtained through training and pruning of deep neural networks this paper focusses on bracketing of dense Hessian chain products with the aim of minimizing the total number of floating-point operations to be performed. The results from a given dynamic programming algorithm for optimized bracketing of the underlying dense Jacobian chain product are used to reduce the computational cost of the corresponding Hessian. Minimal additional algorithmic effort is required.