### Automatic Differentiation

### Related Products

### Related Services

**Learn more about Automatic Differentiation**

### Overview

Automatic Differentiation is an algorithmic means to numerically evaluate the derivatives of functions given by a computer program. Partial derivatives of all specified dependent variables are calculated with respect to all specified independent variables based on the repeated application of the chain rule given by derivative calculus.

Automatic Differentiation overcomes critical
drawbacks that are encountered in
*differentiation by hand*,
*symbolic differentiation*, or *finite
differentiation*:

- No more time consuming and error-prone tasks leading to excessive use of human resources
- No more low speed in symbolic differentiation
- No more difficulties in converting computer programs into expressions meaningful to computer algebra
- No more round-off errors due to discretization
- No more cancellation errors due to finite number precision

### Accuracy

Why is the accuracy of your derivatives so important, and
why should you care about the round-off and cancellation errors
of finite differences? The answer is simple: these
errors, although small, destroy
the convergence of numerical solvers for
non-linear systems of equations. For example,
Newton-Raphson solvers require
derivatives that are more accurate than obtained
from finite differencing in order to achieve *quadratic*
convergence rates (see Press et al.,
in *Numerical Recipes,
Cambridge University Press, 2nd ed.*)
Many methods in non-linear optimization show
similar properties and requirements. In addition, stronger
non-linearities or more equations generally
demand very high accuracy.

By applying derivative calculus, Automatic Differentiation
does not suffer from the round-off and cancellation errors
seen in finite differencing:
machine-precision, accurate derivatives are directly obtained.
Thus, Automatic Differentiation combines the
advantages of *differentiation by hand* with respect to
accuracy while simultaneously eliminating the
time consuming, error-prone and complex tasks
inherent to most other solutions.

### Performance

Despite the drawbacks of
*differentiation by hand*,
many R&D teams still adhere to this
method, because traditional
Automatic Differentiation software tools
are lacking in efficiency and performance.

High Performance is therefore one of the key issues
in Automatic Differentiation. Only if
automatic derivative calculation can compete in speed with
hard-wired solutions is it
acceptable to be incorporated in realistic
applications. The performance issue has
driven many *source-to-source* code
transformation tools,
where efficient derivative code is generated.
However, these tools are most often

- Unstructured in the specification of dependent and independent variables,
- Have difficulties with modern language features, e.g., as in the array operations of FORTRAN 90/95,
- Are not fully compliant with the latest FORTRAN or C/C++ standards, or
- Do not produce warnings in case of mathematically non-differentiable situations.

In summary, *source-to-source* transformation tools
usually lack broad usability. R&D teams spend more
time *getting it to work and running* than they
can and should afford.

The competing method to *source-to-source*
technique is the *operator overloading* approach,
which is now possible with modern FORTRAN and C/C++ compilers.
Usability and compatibility of Automatic Differentiation
software based on *operator overloading* is undoubtedly
superior to the old *source-to-source* transformation
tools. But can it compete with performance?

*Vivlabs* Automatic Differentiation software for FORTRAN
and C/C++ demonstrates that winning performance
is possible with the *operator overloading*
approach. It is achieved with *vivlabs* built-in sparsity
exploits and complexity reductions.

### Usability

Accuracy and performance are two key requirements an Automatic Differentiation tool must meet before it should be considered for use in your real-world R&D applications. However, overall success also depends on the usability of the Automatic Differentiation software. Sooner or later you need to address the following questions:

- How long does it take to add Automatic Differentiation capabilities to your software, what is the impact on your human resources ?
- What changes are necessary to the existing code? Is complexity being added that is more costly to maintain?
- Does your Automatic Differentiation solution make it more complicated to make changes to your existing code?
- Do you need to port your code because your tool cannot cope with the full FORTRAN and C/C++ standards?
- What do you do if your code is written in both FORTRAN and C/C++?

*Vivlabs* provides answers and the best solutions to
all of these important questions. Our software solutions
leap ahead in usability,
thereby giving you an invaluable edge over the
competition.

### Why choose *vivlabs*?

*Vivlabs* strategy is to deliver winning
Automatic Differentiation software tools
that combine at the same time Accuracy,
Performance and
Usability.
In addition, our premier
Support Services
will help you get most out of *vivlabs*
software products.

### Look and Feel

A simple example demonstrates the look and feel of
the *vivlabs* ADF software for FORTRAN. Suppose
you want to calculate first-order derivatives
with respect to the independent vector variable 'x':

01subroutineexample(f, x) 02 03usemod_ADF 04 05type(ADF_dpr) :: f(2), x(2) 06 07 ! ... some computations (no active variables) 08 09 ! x becomes independent 10callADF_independent((/1,2/), x, (/1.0,5.0/)) 11 12 f = sin(x**2) 13 14 ! ... more computations including function calls, 15 subroutine calls and temporary variables 16 16 ! retrieve derivatives 18write(*,*)"df/dx(1) = ", ADF_deriv(f,1) 19 20end subroutineexample

Changes to the original code are highlighted in blue color. The differentiation process is performed completely in the background and the code containing the mathematical evaluation (lines 12-15) is not changed, even though it contains array arithmetic.