Numerical precision

In elements

Normal integers and floats are represented in elements using Python data types. That means integers have arbitrary precision and are only limited by system memory, while floats are usually limited to 53 bits of precision, equivalent to about 16 or 17 digits (a good discussion can be found here).

Numerical arrays in Finesse elements, e.g. for the computation of ABCD matrices, are represented internally using Numpy data structures. Most integers are stored using int64 representation. Larger numbers use uint64 then default back to Python int objects.

The precision limits for integers and floats for your machine can be found in the following way using Numpy:

import numpy as np

For integers:

print(np.iinfo(np.uint64))

Machine parameters for uint64
---------------------------------------------------------------
min = 0
max = 18446744073709551615
---------------------------------------------------------------

For floats:

print(np.finfo(np.float64))

Machine parameters for float64
---------------------------------------------------------------
precision =  15   resolution = 1.0000000000000001e-15
machep =    -52   eps =        2.2204460492503131e-16
negep =     -53   epsneg =     1.1102230246251565e-16
minexp =  -1022   tiny =       2.2250738585072014e-308
maxexp =   1024   max =        1.7976931348623157e+308
nexp =       11   min =        -max
smallest_normal = 2.2250738585072014e-308   smallest_subnormal = 4.9406564584124654e-324
---------------------------------------------------------------

As an example, the following shows the switch from uint64 to object in Numpy:

# Using the maximimum value, we get a uint64.
print(repr(np.array(18446744073709551615)))

# Using one more than the maximum value, we get an object.
print(repr(np.array(18446744073709551616)))

array(18446744073709551615, dtype=uint64)
array(18446744073709551616, dtype=object)

Only numbers that can be represented by Numpy data types (either as scalars or arrays) are fully supported in Finesse. Finesse needs access to certain Numpy functions (e.g. numpy.cos) and these are only implemented for Numpy data types. This effectively places a limit on the minimum and maximum numbers in |Finesse| to those of Numpy.

In simulations

Todo

describe numerical precision in Finesse simulations (KLU etc.).