Requirements for building, running and developing Finesse¶
Finesse requires Python 3.8 or higher. Python 3.8 is a hard requirement due to the use
of positional-only arguments, so-called walrus operators, the use of
functools.singledispatchmethod and the requirement for
dict to be
reversible in the code.
Finesse additionally requires some system dependencies:
Python development header files
SuiteSparse 4.0 or higher
A C compiler (typically
These packages are only required to build Finesse, not to run it. Users who merely wish to use Finesse, not develop it, do not need to install them and can instead rely on the various pre-built packages that are available.
An optional system runtime dependency is GraphViz, used to visualise graphs of interferometer networks. If you don’t plan to use such features, you don’t need to install GraphViz.
If used, the Conda development environment installs GraphViz.
Finesse also depends on various Python packages to be able to build and run, as discussed in the next section.
Python package requirements for Finesse¶
Source code: setup.py, setup.cfg, pyproject.toml, environment.yml, environment-win.yml
As a short summary, runtime requirements should typically be listed in setup.cfg, environment.yml, and environment-win.yml, whereas build requirements should be listed in pyproject.toml, environment.yml, and environment-win.yml.
The following section describes the logic behind this redundancy.
Defining dependencies for Finesse¶
Finesse’s Python requirements are defined for maximum convenience to users and developers using a hybrid approach: they are listed using standards defined by PEP 517 and in Conda environment files.
In theory, with the PEP 517 configuration in setup.cfg and
pyproject.toml, a user or developer can simply run
install finesse or
pip install . to build and install Finesse either from PyPI
or from the local directory, respectively. This is the case for most Linux and OSX
systems on x86 architecture. Due to a lack of availability of dependencies on PyPI for
some systems (such as for AMD64 architecture), environment files for Conda (using the
conda-forge channel which provides the required PyPI packages across a wider variety of
systems) are provided. When used, Conda takes over the responsibility for providing the
required runtime and build dependencies for Finesse, allowing users to use the program
on a wider variety of systems.
Conda is the package manager, conda-forge is the name of the community-driven package channel that is used to install the dependencies for Finesse.
In practice, the hybrid approach means that any changes to Finesse’s runtime or build dependencies require changes to both setup.cfg or pyproject.toml (PEP 517 runtime and build dependencies, respectively), and environment.yml and environment-win.yml (Conda runtime and build dependencies on Linux/OSX and Windows, respectively).
PEP 517 builds¶
PEP 517 requires a Python function available on the current path to act as a build
backend (responsible for compiling the package and adding it to the path). This is
currently set in Finesse to
setuptools.build_meta, though this can be changed
later if a tool offering more useful features becomes available.
setuptools.build_meta backend requires runtime requirements, i.e. those
required to be installed in the environment for Finesse to run, to be listed in
setup.py files in the project root. It’s not strictly necessary
to use both files with
setuptools (indeed it’s now recommended to use only the
setup.cfg where possible), but because Finesse needs Cython extensions
with relatively complex configurations to be built by the backend it uses a
to define the required logic. It may be possible to remove
setup.py later (see
Build requirements, on the other hand, are defined in
PEP 517 recommends (but does not strictly require) that build frontends by default
set up an isolated environment in which the backend can build the project. This
means that build requirements are installed in a temporary location used only for
the building of the package, then deleted when the newly built package is moved to
the user’s environment and the temporary environment is removed. This behaviour is
usually beneficial because it helps to catch missing dependencies and removes the
need for build requirements (e.g.
Cython) to be available in the local
environment, but when debugging build issues it can be useful to switch this
behaviour off. In pip this can be done by adding the
pip install. Note that you’ll need to ensure the
pyproject.toml are installed.
When Finesse is to be built, by default
pip will create a temporary, isolated
environment with only the standard library available, then install the build
dependencies listed in
pyproject.toml using PyPI, before building using the
specified build backend. This isolation is intentional to ensure that the intended build
dependencies are the (only) ones used to build the package. Build isolation is useful to
determine whether an undeclared (Python) dependency is is involved in the build, which
could lead to trouble installing the package on other machines.
When Finesse is to be run, it simply uses the Python packages available on the current
path. If the user installed Finesse using
pip install finesse or
pip install .
these packages will have been installed from PyPI and added to the path.
Conda provides packages for a wide variety of systems, and, unlike pip, takes care of installing system dependencies on behalf of the user. For example, Conda can install the Finesse system requirement SuiteSparse without the user having to do it themselves.
Requirements for Finesse within a Conda environment are defined in environment.yml and environment-win.yml.
When building Finesse within a Conda environment, it is not necessary and often not
desirable to use pip’s PEP 517 isolated build system. As described above, part of the
PEP 517 default build process is to install build requirements using PyPI, but these
requirements are not always available from PyPI for the target platform. It is therefore
typically desirable to build Finesse on such platforms in-tree using
pip install .
--no-build-isolation, which tells pip to use the build dependencies from the existing
(Conda) environment. It’s for this reason that build dependencies, not just runtime
dependencies, are also defined in
Dependencies in the Conda environment files refer to their name in the conda-forge
channel, not their name on PyPI. Most Conda packages use the same name as their PyPI
counterparts, but some do not. For example, interchanging of hyphens and underscores
is allowed by pip but not by Conda. A good mapping of non-standard conda-forge
package names can be found as part of the
grayskull package here.
The extra dependencies for development (such as
sphinx for building the
documentation) are also defined in
therefore get installed when the user sets up their development environment.
The versions of the Finesse runtime requirements are set by a compromise between the
desire to use the latest (and usually greatest) versions and the need to help package
managers include Finesse alongside other tools with their own, possibly conflicting,
requirements. It is therefore best to set only minimum required versions in
leaving the maximum version open unless there are known incompatibilities.
We rely on the regular running of the tests in the continuous integration pipeline to catch issues created by the latest versions of requirements. In cases where a particular dependency version is identified to cause problems with Finesse, it can be forbidden in the version string for the respective dependency in the relevant file (see next section). It is also useful to leave a note next to the requirement to inform others (including package maintainers, who may not be Finesse developers) why this is the case.
Modification of requirements involves editing typically two or more files:
Runtime requirements are listed within the
install_requireskey of the
Build requirements are defined in the
requireskey within the
Extra requirements are defined in the
setup.cfgand can be used to specify optional requirements for users or developers. These are not installed by most frontends by default. The
extras_requiresection contains keys and values representing groups of requirements. For example, the key
docslists requirements for building the documentation. Extras are not currently defined in
environment-win.ymlas they cause issues solving the dependency tree. These are instead installable via pip from within the active Conda environment using
PEP 508 specifies the format for requirements strings.
As described in a previous section, the building of Finesse usually takes place
in an isolated environment, so most build requirements these can be pinned in
pyproject.toml to whatever version is desired without causing trouble elsewhere.
It is important, however, that at least the
are set to the same versions in
setup.cfg because parts
of the Finesse code are compiled and linked against the
numpy C ABI. Using
different versions of
cython for building and running Finesse
frequently results in numerous warnings issued by
numpy, or even errors. It’s
also very useful from the point of view of debugging to maintain binary
compatibility between Finesse built in an isolated environment (i.e. via the full
build process) and in-place (e.g. via
After modifying requirements, verify that the tests still pass and build process still
operates as expected. Also consider making the changes in a branch, then push to GitLab
to allow the continuous integration pipeline to build
and run using the updated dependencies too, before merging into the