Requirements for building, running and developing Finesse

Finesse requires Python 3.9 or higher. Python 3.8 is a hard requirement due to the use of positional-only arguments, so-called walrus operators, the use of functools.singledispatchmethod and the requirement for dict to be reversible in the code. The requirement for Python 3.9 comes from using Cython 3 with SciPy.

Finesse additionally requires some system dependencies:

  • Python development header files

  • SuiteSparse 4.0 or higher

  • A C compiler (typically gcc)

Note

These packages are only required to build Finesse, not to run it. Users who merely wish to use Finesse, not develop it, do not need to install them and can instead rely on the various pre-built packages that are available.

An optional system runtime dependency is GraphViz, used to visualise graphs of interferometer networks. If you don’t plan to use such features, you don’t need to install GraphViz.

Note

If used, the Conda development environment installs GraphViz.

Finesse also depends on various Python packages to be able to build and run, as discussed in the next section.

Python package requirements for Finesse

Source code: setup.py, pyproject.toml, environment.yml, environment-win.yml, generate_environment_files.py

As a short summary, requirements should be manually added to pyproject.toml. There is a pre-commit hook that automatically regenerates the environment.yml and environment-win.yml files.

This ensures these three requirements files are kept in sync.

There is another pre-commit hook that ensures that requirements that are listed multiple times in pyproject.toml are identical (NumPy is listed as a build requirement and a runtime requirement for instance)

The following section describes the logic behind this redundancy.

Defining dependencies for Finesse

Finesse’s Python requirements are defined for maximum convenience to users and developers using a hybrid approach: they are listed using standards defined by PEP 517 and in Conda environment files.

In theory, with the PEP 517 configuration in pyproject.toml, a user or developer can simply run pip install finesse or pip install . to build and install Finesse either from PyPI or from the local directory, respectively. This is the case for most Linux and OSX systems on x86 architecture. Due to a lack of availability of dependencies on PyPI for some systems (such as for AMD64 architecture), environment files for Conda (using the conda-forge channel which provides the required PyPI packages across a wider variety of systems) are provided. When used, Conda takes over the responsibility for providing the required runtime and build dependencies for Finesse, allowing users to use the program on a wider variety of systems.

Note

Nomenclature

Conda is the package manager, conda-forge is the name of the community-driven package channel that is used to install the dependencies for Finesse.

In practice, the hybrid approach means that any changes to Finesse’s runtime or build dependencies require changes to both pyproject.toml (PEP 517 runtime and build dependencies, respectively), and environment.yml and environment-win.yml (Conda runtime and build dependencies on Linux/OSX and Windows, respectively).

PEP 517 builds

PEP 517 requires a Python function available on the current path to act as a build backend (responsible for compiling the package and adding it to the path). This is currently set in Finesse to setuptools.build_meta, though this can be changed later if a tool offering more useful features becomes available.

The setuptools.build_meta backend requires runtime requirements, i.e. those required to be installed in the environment for Finesse to run, to be listed in pyproject.toml and/or setup.py files in the project root. It’s not strictly necessary to use both files with setuptools (indeed it’s now recommended to use only the declarative pyproject.toml where possible), but because Finesse needs Cython extensions with relatively complex configurations to be built by the backend it uses a setup.py to define the required logic. It may be possible to remove setup.py later (see #367).

Build requirements, on the other hand, are defined in pyproject.toml.

Note

PEP 517 recommends (but does not strictly require) that build frontends by default set up an isolated environment in which the backend can build the project. This means that build requirements are installed in a temporary location used only for the building of the package, then deleted when the newly built package is moved to the user’s environment and the temporary environment is removed. This behaviour is usually beneficial because it helps to catch missing dependencies and removes the need for build requirements (e.g. Cython) to be available in the local environment, but when debugging build issues it can be useful to switch this behaviour off. In pip this can be done by adding the --no-build-isolation flag to pip install. Note that you’ll need to ensure the inplacebuild extras listed in pyproject.toml are installed.

When Finesse is to be built, by default pip will create a temporary, isolated environment with only the standard library available, then install the build dependencies listed in pyproject.toml using PyPI, before building using the specified build backend. This isolation is intentional to ensure that the intended build dependencies are the (only) ones used to build the package. Build isolation is useful to determine whether an undeclared (Python) dependency is is involved in the build, which could lead to trouble installing the package on other machines.

When Finesse is to be run, it simply uses the Python packages available on the current path. If the user installed Finesse using pip install finesse or pip install . these packages will have been installed from PyPI and added to the path.

Conda builds

Conda provides packages for a wide variety of systems, and, unlike pip, takes care of installing system dependencies on behalf of the user. For example, Conda can install the Finesse system requirement SuiteSparse without the user having to do it themselves.

Note

Conda commands can be notoriously slow to execute. Mamba is a much faster drop-in replacement for all conda commands (e.g. conda install can be replaced with mamba install). Alternatively, you can add the --solver libmamba argument when using conda commands.

Requirements for Finesse within a Conda environment are defined in environment.yml and environment-win.yml.

When building Finesse within a Conda environment, it is not necessary and often not desirable to use pip’s PEP 517 isolated build system. As described above, part of the PEP 517 default build process is to install build requirements using PyPI, but these requirements are not always available from PyPI for the target platform. It is therefore typically desirable to build Finesse on such platforms in-tree using pip install . --no-build-isolation, which tells pip to use the build dependencies from the existing (Conda) environment. It’s for this reason that build dependencies, not just runtime dependencies, are also defined in environment.yml and environment-win.yml.

Warning

Dependencies in the Conda environment files refer to their name in the conda-forge channel, not their name on PyPI. Most Conda packages use the same name as their PyPI counterparts, but some do not. For example, interchanging of hyphens and underscores is allowed by pip but not by Conda. A good mapping of non-standard conda-forge package names can be found as part of the grayskull package here.

The extra dependencies for development (such as sphinx for building the documentation) are also defined in environment.yml and environment-win.yml, and therefore get installed when the user sets up their development environment.

Warning

The conda command-line interface is (at times of writing) a confusing, badly documented mess. They are working to improve it, but progress seems to be slow. The main confusion is the presence of conda env. It is an older, completely separate program that was merged into the conda codebase. It supports the environment.yml file syntax and is always aware of the environment it is running in, allowing for a simple conda env update/create environment.yml to either update or create an environment.

It unfortunately does not support any of the newer conda features, like specifying the channel with the -c argument or specifying multiple requirement files. The install subcommand is also missing.

The default conda command supports all of the above and adds a few useful commands like search and clean. However, it is not aware of the current environment and always needs to specify the environment with the -n command. On top of that it does not support the environment.yml file syntax at all!

Version pinning

The versions of the Finesse runtime requirements are set by a compromise between the desire to use the latest (and usually greatest) versions and the need to help package managers include Finesse alongside other tools with their own, possibly conflicting, requirements. It is therefore best to set only minimum required versions in pyproject.toml, environment.yml, and environment-win.yml, leaving the maximum version open unless there are known incompatibilities.

We rely on the regular running of the tests in the continuous integration pipeline to catch issues created by the latest versions of requirements. In cases where a particular dependency version is identified to cause problems with Finesse, it can be forbidden in the version string for the respective dependency in the relevant file (see next section). It is also useful to leave a note next to the requirement to inform others (including package maintainers, who may not be Finesse developers) why this is the case.

Modifying requirements

Modification of requirements should only be done in the pyproject.toml file, the conda files should update automatically with the pre-commit hook:

  • Runtime requirements are listed within the dependencies key of the [project] section of pyproject.toml.

  • Build requirements are defined in the requires key within the [build-system] section and the inplacebuild key of the [project.optional-dependencies] of pyproject.toml.

  • Extra requirements are defined in the [project.optional-dependencies] section of pyproject.toml and can be used to specify optional requirements for users or developers. These are not installed by most frontends by default. For example, the key docs lists requirements for building the documentation. Extras are duplicated in environment.yml / environment-win.yml. Note that conda does not support optional dependencies, so the extra dependencies always get installed when using conda.

PEP 508 specifies the format for requirements strings.

Warning

As described in a previous section, the building of Finesse usually takes place in an isolated environment, so most build requirements these can be pinned in pyproject.toml to whatever version is desired without causing trouble elsewhere. It is important, however, that at least the numpy requirement are set to the same versions in pyproject.toml because parts of the Finesse code are compiled and linked against the numpy C ABI. Using different versions of numpy for building and running Finesse frequently results in numerous warnings issued by numpy, or even errors. It’s also very useful from the point of view of debugging to maintain binary compatibility between Finesse built in an isolated environment (i.e. via the full build process) and in-place (e.g. via make).

After modifying requirements, verify that the tests still pass and build process still operates as expected. Also consider making the changes in a branch, then push to GitLab to allow the continuous integration pipeline to build and run using the updated dependencies too, before merging into the develop branch.