Requirements for building, running and developing Finesse

Finesse requires Python 3.8 or higher. Python 3.8 is a hard requirement due to the use of positional-only arguments, so-called walrus operators, the use of functools.singledispatchmethod and the requirement for dict to be reversible in the code.

Finesse additionally requires some system dependencies:

  • Python development header files

  • SuiteSparse 4.0 or higher

  • A C compiler (typically gcc)

Note

These packages are only required to build Finesse, not to run it. Users who merely wish to use Finesse, not develop it, do not need to install them and can instead rely on the various pre-built packages that are available.

An optional system runtime dependency is GraphViz, used to visualise graphs of interferometer networks. If you don’t plan to use such features, you don’t need to install GraphViz.

Note

If used, the Conda development environment installs GraphViz.

>>>>>>> feature/m1-ci

Finesse also depends on various Python packages to be able to build and run, as discussed in the next section.

Python package requirements for Finesse

Source code: setup.py, setup.cfg, pyproject.toml, environment.yml, environment-win.yml

As a short summary, runtime requirements should typically be listed in setup.cfg, environment.yml, and environment-win.yml, whereas build requirements should be listed in pyproject.toml, environment.yml, and environment-win.yml.

The following section describes the logic behind this redundancy.

Defining dependencies for Finesse

Finesse’s Python requirements are defined for maximum convenience to users and developers using a hybrid approach: they are listed using standards defined by PEP 517 and in Conda environment files.

In theory, with the PEP 517 configuration in setup.cfg and pyproject.toml, a user or developer can simply run pip install finesse or pip install . to build and install Finesse either from PyPI or from the local directory, respectively. This is the case for most Linux and OSX systems on x86 architecture. Due to a lack of availability of dependencies on PyPI for some systems (such as for AMD64 architecture), environment files for Conda (using the conda-forge channel which provides the required PyPI packages across a wider variety of systems) are provided. When used, Conda takes over the responsibility for providing the required runtime and build dependencies for Finesse, allowing users to use the program on a wider variety of systems.

Note

Nomenclature

Conda is the package manager, conda-forge is the name of the community-driven package channel that is used to install the dependencies for Finesse.

In practice, the hybrid approach means that any changes to Finesse’s runtime or build dependencies require changes to both setup.cfg or pyproject.toml (PEP 517 runtime and build dependencies, respectively), and environment.yml and environment-win.yml (Conda runtime and build dependencies on Linux/OSX and Windows, respectively).

PEP 517 builds

PEP 517 requires a Python function available on the current path to act as a build backend (responsible for compiling the package and adding it to the path). This is currently set in Finesse to setuptools.build_meta, though this can be changed later if a tool offering more useful features becomes available.

The setuptools.build_meta backend requires runtime requirements, i.e. those required to be installed in the environment for Finesse to run, to be listed in setup.cfg and/or setup.py files in the project root. It’s not strictly necessary to use both files with setuptools (indeed it’s now recommended to use only the declarative setup.cfg where possible), but because Finesse needs Cython extensions with relatively complex configurations to be built by the backend it uses a setup.py to define the required logic. It may be possible to remove setup.py later (see #367).

Build requirements, on the other hand, are defined in pyproject.toml.

Note

PEP 517 recommends (but does not strictly require) that build frontends by default set up an isolated environment in which the backend can build the project. This means that build requirements are installed in a temporary location used only for the building of the package, then deleted when the newly built package is moved to the user’s environment and the temporary environment is removed. This behaviour is usually beneficial because it helps to catch missing dependencies and removes the need for build requirements (e.g. Cython) to be available in the local environment, but when debugging build issues it can be useful to switch this behaviour off. In pip this can be done by adding the --no-build-isolation flag to pip install. Note that you’ll need to ensure the inplacebuild extras listed in pyproject.toml are installed.

When Finesse is to be built, by default pip will create a temporary, isolated environment with only the standard library available, then install the build dependencies listed in pyproject.toml using PyPI, before building using the specified build backend. This isolation is intentional to ensure that the intended build dependencies are the (only) ones used to build the package. Build isolation is useful to determine whether an undeclared (Python) dependency is is involved in the build, which could lead to trouble installing the package on other machines.

When Finesse is to be run, it simply uses the Python packages available on the current path. If the user installed Finesse using pip install finesse or pip install . these packages will have been installed from PyPI and added to the path.

Conda builds

Conda provides packages for a wide variety of systems, and, unlike pip, takes care of installing system dependencies on behalf of the user. For example, Conda can install the Finesse system requirement SuiteSparse without the user having to do it themselves.

Requirements for Finesse within a Conda environment are defined in environment.yml and environment-win.yml.

When building Finesse within a Conda environment, it is not necessary and often not desirable to use pip’s PEP 517 isolated build system. As described above, part of the PEP 517 default build process is to install build requirements using PyPI, but these requirements are not always available from PyPI for the target platform. It is therefore typically desirable to build Finesse on such platforms in-tree using pip install . --no-build-isolation, which tells pip to use the build dependencies from the existing (Conda) environment. It’s for this reason that build dependencies, not just runtime dependencies, are also defined in environment.yml and environment-win.yml.

Warning

Dependencies in the Conda environment files refer to their name in the conda-forge channel, not their name on PyPI. Most Conda packages use the same name as their PyPI counterparts, but some do not. For example, interchanging of hyphens and underscores is allowed by pip but not by Conda. A good mapping of non-standard conda-forge package names can be found as part of the grayskull package here.

The extra dependencies for development (such as sphinx for building the documentation) are also defined in environment.yml and environment-win.yml, and therefore get installed when the user sets up their development environment.

Version pinning

The versions of the Finesse runtime requirements are set by a compromise between the desire to use the latest (and usually greatest) versions and the need to help package managers include Finesse alongside other tools with their own, possibly conflicting, requirements. It is therefore best to set only minimum required versions in setup.cfg, pyproject.toml, environment.yml, and environment-win.yml, leaving the maximum version open unless there are known incompatibilities.

We rely on the regular running of the tests in the continuous integration pipeline to catch issues created by the latest versions of requirements. In cases where a particular dependency version is identified to cause problems with Finesse, it can be forbidden in the version string for the respective dependency in the relevant file (see next section). It is also useful to leave a note next to the requirement to inform others (including package maintainers, who may not be Finesse developers) why this is the case.

Modifying requirements

Modification of requirements involves editing typically two or more files:

  • Runtime requirements are listed within the install_requires key of the [options] section of setup.cfg, and environment.yml / environment-win.yml.

  • Build requirements are defined in the requires key within the [build-system] section of pyproject.toml, and environment.yml / environment-win.yml.

  • Extra requirements are defined in the [options.extras_require] section of setup.cfg and can be used to specify optional requirements for users or developers. These are not installed by most frontends by default. The extras_require section contains keys and values representing groups of requirements. For example, the key docs lists requirements for building the documentation. Extras are not currently defined in environment.yml / environment-win.yml as they cause issues solving the dependency tree. These are instead installable via pip from within the active Conda environment using make develop-conda.

PEP 508 specifies the format for requirements strings.

Warning

As described in a previous section, the building of Finesse usually takes place in an isolated environment, so most build requirements these can be pinned in pyproject.toml to whatever version is desired without causing trouble elsewhere. It is important, however, that at least the numpy and cython requirements are set to the same versions in pyproject.toml and setup.cfg because parts of the Finesse code are compiled and linked against the numpy C ABI. Using different versions of numpy or cython for building and running Finesse frequently results in numerous warnings issued by numpy, or even errors. It’s also very useful from the point of view of debugging to maintain binary compatibility between Finesse built in an isolated environment (i.e. via the full build process) and in-place (e.g. via make).

After modifying requirements, verify that the tests still pass and build process still operates as expected. Also consider making the changes in a branch, then push to GitLab to allow the continuous integration pipeline to build and run using the updated dependencies too, before merging into the develop branch.