Part 2 — Installing and managing tools

pixi, and the conda/mamba world you will meet anyway

Part 1 assumed that the tools were already installed. This part shows how to install them, and explains why this is harder than it sounds.

NoteMost commands on this page are not executed

The examples on this page install software, unlike those in Part 1. They need the internet, they take minutes, and several of them use an installer that you do not have yet. So they are shown, but not run automatically. The version numbers are only examples.

The problem

You install a tool, and it works. Six months later you install a second tool for another project, and the first tool stops working.

This is not bad luck. Tools are built on other tools: libraries, interpreters, system packages. Two programs can need different versions of the same dependency, and these versions may not work together. If you install one version for the whole computer, it replaces the version that the other program needed.

A package manager must do three things to solve this:

  1. Install software without you compiling anything.
  2. Resolve dependencies: find a set of versions that fits every requirement at once, or say clearly that no such set exists.
  3. Keep projects separate, so that the tools of one project cannot break the tools of another.

An environment (dt. Umgebung) does the third job. It is a separate set of tools that belongs to one project. Two environments on the same computer can contain different versions of the same program, and they do not affect each other.

ImportantThe other reason environments matter

An environment can be written down. If a file records the exact versions that produced a result, someone else can recreate the environment and get the same result. That someone can also be you, next year.

Without such a file, you cannot find out why a result differs on another computer. This is the strongest reason to use environments at all, and it is why the file is more important than the tool.

pixi

This material recommends pixi. pixi is a package manager that uses the same package repositories as the older tools, so it can install the same software. But it is organised around the project, not around your computer.

Installing pixi

curl -fsSL https://pixi.sh/install.sh | bash

Close the terminal, open a new one, and check that pixi is installed:

pixi --version
WarningAbout curl … | bash

You have just downloaded a script and run it without reading it. Be aware of this. Many installers work this way, but a harmful script can also get onto your computer this way. So do this only for projects that you trust, and use the official website, not a link that someone sent you.

Starting a project

pixi init myproject
cd myproject

This creates a directory with a file pixi.toml in it. Now add a tool:

pixi add python

Run a command inside the environment of the project:

pixi run python --version

The important point: you did not “activate” anything. pixi run uses the tools of this project because you are in the directory of this project. If you move to another project, the same command uses the tools of that project.

The two files

file what it is commit it?
pixi.toml what you asked for — “python, and samtools” yes
pixi.lock what you actually got — every package, exact version, checksum yes

The lock file makes the environment reproducible. pixi.toml says “a recent python”. pixi.lock says exactly which version, for every platform that the project supports. If someone clones your project and runs:

pixi install

they get exactly what you had, not only something similar. Commit both files. If you delete the lock file to “fix” a problem, you delete the only record of the versions that worked.

Channels

Packages come from channels: repositories of software that is already built. Two channels are important here:

  • conda-forge — general software: Python, R, compilers, libraries.
  • bioconda — tools for the life sciences.

Each project lists its channels in its own pixi.toml. So the project does not depend on how your computer is configured:

[workspace]
channels = ["conda-forge", "bioconda"]
platforms = ["linux-64"]
Warningbioconda does not build for Windows

Most life-science tools have no Windows packages. This will not change, and it is why Part 1 tells Windows users to install WSL2. Inside WSL2 you are on linux-64, and everything works.

macOS is supported, but not for every tool. Some tools have no version for Apple Silicon, and an environment that installs on Linux may not install on a Mac.

conda and mamba

You will see these tools. Most documentation, most methods sections in papers and most older lab protocols are written for conda. So you should know what conda and mamba are, even though this material does not recommend them for new work.

conda the original tool. Same channels, same packages. Resolving dependencies could be extremely slow: minutes to hours for a large environment.
mamba a much faster new version of the part of conda that resolves dependencies. For a long time, the usual advice was “install conda, then use mamba instead”.
pixi organised around the project, and it always writes a lock file. Again the same packages.

Today, conda is much faster than it was, because it now uses the same method as mamba. So the old advice “conda is too slow to use” is out of date. Speed is not the reason to prefer pixi.

Translating between them

If you find instructions for conda, this table shows the pixi equivalents:

conda / mamba pixi
conda create -n myenv pixi init myproject
conda activate myenv (nothing — be in the directory)
conda install samtools pixi add samtools
conda deactivate (nothing)
conda env export > env.yml (already done — pixi.lock)
conda env create -f env.yml pixi install

Why this material recommends pixi

There are three concrete differences. They are not a matter of taste:

  1. The lock file is always there. conda can write one, but only with extra tools and extra effort. pixi writes one every time, so your environment is reproducible without any extra work.
  2. The environment belongs to the project, not to you. You do not need to manage a list of environments, and you cannot forget which one is active. If you are in the directory, you have the right tools.
  3. You cannot forget to activate anything. The most common problem with conda is to run a command in the wrong environment. Then you get a confusing error, or even worse, a wrong result from an older version.
NoteNone of this makes conda wrong

conda is used everywhere, it works, and you will need to read instructions for it. If you join a group that uses conda, use conda. The concepts are the same in both tools: channels, environments, resolving dependencies, and an environment that is written down in a file.

When it goes wrong

“Solving environment” takes a very long time, then fails. This means that no combination of versions fits everything you asked for. Usually one fixed version is the problem. Allow more versions, or install the difficult tool in a separate project. A second environment costs little; a broken environment costs a lot.

The tool installs but does not run. Often the package was built for another processor type: for example, a package for linux-64 on an Apple Silicon Mac, or the other way round. Check which platforms the project lists.

It works for you but not for a colleague. The lock file exists to prevent this. If your colleague uses the lock file and the results still differ, the difference is outside the environment: the operating system, or a tool that was installed in another way.

TipNever install project tools system-wide

sudo apt install <a scientific tool> installs the tool outside every environment. There, no file records it, and nobody can reproduce it. It is also often an old version. Keep the tools of a project in the project.

What you should be able to do now

Where to go next

Part 3 — Git and GitHub shows how to save, share and recover these two files, and the rest of your work.