Proper Python setup with pyenv & Poetry

More than often, I see people struggling with a local (development) setup with Python. I have to admit, it isn’t easy. But at the same time, it’s crucial for stability and the ability to debug problems.

This article will be yet another article to explain how to properly setup a local Python environment. In our case, with pyenv and Poetry. These links already contain all information needed, but may seem daunting since there’s just too much information. In this article, our sole focus is to have a local development setup for the average Python application.

Photo by David Clode on Unsplash

pyenv

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

pyenv installation

The first step is to install pyenv. Since this is specific to different operating systems, please follow the instructions in the docs: https://github.com/pyenv/pyenv?tab=readme-ov-file#installation

When you’re on a Linux distribution, run the following:

curl https://pyenv.run | bash

pyenv will be installed at ~/.pyenv .

It will prompt with some additional steps to add pyenv to the load path. Also execute these steps. In my case, since I’m using ZSH, I need to add these lines to ~/.zshrc . Installing ZSH is not part of this article, feel free to reach out if you need help with this too.

Now re-open the terminal.

Test pyenv by running pyenv in the new terminal. If you see a list of command you’re good to go.

Python installation

The first thing we want to do is check which Python version is default active in our terminal. In my examples, I’m working on a Raspberry Pi 5 (8GB) running Raspberry Pi OS 12.4 (Bookworm).

[~]$ which python
/usr/bin/python

This is the standard Python that comes with the OS.

Next we check the Python versions in pyenv:

[~]$ pyenv versions
* system (set by /home/vandermeij/.pyenv/version)

It tells us it’s empty, so no Python versions have been installed. Which is good for now.

To be able to install any Python versions via pyenv, we need to install some dependencies. Head over to the docs to install these for your respective OS: https://github.com/pyenv/pyenv/wiki#suggested-build-environment

In our case:

sudo apt update; sudo apt install build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Now we can install Python, e.g., version 3.11:

[~]$ pyenv install 3.11
Downloading Python-3.11.7.tar.xz...
-> https://www.python.org/ftp/python/3.11.7/Python-3.11.7.tar.xz
Installing Python-3.11.7...

Next we check the Python versions in pyenv again:

[~]$ pyenv versions
* system (set by /home/vandermeij/.pyenv/version)
  3.11.7

We now see version 3.11.7 is installed.

Let’s activate it and check the versions again:

[~]$ pyenv global 3.11.7
[~]$ pyenv versions
  system
* 3.11.7 (set by /home/vandermeij/.pyenv/version)

We now see Python 3.11.7 is the active Python version and this is global, system-wide.

Check again the default Python version:

[~]$ which python
/home/vandermeij/.pyenv/shims/python

Notice it’s different from /usr/bin/python . It now points to a symlink maintained by pyenv. This is good! It now points to the selected Python version as can be seen with penv versions .

When you change it back to system Python via pyenv, this will only be reflected in pyenv versions as pyenv is now the maintainer of Python versions on your OS.

[~]$ pyenv global system
[~]$ pyenv versions
* system (set by /home/vandermeij/.pyenv/version)
  3.11.7
[~]$ which python
/home/vandermeij/.pyenv/shims/python

Usage

I like to have pyenv as the main Python distribution provider. I never use the default system Python. On my machine you see this:

[~]$ pyenv versions
  system
* 3.11.7 (set by /home/vandermeij/.pyenv/version)

Sometimes, you want to use a different Python version in your project, let’s say Python 3.12. Apart from changing the global Python version, pyenv can also use a local version.

Let’s say we create a little project called testapp .

[~]$ mkdir testapp
[~]$ cd testapp

Next we tell pyenv to use a local Python version:

[~/testapp]$ pyenv local 3.12
pyenv: version `3.12' not installed

It tells us version 3.12 is not installed (yet).

So we will install it:

[~/testapp]$ pyenv install 3.12    
Downloading Python-3.12.1.tar.xz...
-> https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tar.xz
Installing Python-3.12.1...

Try again:

[~/testapp]$ pyenv local 3.12
[~/testapp]$ pyenv versions
  system
  3.11.7
* 3.12.1 (set by /home/vandermeij/testapp/.python-version)

As you can see, Python 3.12.1 is active. This is due to a local file .python-version in this folder, as shown in the terminal, telling pyenv to load the specific version.

When you go to another folder, except for sub-folders, the active Python version is reset:

[~]$ pyenv versions
  system
* 3.11.7 (set by /home/vandermeij/.pyenv/version)
  3.12.1

With pyenv, with either local or global Python versions, you always know which Python version is in use and you have full control over it.

The next step proper development setup endeavours, is to use virtual environments for installing external libraries/dependencies, so they a) don’t litter our pyenv Python distributions and b) don’t conflict with other projects. For this I recommend Poetry.

Poetry

Poetry comes with all the tools you might need to manage your projects in a deterministic way.

Installation

The one and only external library you need in your newly installed pyenv Python versions is Poetry.

First install pipx (copied from https://pipx.pypa.io/stable/installation/, according to https://python-poetry.org/docs/).

[~]$ sudo apt update
[~]$ sudo apt install pipx
[~]$ pipx ensurepath
[~]$ sudo pipx ensurepath --global # optional to allow pipx actions in global scope. See "Global installation" section below.

Re-open the terminal again. Next install Poetry with pipx.

[~]$ pipx install poetry
  installed package poetry 1.8.3, installed using Python 3.11.2
  These apps are now globally available
    - poetry
[~]$ poetry -V
Poetry (version 1.8.3)

Note that pipx is installed using its own Python distribution.

Now you can use Poetry in any installed Python version. Also new versions you will install in the future.

[~/testapp]$ poetry -V 
Poetry (version 1.8.3)

Usage

For a new projects, start by running poetry init -q. This will create a pyproject.toml file.

Next you will need to set the correct Python version.

So in full:

[~/testapp]$ poetry init -q
[~/testapp]$ poetry env use $(pyenv which python)

If you don’t run the above, the Poetry environment will be bound to the Python version it used to install Poetry itself (using pipx).

If you want to play around yourself with a pyenv & Poetry sandbox, feel free to checkout my Github repository for an example Dockerfile. The Dockerfile is prepared to run all steps above, up until here.

Now you can add your dependencies. But before we start doing that, we will activate the virtual environment. Run poetry shell:

[~/testapp]$ poetry shell
Creating virtualenv testapp-b6lGsqOc-py3.12 in /home/vandermeij/.cache/pypoetry/virtualenvs
Spawning shell within /home/vandermeij/.cache/pypoetry/virtualenvs/testapp-b6lGsqOc-py3.12
[~/testapp]$ emulate bash -c '. /home/vandermeij/.cache/pypoetry/virtualenvs/testapp-b6lGsqOc-py3.12/bin/activate'
(testapp-py3.12) [~/testapp]$

This will create the virtual environment (virtualenv), if not already existing, and then spawn a new shell with that virtual environment activated. This can be seen in the prompt (PS1): (testapp-py3.12) [~/testapp]$.

If you now run which python, you will see another symlink, linking to the virtual environment:

(testapp-py3.12) [~/testapp]$ which python
/home/vandermeij/.cache/pypoetry/virtualenvs/testapp-b6lGsqOc-py3.12/bin/python

Notice it’s still the same Python version as activated by pyenv, but now wrapped in a virtual environment.

(testapp-py3.12) [~/testapp]$ python -V
Python 3.12.1

In this virtual environment, we will install our dependencies.

When you have an existing pyproject.toml file, because you just cloned a git repository containing one, you can run poetry install or poetry install — no-root , depending on whether your project itself is a Python library. In our case, with testapp, this isn’t a Python library:

(testapp-py3.12) [~/testapp]$ poetry install --no-root
Installing dependencies from lock file

You can now add new dependencies, e.g., FastAPI, using poetry add fastapi or edit the pyproject.toml file manually. I usually do the latter.

Let’s say we’ll add FastAPI manually. Open pyproject.toml in your favourite editor and change the following lines (add fastapi = “*”):

[tool.poetry.dependencies]
python = "^3.12"
fastapi = "*"

Now run poetry update:

(testapp-py3.12) [~/testapp]$ poetry update
Updating dependencies
Resolving dependencies... (6.6s)

Package operations: 9 installs, 0 updates, 0 removals

  • Installing idna (3.6)
  • Installing sniffio (1.3.0)
  • Installing typing-extensions (4.9.0)
  • Installing annotated-types (0.6.0)
  • Installing anyio (4.2.0)
  • Installing pydantic-core (2.14.6)
  • Installing pydantic (2.5.3)
  • Installing starlette (0.35.1)
  • Installing fastapi (0.109.0)

Writing lock file

Your virtual environment is now enriched with FastAPI and all its dependencies. As a bonus, Poetry created a poetry.lock file, containing the specific versions of each of the (sub)dependencies that are installed.

Add these files to git:

• .python-version
• pyproject.toml
• poetry.lock

Now you have a clean, proper Python development setup with pyenv and Poetry.

For a full overview of all virtual environments, run poetry env list:

(testapp-py3.12) [~/testapp]$ poetry env list
testapp-b6lGsqOc-py3.12 (Activated)

To exit the dedicated shell press CTRL+D or type exit :

(testapp-py3.12) [~/testapp]$ exit
[~/testapp]$

Conclusion

pyenv and Poetry provide powerful tools for your local development setup. I recommend using them as I’m very happy with doing so.

As mentioned briefly above, please use git to store the essential files. Furthermore, use the poetry.lock file everywhere you want to install the application, e.g., in Docker.

As mentioned above as well, if you want to play around yourself with a pyenv & Poetry sandbox, feel free to checkout my Github repository for an example Dockerfile.

Feel free to reach out if you have any questions, inquiries or remarks regarding this article. I’m happy to help. You can reach out to me here or on my personal website.

If you would like personal assistance with setting it up, join our free course.