The ecosystem of tools for packaging, testing, distributing, and testing Python projects is overwhelming, especially if you’re coming to Python from other languages. In this post, we’ll survey the evolution of Python tooling from its inception in 1991 through the present day, touching on significant milestones. We’ll explain each tool’s benefits, the problems it addresses, and any configuration files you’ll find associated with it in Python projects. After reading this post, you’ll have a richer understanding of the many setup, testing, and packaging patterns you’re likely to find in open-source Python projects and be better equipped to work with them. You’ll see how to migrate your project to a modern dependency management and packaging tool such as Poetry. We’ll explain why we are converting the Pinecone Python client to Poetry, the new contributor-facing guides we’ve added to ease this migration, and how this makes the project easier to develop locally and even within Jupyter Notebooks and alongside popular libraries such as LangChain. Along the way, we’ll take an unflinching look at some of the pain involved in developing Python projects to set the stage for understanding how the latest tooling can make developers’ lives easier. Abandon all hope ye who enter dependency hell. Guido van Rossum released Python on February 20, 1991. At the time of this writing, that was 32 years ago. To understand why today’s preferred tools work the way they do, we must first look backward to understand the evolution of Python. If you’ve ever worked with Python, you’re probably familiar with the following tools, patterns, digital duct tape, and the headaches that come with a scattered landscape of options. Most readers familiar with Python have probably run pip install -r requirements.txt to tell the pip package manager binary to read a requirements text file, which lists the dependencies (external libraries) required by a given Python script. pip install is probably the most common and straightforward way to get up and running with a given Python program. Now, consider why this typical pattern is fraught with complexity and suffering. Until macOS Catalina (10.15), released in 2019, Python 2 was pre-installed on your OSX machine, requiring developers who needed Python 3 to install it separately. Folks in a hurry could accidentally execute commands against python instead of python3, leading to errors. The two versions also had dependency conflicts because each maintained a separate site-packages, the directory where Python installs packages, leading to potential duplicate packages and version incompatibility on the same system. The shebang lines at the top of Python scripts that signal which interpreter were equally confusing. #!/usr/bin/env python defaulted to Python 2, which could be problematic for code intended to be run by Python 3. This confusion spread to the use of pip itself. You might have meant to pip3 install a given requirements.txt file because pip3 is not the same binary as pip. If you have Python 2 and Python 3 installed on your system, pip and pip3 will install packages in different locations to avoid conflicts. However, if you only have Python 3 installed, pip might be an alias for pip3; in that case, they
