Deep Learning
Managing Python Dependencies with Poetry vs Conda & Pip
March 7, 2024
9 min read
Managing Dependencies for Your Python Projects
Effectively managing dependencies, including libraries, functions, and packages crucial for project functionality, is facilitated by utilizing a package manager. Pip, a widely adopted classic, serves as a go-to for many developers, enabling the seamless installation of Python packages from the Python Package Index (PyPI). Conda, recognized not only as a package manager but also as an environment manager, extends its capabilities to handle both Python and non-Python dependencies, making it a versatile tool. For our purposes, we will focus on using it mainly for Python-only environments.
Pip and Conda stand out as reliable tools, extensively used and trusted by the developer community. However, as projects expand, maintaining organization amid a growing number of dependencies becomes a challenge. In this context, Poetry emerges as a modern and organized solution for dependency management.
Poetry, built on top of Pip, introduces a contemporary approach to managing dependencies. It extends beyond being a simple fusion of Pip and a virtual environment serving as a comprehensive tool that encompasses dependency management, project packaging, and build processes. The comparison to Conda is nuanced; Poetry aims to simplify the packaging and distribution of Python projects, offering a distinct set of features.
Pip and Conda remain valuable choices for managing dependencies, with Conda's versatility in handling diverse dependencies. Poetry, on the other hand, provides a modernized and comprehensive solution, offering simplicity in managing Python projects and their dependencies. Choosing the appropriate tool depends on the specific requirements of the project and the preferences of the developer.
Package Management
Poetry uses a pyproject.toml file to specify the configuration for your project accompanied by an automatically generated lockfile. The pyproject.toml file looks like this:
[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.5"
numpy = "^1.24.3"
[tool.poetry.dev.dependencies]
pytest = "^7.3.2"
precomit = "^3.3.3"
Like other dependency managers, Poetry diligently keeps track of package versions in the current environment through a lockfile. This lockfile contains project metadata, package version parameters, and more, ensuring consistency across different environments. Developers can intelligently separate dependencies into dev-based and prod-based categories within the toml files, streamlining deployment environments and reducing the risk of conflicts, especially on different operating systems.
Poetry's pyproject.toml file is designed to address certain limitations found in Pip's requirement.txt and Conda's environment.yaml files. Unlike Pip and Conda, which often produce lengthy dependency lists without metadata in a separate file, Poetry aims for a more organized and concise representation.
While it's true that Pip and Conda, by default, lack a lock feature, it's important to note that recent versions offer options for generating lockfiles via installed libraries like pip-tools and conda-lock. This functionality ensures that different users can install the intended library versions specified in the requirements.txt file, promoting reproducibility.
Poetry emerges as a modern and organized solution for Python dependency management, offering improved organization, version control, and flexibility compared to traditional tools like Pip and Conda.
Updating, Installing, and Removing Dependencies
With Poetry, updating libraries is simple and accounts for other dependencies to ensure they are up to date accordingly. Poetry has a mass update command that will update your dependencies (according to your toml file) while keeping all dependencies still compatible with one another and maintaining package version parameters inside found in the lock file. This will simultaneously update your lock file.
As for installation, it could not get any simpler. To install dependencies with Poetry you can use the poetry add function that you can either specify the version, use logic to specify version parameters (greater than less than), or use flags like @latest which will install the most recenter version of the package from PyPI. You can even group multiple packages in the same add function. Any newly installed package is automatically resolved to maintain the correct dependencies.
$poetry add requests pandas@latest
As for the classic dependency managers, let's test to see what happens when we try to install an older incompatible version. Pip installed packages will output errors and conflicts but will ultimately still install the package which can lead to development that is not ideal. Conda does have a solver for errors in compatibility and will notify the user, but immediately goes into search mode to solve the compatibility issue outputting a secondary error when it cannot find a solution.
(test-env) user:~$ pip install "numpy<1.18.5"
Collecting numpy<1.18.5
Downloading numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
|████████████████████████████████| 20.7 MB 10.9 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Uninstalling numpy-1.22.3:
Successfully uninstalled numpy-1.22.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas 1.4.2 requires numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you have numpy 1.18.4 which is incompatible.
Successfully installed numpy-1.18.4
(test-env) user:~$ pip list
Package Version
--------------- -------
numpy 1.18.4
pandas 1.4.2
pip 21.1.1
python-dateutil 2.8.2
pytz 2022.1
six 1.16.0
Poetry has an immediate response to dependency compatibility errors for fast and early notice of conflicts. It refuses to continue the installation, so the user is now in charge of either finding a different version of the new package or existing package. We feel that this allows more control as opposed to Conda’s immediate action.
user:~$ poetry add "numpy<1.18.5"
Updating dependencies
Resolving dependencies... (53.1s)
SolverProblemError
Because pandas (1.4.2) depends on numpy (>=1.18.5)
and no versions of pandas match >1.4.2,<2.0.0, pandas (>=1.4.2,<2.0.0) requires numpy (>=1.18.5).
So, because dependency-manager-test depends on both pandas (^1.4.2) and numpy (<1.18.5), version solving failed.
...
user:~$ poetry show
numpy 1.22.3 NumPy is the fundamental package for array computing with Python.
pandas 1.4.2 Powerful data structures for data analysis, time series, and statistics
python-dateutil 2.8.2 Extensions to the standard Python datetime module
pytz 2022.1 World timezone definitions, modern and historical
six 1.16.0 Python 2 and 3 compatibility utilities
Last but not least is Poetry’s uninstallation of packages. Some packages require more dependencies that are installed. For Pip, its removal of a package will only uninstall the defined package and nothing else. Conda will remove some packages but not all dependencies. Poetry on the other hand will remove the package and all its dependencies to keep your list of dependencies clutter free.
Is Poetry Compatible with Existing Pip or Conda Projects?
Yes, Poetry is compatible with existing projects managed by Pip or Conda. Just initialize your code using Poetry's Poetry.toml format and run it to grab the library of packages and its dependencies, allowing for a seamless transition.
If you have an existing project that uses Pip or Conda, you can migrate it to Poetry without much difficulty. Poetry uses its own pyproject.toml file to manage project dependencies and settings. To start using Poetry in your project, you can follow these steps:
1. Install Poetry either by curling and piping or using Pip
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
2. Navigate to the root directory of your existing project.
3. Initialize Poetry in your project directory:
poetry init
This command will guide you through a series of prompts to set up the initial configuration for your project.
4. Once the initialization is complete, Poetry will create the pyproject.toml in your project director. Open the toml file to add or modify your project's dependencies
5. To install the existing dependencies in your project to
poetry install
This will create a virtual environment and install the project dependencies within it.
6. You can now use the Poetry run command to execute your project's scripts, similar to how you would use Python or Conda commands.
poetry run python my_script.py
Poetry manages the virtual environment and dependency resolution for your project, making it compatible with existing Pip or Conda projects. It simplifies the management of dependencies and allows for consistent package installations across different environments.
Note: It's always a good practice to back up your project before making any significant changes to its configuration or dependency management tools.
Closing Thoughts
Making sure the right versions of the packages are in your code environment is essential for getting the right results every time. Slight changes to the backend of your code can alter the outcome. But also, keeping those packages and libraries up to date is just as important, to leverage the innovations each patch provides the next.
To manage these dependencies in your code, Poetry is a great tool for those working with more complex and diverse projects with a higher number of dependencies. While Pip and Conda are still viable options, they are more suited for smaller environments that are less complex. Not everyone might use Poetry, but since Pip has been around forever, it may be worth the ease of use to just use Pip.
But if your project and your workload value the importance of organization and are willing to explore new tools to improve your process, Poetry is a tool you should consider. The extended functionality from Pip to Poetry really makes a difference. We encourage you to try Poetry out for yourself.
Data Science and Deep Learning are making waves in the tech industry with new AI tools popping up left and right. Develop, train, and deploy your own AI models accelerated by an Exxact Deep Learning GPU solution, and boost your computing to achieve more.
Have any questions?
Contact Us Today!
Deep Learning
Managing Python Dependencies with Poetry vs Conda & Pip
March 7, 20249 min read
Managing Dependencies for Your Python Projects
Effectively managing dependencies, including libraries, functions, and packages crucial for project functionality, is facilitated by utilizing a package manager. Pip, a widely adopted classic, serves as a go-to for many developers, enabling the seamless installation of Python packages from the Python Package Index (PyPI). Conda, recognized not only as a package manager but also as an environment manager, extends its capabilities to handle both Python and non-Python dependencies, making it a versatile tool. For our purposes, we will focus on using it mainly for Python-only environments.
Pip and Conda stand out as reliable tools, extensively used and trusted by the developer community. However, as projects expand, maintaining organization amid a growing number of dependencies becomes a challenge. In this context, Poetry emerges as a modern and organized solution for dependency management.
Poetry, built on top of Pip, introduces a contemporary approach to managing dependencies. It extends beyond being a simple fusion of Pip and a virtual environment serving as a comprehensive tool that encompasses dependency management, project packaging, and build processes. The comparison to Conda is nuanced; Poetry aims to simplify the packaging and distribution of Python projects, offering a distinct set of features.
Pip and Conda remain valuable choices for managing dependencies, with Conda's versatility in handling diverse dependencies. Poetry, on the other hand, provides a modernized and comprehensive solution, offering simplicity in managing Python projects and their dependencies. Choosing the appropriate tool depends on the specific requirements of the project and the preferences of the developer.
Package Management
Poetry uses a pyproject.toml file to specify the configuration for your project accompanied by an automatically generated lockfile. The pyproject.toml file looks like this:
[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.5"
numpy = "^1.24.3"
[tool.poetry.dev.dependencies]
pytest = "^7.3.2"
precomit = "^3.3.3"
Like other dependency managers, Poetry diligently keeps track of package versions in the current environment through a lockfile. This lockfile contains project metadata, package version parameters, and more, ensuring consistency across different environments. Developers can intelligently separate dependencies into dev-based and prod-based categories within the toml files, streamlining deployment environments and reducing the risk of conflicts, especially on different operating systems.
Poetry's pyproject.toml file is designed to address certain limitations found in Pip's requirement.txt and Conda's environment.yaml files. Unlike Pip and Conda, which often produce lengthy dependency lists without metadata in a separate file, Poetry aims for a more organized and concise representation.
While it's true that Pip and Conda, by default, lack a lock feature, it's important to note that recent versions offer options for generating lockfiles via installed libraries like pip-tools and conda-lock. This functionality ensures that different users can install the intended library versions specified in the requirements.txt file, promoting reproducibility.
Poetry emerges as a modern and organized solution for Python dependency management, offering improved organization, version control, and flexibility compared to traditional tools like Pip and Conda.
Updating, Installing, and Removing Dependencies
With Poetry, updating libraries is simple and accounts for other dependencies to ensure they are up to date accordingly. Poetry has a mass update command that will update your dependencies (according to your toml file) while keeping all dependencies still compatible with one another and maintaining package version parameters inside found in the lock file. This will simultaneously update your lock file.
As for installation, it could not get any simpler. To install dependencies with Poetry you can use the poetry add function that you can either specify the version, use logic to specify version parameters (greater than less than), or use flags like @latest which will install the most recenter version of the package from PyPI. You can even group multiple packages in the same add function. Any newly installed package is automatically resolved to maintain the correct dependencies.
$poetry add requests pandas@latest
As for the classic dependency managers, let's test to see what happens when we try to install an older incompatible version. Pip installed packages will output errors and conflicts but will ultimately still install the package which can lead to development that is not ideal. Conda does have a solver for errors in compatibility and will notify the user, but immediately goes into search mode to solve the compatibility issue outputting a secondary error when it cannot find a solution.
(test-env) user:~$ pip install "numpy<1.18.5"
Collecting numpy<1.18.5
Downloading numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
|████████████████████████████████| 20.7 MB 10.9 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Uninstalling numpy-1.22.3:
Successfully uninstalled numpy-1.22.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas 1.4.2 requires numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you have numpy 1.18.4 which is incompatible.
Successfully installed numpy-1.18.4
(test-env) user:~$ pip list
Package Version
--------------- -------
numpy 1.18.4
pandas 1.4.2
pip 21.1.1
python-dateutil 2.8.2
pytz 2022.1
six 1.16.0
Poetry has an immediate response to dependency compatibility errors for fast and early notice of conflicts. It refuses to continue the installation, so the user is now in charge of either finding a different version of the new package or existing package. We feel that this allows more control as opposed to Conda’s immediate action.
user:~$ poetry add "numpy<1.18.5"
Updating dependencies
Resolving dependencies... (53.1s)
SolverProblemError
Because pandas (1.4.2) depends on numpy (>=1.18.5)
and no versions of pandas match >1.4.2,<2.0.0, pandas (>=1.4.2,<2.0.0) requires numpy (>=1.18.5).
So, because dependency-manager-test depends on both pandas (^1.4.2) and numpy (<1.18.5), version solving failed.
...
user:~$ poetry show
numpy 1.22.3 NumPy is the fundamental package for array computing with Python.
pandas 1.4.2 Powerful data structures for data analysis, time series, and statistics
python-dateutil 2.8.2 Extensions to the standard Python datetime module
pytz 2022.1 World timezone definitions, modern and historical
six 1.16.0 Python 2 and 3 compatibility utilities
Last but not least is Poetry’s uninstallation of packages. Some packages require more dependencies that are installed. For Pip, its removal of a package will only uninstall the defined package and nothing else. Conda will remove some packages but not all dependencies. Poetry on the other hand will remove the package and all its dependencies to keep your list of dependencies clutter free.
Is Poetry Compatible with Existing Pip or Conda Projects?
Yes, Poetry is compatible with existing projects managed by Pip or Conda. Just initialize your code using Poetry's Poetry.toml format and run it to grab the library of packages and its dependencies, allowing for a seamless transition.
If you have an existing project that uses Pip or Conda, you can migrate it to Poetry without much difficulty. Poetry uses its own pyproject.toml file to manage project dependencies and settings. To start using Poetry in your project, you can follow these steps:
1. Install Poetry either by curling and piping or using Pip
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
2. Navigate to the root directory of your existing project.
3. Initialize Poetry in your project directory:
poetry init
This command will guide you through a series of prompts to set up the initial configuration for your project.
4. Once the initialization is complete, Poetry will create the pyproject.toml in your project director. Open the toml file to add or modify your project's dependencies
5. To install the existing dependencies in your project to
poetry install
This will create a virtual environment and install the project dependencies within it.
6. You can now use the Poetry run command to execute your project's scripts, similar to how you would use Python or Conda commands.
poetry run python my_script.py
Poetry manages the virtual environment and dependency resolution for your project, making it compatible with existing Pip or Conda projects. It simplifies the management of dependencies and allows for consistent package installations across different environments.
Note: It's always a good practice to back up your project before making any significant changes to its configuration or dependency management tools.
Closing Thoughts
Making sure the right versions of the packages are in your code environment is essential for getting the right results every time. Slight changes to the backend of your code can alter the outcome. But also, keeping those packages and libraries up to date is just as important, to leverage the innovations each patch provides the next.
To manage these dependencies in your code, Poetry is a great tool for those working with more complex and diverse projects with a higher number of dependencies. While Pip and Conda are still viable options, they are more suited for smaller environments that are less complex. Not everyone might use Poetry, but since Pip has been around forever, it may be worth the ease of use to just use Pip.
But if your project and your workload value the importance of organization and are willing to explore new tools to improve your process, Poetry is a tool you should consider. The extended functionality from Pip to Poetry really makes a difference. We encourage you to try Poetry out for yourself.
Data Science and Deep Learning are making waves in the tech industry with new AI tools popping up left and right. Develop, train, and deploy your own AI models accelerated by an Exxact Deep Learning GPU solution, and boost your computing to achieve more.
Have any questions?
Contact Us Today!
