Troubleshooting “Externally Managed Environment” Pip Errors on Ubuntu

If you’ve recently encountered the “Externally Managed Environment” error when using pip on Ubuntu, you’re not alone.

This error, often accompanied by a message mentioning “PEP 668,” is a result of changes designed to improve system stability and prevent conflicts between the system package manager (apt) and pip. This article will explain the best ways to work around this issue, along with the reasoning behind these changes.

2. Understanding the Problem: PEP 668 and System Integrity

PEP 668 (“Marking Python base environments as “externally managed””) is the core reason for this error. Modern Linux distributions, including Ubuntu, rely heavily on Python for system tools and utilities. If you use pip to install packages globally (without a virtual environment), you risk overwriting or breaking system-provided Python packages. This can lead to unpredictable behavior and even break your system.

To prevent this, PEP 668 introduces a marker file (EXTERNALLY-MANAGED) that tells pip that the Python environment is managed by the system package manager (apt in Ubuntu’s case). When pip sees this file, it refuses to install packages globally, displaying the “Externally Managed Environment” error.

Read: Mastering Python Virtual Environments: A Comprehensive Guide to venv, pipenv, poetry, and More

3. Solutions and Workarounds

The key takeaway is: You should almost always use virtual environments for your Python projects. This isolates your project’s dependencies and avoids conflicts with the system Python installation.

3.1 Use Virtual Environments (Best Practice)

This is the right way to manage Python projects. Virtual environments create isolated spaces for your projects, each with its own set of packages. This prevents any conflicts with the system Python or other projects.

Create a Virtual Environment:

Use the venv module (recommended for Python 3.3 and later):

python3 -m venv <your_environment_name>

For example:

python3 -m venv my_project_env

This creates a directory named my_project_env containing the virtual environment.

Activate the Virtual Environment:

On Linux and macOS:

source <your_environment_name>/bin/activate

For example:

source my_project_env/bin/activate

Your command prompt will change to show the active environment name (e.g., (my_project_env) user@host:~$).

Install Packages with Pip:

Now, you can use pip install without any issues:

pip install <package_name>

For example:

pip install requests

These packages will be installed only within the active virtual environment.

Deactivate the Virtual Environment:

When you’re finished working on the project, deactivate the environment:

deactivate

Read: How to install pip on Ubuntu 18.04 or Ubuntu 20.04

3.2 Use pipx (For Command-Line Tools)

If you’re trying to install a Python application that provides a command-line tool (like youtube-dl or black), pipx is an excellent solution. pipx installs each application in its own isolated virtual environment, making it available globally without risking system conflicts.

Install pipx:

You might need to install pipx itself using a workaround (see Option 3.4 below) or via apt if it’s available in your distribution’s repositories:

sudo apt install pipx

Or, if not available, you can try the following:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Install Applications with pipx:

pipx install <package_name>

For example:

pipx install youtube-dl

Run the Application:

pipx automatically adds the installed application to your PATH, so you can run it directly:

youtube-dl ...

3.3 User Installs with --user (Limited Use Cases)

The --user flag installs packages into your user’s site-packages directory. This avoids modifying the system-wide Python installation, but it’s still not recommended for general project development. It can lead to conflicts between different user-installed packages, and it doesn’t provide the isolation of virtual environments.

python3 -m pip install --user <package_name>

Make sure ~/.local/bin is in your PATH environment variable. To check this, run:

echo $PATH

If it’s not there, add the following to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc):

export PATH="$HOME/.local/bin:$PATH"

Then, reload your shell configuration:

source ~/.bashrc  # Or ~/.zshrc, etc.

3.4 Removing the EXTERNALLY-MANAGED File (Strongly Discouraged)

This is the least recommended approach. It directly circumvents the protection offered by PEP 668. You should only consider this as a temporary workaround if you absolutely must install a package globally and you understand the risks.

The EXTERNALLY-MANAGED file is usually located in the /usr/lib/python3.*/ directory. You can remove it, but it will likely be recreated by system updates. Here’s how to find and remove it (use with extreme caution):

sudo find /usr/lib/python3* -name EXTERNALLY-MANAGED -print -delete

Warning: Removing this file can destabilize your system. You might break system tools or create conflicts that are difficult to diagnose. If you do this, be prepared to reinstall your system if something goes wrong.

You can also remove it manually. First, locate it:

locate EXTERNALLY-MANAGED

Then remove it:

sudo rm /usr/lib/python3.11/EXTERNALLY-MANAGED # Example.  Use the correct path.

4. Common Pitfalls

• Ignoring Virtual Environments: The biggest mistake is to skip virtual environments altogether. They are essential for proper Python development on modern systems.
• Mixing sudo pip and pip: Never use sudo pip. If you need to install something globally, use your system’s package manager (apt) or pipx.
• Forgetting to Activate/Deactivate Environments: Make sure you activate your virtual environment before installing packages and deactivate it when you’re finished.

5. Verifying the Solution

After setting up a virtual environment, you should be able to use pip install without the “Externally Managed Environment” error. To verify:

1. Create and activate a virtual environment.
2. Try to install a package:

   pip install requests
   

If the installation succeeds without errors, you’ve successfully configured your environment.

6. Python Version Specifics

The “Externally Managed Environment” error, enforced by PEP 668, is more prevalent in newer Python versions and Linux distributions. While it can appear in Python 3.7 and later, it’s become a standard practice in distributions shipping with Python 3.9, 3.10, 3.11, and beyond. Ubuntu versions 22.04 LTS and later, for instance, are likely to enforce this. Older versions of Ubuntu might not have this restriction, but it’s still best practice to use virtual environments regardless.

7. Modern Tools: Poetry and PDM

Beyond venv and virtualenv, modern tools like Poetry and PDM offer more comprehensive dependency management and build tooling, including built-in virtual environment handling.

• Poetry: Poetry uses a pyproject.toml file to manage dependencies, build packages, and publish to PyPI. It automatically creates and manages virtual environments for your projects. You can install Poetry using pipx:

  pipx install poetry
  

  Then, within your project directory:

  poetry install  # Installs dependencies and creates a virtual environment
  poetry add <package_name>  # Adds a new dependency
  poetry shell  # Activates the virtual environment
  

• PDM (Python Development Master): PDM is another modern package manager that also uses pyproject.toml and offers a similar workflow to Poetry. It’s known for its speed and PEP 582 support (which avoids the need for virtual environment activation in some cases). Install PDM with pipx:

  pipx install pdm
  

  Then, in your project:

  pdm install  # Installs dependencies
  pdm add <package_name>  # Adds a new dependency
  pdm run python your_script.py # Run without explicit activation (PEP 582)
  

Both Poetry and PDM are excellent choices for managing complex projects and streamline the development workflow considerably.

8. Example of a Broken System

Imagine you’ve globally installed an older version of the requests library using pip without a virtual environment. Later, a system update requires a newer version of requests for a critical system utility. The system updater tries to upgrade requests, but it encounters a conflict with your manually installed version. This could result in:

• Broken system utilities: The utility that needed the updated requests library might fail to start or function correctly. You might see cryptic error messages or unexpected behavior.
• Dependency conflicts: Other Python applications might depend on different versions of requests, leading to further conflicts and instability.
• Difficult troubleshooting: Diagnosing the root cause of the problem can be time-consuming, as you’ll need to untangle the dependencies and potentially reinstall packages.
• In extreme cases: you may need to use apt to reinstall core packages, or even to reinstall the operating system. This is a simplified example, but it illustrates the potential consequences of bypassing the safeguards provided by PEP 668 and virtual environments. Using virtual environments avoids this entire class of problems.

The “Externally Managed Environment” error is a protective measure, not a bug. Embrace virtual environments, or use tools like pipx, Poetry, or PDM, and you’ll have a much smoother and more stable Python development experience on Ubuntu. This best practice might seem like an extra step at first, but it will save you countless headaches in the long run.