Python Packaging — pyproject.toml

Python Namespace Packages

References:

• Packaging python
• Setup tools - doc

Mypy is a static type checker for Python. It allows developers to add type annotations to their Python code, making the codebase easier to understand, maintain, and debug. Mypy checks the types at compile time, not at runtime, ensuring that potential type errors are caught early.

How to Use Mypy

1. Install Mypy:

   pip install mypy

2. Add Type Annotations:

   def greet(name: str) -> str:
       return f"Hello, {name}"

3. Run Mypy:

   mypy script.py

Type Annotations Syntax:

• Variables:

  age: int = 25
  name: str = "Alice"
  is_active: bool = True

• Function Annotations:

  def add(a: int, b: int) -> int:
      return a + b

• Optional Types:

  from typing import Optional
  
  def get_user(user_id: int) -> Optional[str]:
      if user_id == 1:
          return "Alice"
      return None

• Lists, Tuples, and Dicts:

  from typing import List, Tuple, Dict
  
  names: List[str] = ["Alice", "Bob"]
  point: Tuple[int, int] = (10, 20)
  user_data: Dict[str, int] = {"age": 30}

Why Use Mypy?

• Early Error Detection: Catches potential bugs before runtime.
• Code Clarity: Type annotations serve as documentation.
• Refactoring Safety: Easier to maintain and refactor codebases.

When you run mypy script.py, Mypy checks your Python file (script.py) for type correctness based on the type annotations you provided. Here's what the output might look like, depending on the state of your code:

1. No Type Errors (Success Case)

Success: no issues found in 1 source file

Explanation:
This means Mypy found no type-related issues in your file. The code complies with all the type hints and annotations provided.

2. Type Errors Found Example Code (script.py):

def add(a: int, b: int) -> int:
    return a + b

result = add(10, "20")  # Incorrect argument type

Running Mypy:

mypy script.py

Output:

script.py:4: error: Argument 2 to "add" has incompatible type "str"; expected "int"
Found 1 error in 1 file (checked 1 source file)

Explanation:

• script.py:4 - The file and line number where the issue occurred.
• Argument 2 to "add" - Describes the problematic argument.
• incompatible type "str"; expected "int" - Indicates that a str was passed, while an int was expected based on the function signature.

3. Multiple Errors If multiple issues exist, Mypy lists them one by one:

script.py:3: error: Argument 1 to "add" has incompatible type "str"; expected "int"
script.py:4: error: Incompatible return value type (got "str", expected "int")
Found 2 errors in 1 file (checked 1 source file)

4. Warnings and Additional Info If you enable more strict options like --strict, Mypy provides detailed warnings such as:

• Missing type hints (--disallow-untyped-defs)
• Return types not matching
• Incompatible operations

Common Command-Line Options

• --strict: Enables strict type checking.
• --ignore-missing-imports: Ignores errors related to missing module type hints.
• --check-untyped-defs: Checks code even when functions lack type annotations.

Pylint is a popular static code analysis tool for Python that checks your code against the PEP 8 style guide and best practices. It helps identify potential issues such as code smells, logical errors, and violations of coding standards.

How to Install Pylint

pip install pylint

Running Pylint

pylint script.py

Pylint Output Breakdown When you run Pylint, it generates a detailed report, assigning scores and showing errors, warnings, and suggestions.

Example: Basic Python Script (script.py)

# script.py

def greet(name):
    print(f"Hello, {name}")

greet("Alice")

Running Pylint:

pylint script.py

Sample Output:

************* Module script
script.py:1:0: C0114: Missing module docstring (missing-module-docstring)
script.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
script.py:3:16: W0613: Unused argument 'name' (unused-argument)

--------------------------------------------------------------------
Your code has been rated at 7.50/10 (previous run: 8.33/10, -0.83)

Understanding the Output:

1. C0114 - Missing module docstring:
   No top-level documentation provided for the module (script.py).

2. C0116 - Missing function docstring:
   The function greet lacks a docstring explaining what it does.

3. W0613 - Unused argument:
   The argument name is not used in the function.

4. Code Rating:
   The rating (e.g., 7.50/10) reflects code quality. A perfect score is 10/10.

Pylint Message Categories:

Fixing the Example:

"""
This module demonstrates a simple greeting function.
"""

def greet(name: str) -> None:
    """Greets the user by name."""
    print(f"Hello, {name}")

greet("Alice")

Re-running Pylint:

pylint script.py

New Output:

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 7.50/10, +2.50)

Commonly Used Pylint Options:

1. Run with a specific message category:

   pylint --disable=C script.py  # Disable convention messages

2. Generate a detailed report:

   pylint --reports=y script.py

3. Ignore specific errors:

   pylint --disable=missing-docstring script.py

4. Configure Pylint using a config file:

   pylint --generate-rcfile > .pylintrc

Why Use Pylint?

• Code Consistency: Ensures adherence to PEP 8 standards.
• Early Error Detection: Finds logical errors before execution.
• Improved Maintainability: Helps keep the code clean and readable.

Running pylint --reports=y script.py generates a more detailed report with a summary of various metrics about your code. This includes statistics like the number of code lines, comments, classes, functions, and more.

Example: script.py

"""
This module demonstrates a simple greeting function.
"""

def greet(name: str) -> None:
    """Greets the user by name."""
    print(f"Hello, {name}")

def add(a: int, b: int) -> int:
    """Adds two numbers."""
    return a + b

greet("Alice")
result = add(2, 3)
print(result)

Running Pylint with Reports:

pylint --reports=y script.py

Sample Output:

************* Module script
script.py:1:0: C0114: Missing module docstring (missing-module-docstring)

-------------------------------------------------------------------
Your code has been rated at 9.00/10 (previous run: 8.00/10, +1.00)

-------------------------------------------------------------------
Statistics by type
-------------------------------------------------------------------
+---------+-------+-----------+-----------+-----------+----------------+
|  Type   | Total |  Missed   |  Covered  |  % Good  |    Average     |
+---------+-------+-----------+-----------+-----------+----------------+
| Classes |   0   |     0     |     0     |   100%   |       0        |
| Methods |   0   |     0     |     0     |   100%   |       0        |
| Functions |  2   |     0     |     2     |   100%   |       2        |
| Statements|  6   |     0     |     6     |   100%   |       6        |
| Comments |   2   |     0     |     2     |   100%   |       2        |
| Docstrings|  2   |     0     |     2     |   100%   |       2        |
+---------+-------+-----------+-----------+-----------+----------------+

-------------------------------------------------------------------
Messages
-------------------------------------------------------------------
+----------------------------+------------+
| Message Type               | Occurrences|
+----------------------------+------------+
| convention (C)             |     1      |
| refactor (R)               |     0      |
| warning (W)                |     0      |
| error (E)                  |     0      |
| fatal (F)                  |     0      |
+----------------------------+------------+

-------------------------------------------------------------------
Global evaluation
-------------------------------------------------------------------
Your code has been rated at 9.00/10 (previous run: 8.00/10, +1.00)

Report Breakdown:

1. Statistics by Type:

   • Classes: Number of classes defined.
   • Methods: Number of methods in the classes.
   • Functions: Number of standalone functions.
   • Statements: Total executable lines of code.
   • Comments: Total number of comments.
   • Docstrings: Number of docstrings for functions/classes/modules.

2. Messages Summary:

   • Convention (C): Minor issues like missing docstrings or naming style violations.
   • Refactor (R): Code that can be improved for clarity or performance.
   • Warning (W): Potential problems or questionable code.
   • Error (E): Definite errors that would cause the program to fail.
   • Fatal (F): Serious issues that prevent Pylint from proceeding.

3. Global Evaluation:

   • Displays the overall score out of 10/10, reflecting the code quality and adherence to best practices.

Why Use --reports=y?

• Detailed Metrics: Helps analyze code structure and maintainability.
• Team Collaboration: Useful for team code reviews and continuous integration (CI).
• Historical Tracking: Tracks improvements by comparing code quality scores.

Poetry is a tool for dependency management and packaging in Python. It simplifies the process of creating and managing Python projects by handling dependencies, virtual environments, and packaging in an efficient and user-friendly way.

Features of Poetry:

1. Dependency Management: Poetry ensures you have all dependencies for your project and resolves version conflicts automatically.
2. Virtual Environments: Poetry automatically creates and manages virtual environments for your project.
3. Version Management: Supports semantic versioning and can automatically update versions based on changes.
4. Publishing: Easily publish your projects to package repositories like PyPI.

Installation

curl -sSL https://install.python-poetry.org | python3 -

After installation, ensure poetry is available in your PATH by checking its version:

poetry --version

Basic Commands

1. Create a New Project

   poetry new <project_name>

   This creates a project structure like:

   project_name/
   ├── pyproject.toml
   ├── README.rst
   ├── project_name/
   │   └── __init__.py
   └── tests/
       └── __init__.py
   

2. Initialize Poetry in an Existing Project

   poetry init

   Interactive prompt to set up pyproject.toml.

3. Add Dependencies

   poetry add <package_name>
   poetry add <package_name>@<version>
   poetry add <package_name> --dev  # Development dependency

4. Remove Dependencies

   poetry remove <package_name>

5. Install All Dependencies

   poetry install

6. Update Dependencies

   poetry update

7. Run Scripts or Commands

   poetry run <command>

8. Activate the Virtual Environment

   poetry shell

9. Check Dependency Status

   poetry check

10. Publish a Package

    poetry publish --build

11. Version Management

    poetry version <major|minor|patch>

Sample Project: Versioning Automatically

Here, we’ll create a simple project that demonstrates automatic versioning with Poetry.

Step 1: Create a Project

poetry new poetry_version_demo
cd poetry_version_demo

Step 2: Add a Sample Script Edit poetry_version_demo/__init__.py:

def main():
    print("Hello from Poetry Version Demo!")

Step 3: Manage Dependencies Add semver (Semantic Versioning library):

poetry add semver

Step 4: Automatic Versioning Script Create a script to manage version updates in scripts/update_version.py:

import semver
import toml

def update_version(part):
    pyproject_file = "pyproject.toml"
    with open(pyproject_file, "r") as f:
        data = toml.load(f)

    current_version = data["tool"]["poetry"]["version"]
    new_version = str(semver.VersionInfo.parse(current_version).next_version(part))
    
    data["tool"]["poetry"]["version"] = new_version
    
    with open(pyproject_file, "w") as f:
        toml.dump(data, f)
    
    print(f"Updated version from {current_version} to {new_version}")

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2 or sys.argv[1] not in ["major", "minor", "patch"]:
        print("Usage: python update_version.py <major|minor|patch>")
        sys.exit(1)

    update_version(sys.argv[1])

Step 5: Test Automatic Versioning Run the script to update the version:

python scripts/update_version.py patch

This updates the version in pyproject.toml automatically.

Step 6: Run the Project

poetry run python -m poetry_version_demo

Summary of Commands Used

1. Project Creation: poetry new poetry_version_demo
2. Add Dependency: poetry add semver
3. Version Update: python scripts/update_version.py patch
4. Run Script: poetry run python -m poetry_version_demo

Detailed Explanation of Poetry Commands

1. poetry install

Description:
This command installs all the dependencies listed in your pyproject.toml file, including their specific versions as locked in poetry.lock.

Key Options:

• --no-root: Skip installing the current project as a package. Useful when you don't need the project itself installed.
• --only <group>: Install dependencies only for a specific group, such as dev or test.
• --without <group>: Skip installing dependencies for a specific group.
• --with <group>: Include optional groups not installed by default.

Usage:

• Install all dependencies:

  poetry install

• Install only development dependencies:

  poetry install --only dev

• Skip development dependencies:

  poetry install --without dev

2. poetry update

Description:
This command updates all dependencies to their latest versions allowed by the constraints in pyproject.toml. It also regenerates the poetry.lock file.

Key Options:

• --with <group>: Include optional groups during the update.
• --without <group>: Skip specific groups during the update.
• --dry-run: Simulate the update without applying changes.
• --lock: Update the lock file only, without installing the updated dependencies.

Usage:

• Update all dependencies:

  poetry update

• Update specific dependency:

  poetry update <package_name>

• Preview updates without applying:

  poetry update --dry-run

3. poetry run <command>

Description:
Run a command within the context of the project's virtual environment. This ensures that all dependencies are correctly configured.

Usage Examples:

• Run a Python script:

  poetry run python script.py

• Run a shell command:

  poetry run ls

• Execute a specific module:

  poetry run python -m http.server

Tips:

• Use poetry run when you want to execute commands with dependencies installed in the Poetry-managed virtual environment.

4. poetry shell

Description:
Spawns a new shell with the virtual environment activated. This allows you to run commands directly without prefixing them with poetry run.

Usage:

• Activate the virtual environment:

  poetry shell

• Exit the virtual environment:

  exit

Key Features:

• You can directly run Python scripts or commands within the activated environment without using poetry run.
• Useful for interactive work in the virtual environment.

5. poetry check

Description:
Validates the pyproject.toml file and checks for issues with the project setup.

Key Use Cases:

• Ensures the pyproject.toml file is well-formed.
• Verifies that the project dependencies are consistent and correctly specified.

Usage:

• Run a check:

  poetry check

Output:

• If there are issues, Poetry will list them, such as missing fields or invalid syntax in pyproject.toml.

Summary Table

This detailed explanation and usage guide should help you master these Poetry commands for managing Python projects effectively.