Python Packaging — pyproject.toml

Python Namespace Packages

References:

• Packaging python
• Setup tools - doc

Mypy is a static type checker for Python. It allows developers to add type annotations to their Python code, making the codebase easier to understand, maintain, and debug. Mypy checks the types at compile time, not at runtime, ensuring that potential type errors are caught early.

How to Use Mypy

1. Install Mypy:

   pip install mypy

2. Add Type Annotations:

   def greet(name: str) -> str:
       return f"Hello, {name}"

3. Run Mypy:

   mypy script.py

Type Annotations Syntax:

• Variables:

  age: int = 25
  name: str = "Alice"
  is_active: bool = True

• Function Annotations:

  def add(a: int, b: int) -> int:
      return a + b

• Optional Types:

  from typing import Optional
  
  def get_user(user_id: int) -> Optional[str]:
      if user_id == 1:
          return "Alice"
      return None

• Lists, Tuples, and Dicts:

  from typing import List, Tuple, Dict
  
  names: List[str] = ["Alice", "Bob"]
  point: Tuple[int, int] = (10, 20)
  user_data: Dict[str, int] = {"age": 30}

Why Use Mypy?

• Early Error Detection: Catches potential bugs before runtime.
• Code Clarity: Type annotations serve as documentation.
• Refactoring Safety: Easier to maintain and refactor codebases.

When you run mypy script.py, Mypy checks your Python file (script.py) for type correctness based on the type annotations you provided. Here's what the output might look like, depending on the state of your code:

1. No Type Errors (Success Case)

Success: no issues found in 1 source file

Explanation:
This means Mypy found no type-related issues in your file. The code complies with all the type hints and annotations provided.

2. Type Errors Found Example Code (script.py):

def add(a: int, b: int) -> int:
    return a + b

result = add(10, "20")  # Incorrect argument type

Running Mypy:

mypy script.py

Output:

script.py:4: error: Argument 2 to "add" has incompatible type "str"; expected "int"
Found 1 error in 1 file (checked 1 source file)

Explanation:

• script.py:4 - The file and line number where the issue occurred.
• Argument 2 to "add" - Describes the problematic argument.
• incompatible type "str"; expected "int" - Indicates that a str was passed, while an int was expected based on the function signature.

3. Multiple Errors If multiple issues exist, Mypy lists them one by one:

script.py:3: error: Argument 1 to "add" has incompatible type "str"; expected "int"
script.py:4: error: Incompatible return value type (got "str", expected "int")
Found 2 errors in 1 file (checked 1 source file)

4. Warnings and Additional Info If you enable more strict options like --strict, Mypy provides detailed warnings such as:

• Missing type hints (--disallow-untyped-defs)
• Return types not matching
• Incompatible operations

Common Command-Line Options

• --strict: Enables strict type checking.
• --ignore-missing-imports: Ignores errors related to missing module type hints.
• --check-untyped-defs: Checks code even when functions lack type annotations.

Pylint is a popular static code analysis tool for Python that checks your code against the PEP 8 style guide and best practices. It helps identify potential issues such as code smells, logical errors, and violations of coding standards.

How to Install Pylint

pip install pylint

Running Pylint

pylint script.py

Pylint Output Breakdown When you run Pylint, it generates a detailed report, assigning scores and showing errors, warnings, and suggestions.

Example: Basic Python Script (script.py)

# script.py

def greet(name):
    print(f"Hello, {name}")

greet("Alice")

Running Pylint:

pylint script.py

Sample Output:

************* Module script
script.py:1:0: C0114: Missing module docstring (missing-module-docstring)
script.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
script.py:3:16: W0613: Unused argument 'name' (unused-argument)

--------------------------------------------------------------------
Your code has been rated at 7.50/10 (previous run: 8.33/10, -0.83)

Understanding the Output:

1. C0114 - Missing module docstring:
   No top-level documentation provided for the module (script.py).

2. C0116 - Missing function docstring:
   The function greet lacks a docstring explaining what it does.

3. W0613 - Unused argument:
   The argument name is not used in the function.

4. Code Rating:
   The rating (e.g., 7.50/10) reflects code quality. A perfect score is 10/10.

Pylint Message Categories:

Fixing the Example:

"""
This module demonstrates a simple greeting function.
"""

def greet(name: str) -> None:
    """Greets the user by name."""
    print(f"Hello, {name}")

greet("Alice")

Re-running Pylint:

pylint script.py

New Output:

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 7.50/10, +2.50)

Commonly Used Pylint Options:

1. Run with a specific message category:

   pylint --disable=C script.py  # Disable convention messages

2. Generate a detailed report:

   pylint --reports=y script.py

3. Ignore specific errors:

   pylint --disable=missing-docstring script.py

4. Configure Pylint using a config file:

   pylint --generate-rcfile > .pylintrc

Why Use Pylint?

• Code Consistency: Ensures adherence to PEP 8 standards.
• Early Error Detection: Finds logical errors before execution.
• Improved Maintainability: Helps keep the code clean and readable.

Running pylint --reports=y script.py generates a more detailed report with a summary of various metrics about your code. This includes statistics like the number of code lines, comments, classes, functions, and more.

Example: script.py

"""
This module demonstrates a simple greeting function.
"""

def greet(name: str) -> None:
    """Greets the user by name."""
    print(f"Hello, {name}")

def add(a: int, b: int) -> int:
    """Adds two numbers."""
    return a + b

greet("Alice")
result = add(2, 3)
print(result)

Running Pylint with Reports:

pylint --reports=y script.py

Sample Output:

************* Module script
script.py:1:0: C0114: Missing module docstring (missing-module-docstring)

-------------------------------------------------------------------
Your code has been rated at 9.00/10 (previous run: 8.00/10, +1.00)

-------------------------------------------------------------------
Statistics by type
-------------------------------------------------------------------
+---------+-------+-----------+-----------+-----------+----------------+
|  Type   | Total |  Missed   |  Covered  |  % Good  |    Average     |
+---------+-------+-----------+-----------+-----------+----------------+
| Classes |   0   |     0     |     0     |   100%   |       0        |
| Methods |   0   |     0     |     0     |   100%   |       0        |
| Functions |  2   |     0     |     2     |   100%   |       2        |
| Statements|  6   |     0     |     6     |   100%   |       6        |
| Comments |   2   |     0     |     2     |   100%   |       2        |
| Docstrings|  2   |     0     |     2     |   100%   |       2        |
+---------+-------+-----------+-----------+-----------+----------------+

-------------------------------------------------------------------
Messages
-------------------------------------------------------------------
+----------------------------+------------+
| Message Type               | Occurrences|
+----------------------------+------------+
| convention (C)             |     1      |
| refactor (R)               |     0      |
| warning (W)                |     0      |
| error (E)                  |     0      |
| fatal (F)                  |     0      |
+----------------------------+------------+

-------------------------------------------------------------------
Global evaluation
-------------------------------------------------------------------
Your code has been rated at 9.00/10 (previous run: 8.00/10, +1.00)

Report Breakdown:

1. Statistics by Type:

   • Classes: Number of classes defined.
   • Methods: Number of methods in the classes.
   • Functions: Number of standalone functions.
   • Statements: Total executable lines of code.
   • Comments: Total number of comments.
   • Docstrings: Number of docstrings for functions/classes/modules.

2. Messages Summary:

   • Convention (C): Minor issues like missing docstrings or naming style violations.
   • Refactor (R): Code that can be improved for clarity or performance.
   • Warning (W): Potential problems or questionable code.
   • Error (E): Definite errors that would cause the program to fail.
   • Fatal (F): Serious issues that prevent Pylint from proceeding.

3. Global Evaluation:

   • Displays the overall score out of 10/10, reflecting the code quality and adherence to best practices.

Why Use --reports=y?

• Detailed Metrics: Helps analyze code structure and maintainability.
• Team Collaboration: Useful for team code reviews and continuous integration (CI).
• Historical Tracking: Tracks improvements by comparing code quality scores.

Poetry is a tool for dependency management and packaging in Python. It simplifies the process of creating and managing Python projects by handling dependencies, virtual environments, and packaging in an efficient and user-friendly way.

Features of Poetry:

1. Dependency Management: Poetry ensures you have all dependencies for your project and resolves version conflicts automatically.
2. Virtual Environments: Poetry automatically creates and manages virtual environments for your project.
3. Version Management: Supports semantic versioning and can automatically update versions based on changes.
4. Publishing: Easily publish your projects to package repositories like PyPI.

Installation

curl -sSL https://install.python-poetry.org | python3 -

After installation, ensure poetry is available in your PATH by checking its version:

poetry --version

Basic Commands

1. Create a New Project

   poetry new <project_name>

   This creates a project structure like:

   project_name/
   ├── pyproject.toml
   ├── README.rst
   ├── project_name/
   │   └── __init__.py
   └── tests/
       └── __init__.py
   

2. Initialize Poetry in an Existing Project

   poetry init

   Interactive prompt to set up pyproject.toml.

3. Add Dependencies

   poetry add <package_name>
   poetry add <package_name>@<version>
   poetry add <package_name> --dev  # Development dependency

4. Remove Dependencies

   poetry remove <package_name>

5. Install All Dependencies

   poetry install

6. Update Dependencies

   poetry update

7. Run Scripts or Commands

   poetry run <command>

8. Activate the Virtual Environment

   poetry shell

9. Check Dependency Status

   poetry check

10. Publish a Package

    poetry publish --build

11. Version Management

    poetry version <major|minor|patch>

Sample Project: Versioning Automatically

Here, we’ll create a simple project that demonstrates automatic versioning with Poetry.

Step 1: Create a Project

poetry new poetry_version_demo
cd poetry_version_demo

Step 2: Add a Sample Script Edit poetry_version_demo/__init__.py:

def main():
    print("Hello from Poetry Version Demo!")

Step 3: Manage Dependencies Add semver (Semantic Versioning library):

poetry add semver

Step 4: Automatic Versioning Script Create a script to manage version updates in scripts/update_version.py:

import semver
import toml

def update_version(part):
    pyproject_file = "pyproject.toml"
    with open(pyproject_file, "r") as f:
        data = toml.load(f)

    current_version = data["tool"]["poetry"]["version"]
    new_version = str(semver.VersionInfo.parse(current_version).next_version(part))
    
    data["tool"]["poetry"]["version"] = new_version
    
    with open(pyproject_file, "w") as f:
        toml.dump(data, f)
    
    print(f"Updated version from {current_version} to {new_version}")

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2 or sys.argv[1] not in ["major", "minor", "patch"]:
        print("Usage: python update_version.py <major|minor|patch>")
        sys.exit(1)

    update_version(sys.argv[1])

Step 5: Test Automatic Versioning Run the script to update the version:

python scripts/update_version.py patch

This updates the version in pyproject.toml automatically.

Step 6: Run the Project

poetry run python -m poetry_version_demo

Summary of Commands Used

1. Project Creation: poetry new poetry_version_demo
2. Add Dependency: poetry add semver
3. Version Update: python scripts/update_version.py patch
4. Run Script: poetry run python -m poetry_version_demo

Detailed Explanation of Poetry Commands

1. poetry install

Description:
This command installs all the dependencies listed in your pyproject.toml file, including their specific versions as locked in poetry.lock.

Key Options:

• --no-root: Skip installing the current project as a package. Useful when you don't need the project itself installed.
• --only <group>: Install dependencies only for a specific group, such as dev or test.
• --without <group>: Skip installing dependencies for a specific group.
• --with <group>: Include optional groups not installed by default.

Usage:

• Install all dependencies:

  poetry install

• Install only development dependencies:

  poetry install --only dev

• Skip development dependencies:

  poetry install --without dev

2. poetry update

Description:
This command updates all dependencies to their latest versions allowed by the constraints in pyproject.toml. It also regenerates the poetry.lock file.

Key Options:

• --with <group>: Include optional groups during the update.
• --without <group>: Skip specific groups during the update.
• --dry-run: Simulate the update without applying changes.
• --lock: Update the lock file only, without installing the updated dependencies.

Usage:

• Update all dependencies:

  poetry update

• Update specific dependency:

  poetry update <package_name>

• Preview updates without applying:

  poetry update --dry-run

3. poetry run <command>

Description:
Run a command within the context of the project's virtual environment. This ensures that all dependencies are correctly configured.

Usage Examples:

• Run a Python script:

  poetry run python script.py

• Run a shell command:

  poetry run ls

• Execute a specific module:

  poetry run python -m http.server

Tips:

• Use poetry run when you want to execute commands with dependencies installed in the Poetry-managed virtual environment.

4. poetry shell

Description:
Spawns a new shell with the virtual environment activated. This allows you to run commands directly without prefixing them with poetry run.

Usage:

• Activate the virtual environment:

  poetry shell

• Exit the virtual environment:

  exit

Key Features:

• You can directly run Python scripts or commands within the activated environment without using poetry run.
• Useful for interactive work in the virtual environment.

5. poetry check

Description:
Validates the pyproject.toml file and checks for issues with the project setup.

Key Use Cases:

• Ensures the pyproject.toml file is well-formed.
• Verifies that the project dependencies are consistent and correctly specified.

Usage:

• Run a check:

  poetry check

Output:

• If there are issues, Poetry will list them, such as missing fields or invalid syntax in pyproject.toml.

Summary Table

This detailed explanation and usage guide should help you master these Poetry commands for managing Python projects effectively.

Pyro5 (Python Remote Objects 5) is a remote method invocation (RMI) library for Python. It lets you call methods on Python objects located on another machine (or process) as if they were local.

👉 Think of it like RPC (Remote Procedure Call) for Python objects, similar to Java RMI or a lightweight alternative to gRPC.

• You expose (publish) a Python object on one process/machine (the server).
• Other processes/machines (the clients) obtain a proxy to that object.
• Clients can call methods on the proxy, and Pyro5 takes care of the serialization, network communication, and returning results.

So, you don’t need to manually manage sockets, serialization, etc.

How to Use Pyro5

Install

pip install Pyro5

Server (expose an object)

import Pyro5.api

@Pyro5.api.expose
class Calculator:
    def add(self, a, b):
        return a + b

# Start a Pyro daemon (server)
daemon = Pyro5.api.Daemon()  
uri = daemon.register(Calculator)  
print("Ready. Object uri =", uri)

daemon.requestLoop()  # keep server running

This server exposes a Calculator object.

import Pyro5.api

# Replace with URI printed by server
uri = "PYRO:obj_123456789@localhost:50000"
calculator = Pyro5.api.Proxy(uri)

print("3 + 4 =", calculator.add(3, 4))

✅ When you call calculator.add(3, 4), the request goes across the network, executes on the server, and returns the result.

Use Cases Pyro5 Solves

1. Distributed Computing

   • Run computation-heavy objects remotely (on another machine or process) and call them like local functions.

2. Microservices in Python

   • Expose Python services without needing REST or gRPC.
   • Useful for small internal services where full-blown frameworks are overkill.

3. Multi-Process Coordination

   • Share Python objects between different processes (even across machines) without manually handling sockets.

4. Prototyping Distributed Systems

   • Quick and easy to prototype communication between distributed components.

5. Reusing Legacy Code

   • Wrap an old Python module in a Pyro service and expose it without rewriting into HTTP/gRPC APIs.

Comparison with Alternatives

• vs. multiprocessing → Pyro5 works across machines, not just local processes.
• vs. socket / zmq → Pyro5 hides serialization, connection handling, and makes remote calls look like local calls.
• vs. gRPC/REST → Pyro5 is Python-only, simpler, and object-oriented (not language-agnostic).

When to Use Pyro5

• You have a Python-only distributed system.
• You want object-oriented remote calls instead of message-passing.
• You’re prototyping quickly and don’t want heavy infra like gRPC/REST.

RPyC stands for Remote Python Call. It’s a transparent and symmetric RPC (Remote Procedure Call) framework for Python.

👉 It lets you call functions, access objects, or even execute code on a remote machine as if you were working locally.

Think of it as:

• SSH + Python REPL + RPC combined 🚀

Key Features

• Symmetric: Both client and server can call each other’s functions/objects.
• Transparent: You don’t have to serialize/deserialize manually; remote objects look like local ones.
• Object proxying: You can directly use remote objects (list, dict, classes, etc.).
• Standard RPC and classic service model (like exposing services).

How to Use RPyC

Install

pip install rpyc

Start a Classic RPyC Server

# server.py
import rpyc

class MyService(rpyc.Service):
    def on_connect(self, conn):
        print("Client connected")

    def on_disconnect(self, conn):
        print("Client disconnected")

    def exposed_add(self, x, y):  # exposed_* methods are accessible remotely
        return x + y

if __name__ == "__main__":
    from rpyc.utils.server import ThreadedServer
    t = ThreadedServer(MyService, port=18861)
    t.start()

Client

# client.py
import rpyc

conn = rpyc.connect("localhost", 18861)   # connect to server
print(conn.root.add(5, 7))  # calls exposed_add remotely

✅ Output:

12

Advanced: Transparent Access to Remote Objects

Unlike Pyro5, RPyC lets you directly work with remote objects:

Server:

# server.py
import rpyc
class ListService(rpyc.Service):
    def exposed_get_list(self):
        return [1, 2, 3, 4]

Client:

import rpyc
conn = rpyc.connect("localhost", 18861)

remote_list = conn.root.get_list()
print(remote_list)          # [1, 2, 3, 4]
print(remote_list[0])       # works like a normal list!

⚡ This happens because RPyC wraps the list in a proxy, so you don’t need serialization/deserialization explicitly.

Use Cases RPyC Solves

1. Distributed Computing

   • Run Python code remotely (like a lightweight cluster).
   • Example: Offload heavy computation to another machine.

2. Remote Object Access

   • Directly interact with remote objects without converting them into messages or APIs.

3. Remote Administration

   • Connect to a Python process on another server and run commands interactively.
   • (It’s almost like an SSH session but Python-native).

4. Microservices / Remote APIs

   • Expose Python functions as services, similar to Pyro5 but with symmetric access.

5. Testing & Debugging

   • Access internal state of remote Python processes for inspection/debugging.

RPyC vs Pyro5