后端技术 - duxiaoyao/pdp GitHub Wiki

fastapi - ASGI web api framework

WSGI: falcon vs flask vs hug
ASGI: fastapi

当前选择：fastapi + asyncpg + PostgreSQL

Q: Why fastapi?
fastapi is a high performance ASGI web API framework with integrated automatic data validation, serialization and documentation.

  1. ASGI: Starlette & Uvicorn
  2. RESTful API: openAPI, JSON Schema
  3. data validation, serialization and documentation: Pydantic using python type hinting.

PostgreSQL + asyncpg

Dev: asyncpg
Tools: pgcli, schemaspy, barman

sudo su - postgres
vi /etc/postgresql/11/main/postgresql.conf # check port
vi /etc/postgresql/11/main/pg_hba.conf # change peer to md5 for users except postgres
sudo service postgresql restart
createuser -d -l -P eric
dropuser eric

[python-dotenv better, pydantic enough most time] dynaconf - layered configuration system for Python applications

Per https://hackernoon.com/4-ways-to-manage-the-configuration-in-python-4623049e841b, there are four ways to manage configurations:

  1. use built-in data structure: class Config, DevConfig, TestConfig, ProdConfig, pick up the right env from command-line option "--env dev/test/prod"
  2. use external config file: .toml, .ini - configparser, .json, .yaml
  3. use environment variables: os.environ.get, python-dotenv
  4. use dynamic loading: clone both configuration and app repositories and make the app able to use configuration dynamically.
  5. use configuration service: central reliable service

Notes

  1. Configurations are separated into environments: [development], [staging], [testing] and [production]. Use [default] to provide comprehensive default values and [global] to provide global values to override in any other environment. The working environment is selected via the environment variable ENV_FOR_DYNACONF which defaults to [development].
  2. Sensitive data like tokens, secret keys and password can be stored in .secrets.* files and/or external storages like Redis or vault secrets server.
  3. Recommend dynaconf with external services like redis server

Practices

cd <project root>
mkdir config # placehold for config files
pipenv run dynaconf init -p config

# now you can edit settings.toml, .secrets.toml and .env under config directory
# for example, 

pipenv run dynaconf list
pipenv run dynaconf list --help
# override working env.
ENV_FOR_DYNACONF=production pipenv run dynaconf list

pytest

pytest in pycharm

cd tests
pipenv run pytest
pipenv run pytest -s
pipenv run pytest --show-capture=all --log-level=info

other tools

dependency&build&package management: pyenv + pipx + poetry, venv + pip for simple scenarios, Docker
IDE: Pycharm Community Edition (support IPython console), Visual Studio Code, JupyterLab
code linter, formatter and analysis: Ruff replacing flake8, autoflake, isort, pyupgrade and black
static type checker: pyright, mypy
import sorter: isort
data parsing, validation and setting management: pydantic
Web/API: FastAPI, Robyn(provided muchbetter performance with a Rust runtime)
CLI: typer + rich, click
DB: PostgreSQL, redis, SQLite, DuckDB - in-process OLAP, ArcticDB - DataFrame database
Test: pytest and its ecosystem (such as pytest-asyncio, pytest-bdd, pytest-mock, pytest-benchmark), Hypothesis, Tox, bandit (Static application security testing (SAST) designed to find common security issues in Python code)
TUI (Texual UI or UI in Terminal): Textual
Rich text and styles in Terminal: Rich
shell commands: subprocess
Remote execution and deployment: Fabric
file & directory: pathlib
Filesystem events monitor: watchdog, watchfiles
queue: arq (async+redis, for async scenarios), rq (for sync world), pgmq (PostgreSQL extention), huey (for simple scenarios, no reliable delivery), dramatiq & dramatiq_dashboard, schedule, repeat_every, ZeroMQ
stream processing: faust
cache: lru_cache, cacheout, diskcache, cachetools, redis, joblib memory
Distributed computing: Ray, Dask, Joblib, ZeroMQ
redis client: aioredis
http client: httpx
Workflow orchestration: Apache Airflow
Git: pre-commit
process management: supervisor
hash: hashlib
password hash: argon2-cffi
cryptographic: cryptography, PyCryptodome
Installer: PyInstaller, Nuitka
auth (OAuth, OpenID, JWS, JWK, JWA, JWT etc.): authlib
authorization (ACL, RBAC, ABAC): pycasbin
retrying: Tenacity
AI: PyTorch Lightning, pytorch-forecasting, PyTorch, sktime, Scikit-learn, PyCaret, Optuna, jieba, NLTK, spaCy
GUI: PyQt6/PySide6, PyQtGraph - Scientific Graphics and GUI Library for Python , Dear PyGui, Tkinter, Kivy
profiler: Scalene, py-spy, line_profiler, cProfile or profile (standard modules)
Visual Analysis: PyGWalker, ydata-profiling