Batch - KeynesYouDigIt/Knowledge GitHub Wiki

Batch processes process (relatively) large amounts of data in one unit of work.

Interesting case studies on why you would ever "batch" data Case-Study::why-batching-requests-is-faster-than-making-many-individual-requests. Note that in the Case study, Node, which is asynnchronous, has to manage and swap (or simply runs out of) resources for managing multiple database connections and round trips. However, even in a synchonous language like python, alloc and deallocating resources for each database roundtrip is inefficient as well. I think, Until our batches are so big that they overwhelm the DB, larger batches are more efficient (?) https://github.com/KeynesYouDigIt/Knowledge/wiki/Case-study::batching-in-psycopg2

• DAGs
• tools for DAGs
• Dagster
• Airflow
• dbt