Next Steps - VladimirSiv/pytransflow GitHub Wiki

It would be nice to extend pytransflow with the following features

Transformation snippet would allow templating and parameterization of a group of transformations. For example, let's say that we are performing the same group of transformations multiple times just on different fields:

transformations:
  - add_field:
      name: a
      value: test
  - prefix:
      field: a
      value: pre_
  - postfix:
      field: a
      value: _post
  - add_field:
      name: b
      value: test
  - prefix:
      field: b
      value: pre_
  - postfix:
      field: b
      value: _post
    ...

It would be nice to have a feature to define a transformation snippet

snippet:
  name: snippet_example
  transformations:
    - add_field:
        name: <field>
        value: <val_1>
    - prefix:
        field: <field>
        value: <val_2>
    - postfix:
        field: <field>
        value: <val_3>

and the use it in a flow

transformations:
  - snippet:
      name: snippet_example
      field: a
      val_1: test
      val_2: pre_
      val_3: _post
  - snippet:
      name: snippet_example
      field: b
      val_1: test
      val_2: pre_
      val_3: _post
    ...

These snippets can be defined on a flow or project level:

• Flow - Use snippet key in flow configuration. If the snippet is defined on a flow level, it will be available only in that flow.
• Project - Allow importing of snippets from a SNIPPETS_PATH directory. These snippets will be available in all flows by default.

Currently, pytransflow works only on dict records. Meaning, the input records are defined and used in the following fashion

records = [{"a": 1}, {...}, {...}, ...]
flow = Flow(name="<flow>")
flow.process(records)

It would be nice to extend this and allow completely custom objects as input records. This could be achieved by providing a custom class to pytransflow, and replacing the default dict behavior. However, this custom class has to implement required method in order to be used as Record.

For more information see pytransflow.core.record.record.Record class

A nice to have feature would be to define ignore_errors on a flow level. These ignore errors will be applied to all transformation in the flow. This would improve the readability and simplicity of flow configurations.

Instead of

transformations:
  - add_field:
      name: a
      value: 1
      ignore_errors:
        - output_already_exists
  - add_field:
      name: b
      value: 2
      ignore_errors:
        - output_already_exists
  - add_field:
      name: c
      value: 3
      ignore_errors:
        - output_already_exists
  ...

Support

ignore_errors:
  - output_already_exists
transformations:
  - add_field:
      name: a
      value: 1
  - add_field:
      name: b
      value: 2
  - add_field:
      name: c
      value: 3
  ...

Because pytransflow defines a sequence of transformations that begin with an initial record and apply subsequent transformations sequentially, the underlying structure can be represented as Directed Acyclic Graph (DAG).

This feature would give users a nice graph representation of the flow, how transformations are connected and how datasets are routed between them. Having this feature would greatly improve the readability, documenting, and debugging of large and complex flows.

Parsing of the flow and defining graph structure should be done within pytransflow, while the visualization should be done using some external third-party library.

pytransflow would significantly enhance its functionality with an extensive debugging mode that preserves the history of records. This mode would function in the same fashion as a data lineage within the workflow. Whenever a Failed Record or a successfully processed record occurs, users would be able to examine its complete history, tracing all transformations back to the original record.

Maintaining this history and enabling record traceability in case of errors would greatly improve the readability, testability, and maintenance of complex workflows. Moreover, leveraging DAG parsing, as previously mentioned, could further extend this history to generate clear graphs and comprehensive diagrams illustrating the transformation of each record through the processing pipeline.