Updating Codegen Providers - RepreZen/GenFlow GitHub Wiki

In GenFlow, a Codegen Provider is a library that generates code, documentation, or other artifacts from a machine-readable API description and/or other inputs. GenFlow defines a separate GenTemplate for each distinct type of code that the Codegen Provider can generate.

This topic describes the process of updating Codegen Providers, specifically Swagger Codegen and OpenAPI Generator.

GenFlow's integration of Swagger Codegen (SCG) and OpenAPI Generator (OAG) makes use of an IGenTemplateGroup implementation that uses ServiceLoader to find all the generation modules and wraps each in an IGenTemplate implemenation. (ServiceLoader itself is not actually used, but rather the classloader is used to find resources that use the ServiceLoader resource naming conventions, and those resources are used to collect generator class names.)

An additional feature built into the support classes is a capability to manage the offered GenTemplates in a few ways:

• GenTemplates for individual generator modules can be marked for suppression
• GenTemplate names can be supplied for specific generator modules
• Named parameters can be marked as required and/or supplied with default values.

These things are maintained with CSV files, known as module info files, that are kept in the source trees.

Module info files are created by the GenFlow build process, each time a build is performed with a previously unused version of the SCG and/or OAG libraries configured in the GenFlow root pom file. This is done by running main classes ScgCodegenModuleDiscovery and OagCodegenModuleDiscovery during the generate-sources build phase. Both classes are simple extensions of GenModuleDiscovery, which is in the genflow-common project.

The discovery process proceeds as follows:

1. If module info files already exist for whatever version of the codegen library (SCG or OAG) is currently configured in the GenFlow root pom file (the current version), the program exits without doing anything. Therefore, module info files for a given library version are never re-generated once they have been checked into git, unless someone explicitly causes them to be re-generated, e.g. by running the discovery class manually or removing the existing module info files.
   
   
2. Otherwise, the most recent prior module info is loaded into in-memory structures (as defined in the GenModuleInfo class in genflow-common). This forms the base info for the new module info. "Most recent prior" means module info for the highest library version for which the files already exist, and which is lower than the current version. Only M.m.p version labels are understood, where M, m, and p are numerical major, minor and patch version numbers - i.e., no snapshot or other qualifiers can be used. Version ordering is numerical by major, minor, and patch version numbers, in that order of preference.
   
   
3. Each module appearing in the base info is copied to the new info, except those marked as deleted.
   
   
4. ServiceLoader is used to scan the classpath for all available codegen modules. This should discover only classes that are supplied by the SCG/OAG libraries. These are classes that extend io.swagger.codegen.CodegenConfig or org.openapitools.codegen.CodegenConfig, respectively.
   
   
5. For each module, we locate its module info record in the base info set, or create a new record if it's not present. Then we obtain the following information from the module class instance returned by ServiceLoader and copy it into the module info record:
   
   
   • The module's class name (this is the record key)
     
     
   • The module's declared type (e.g. CLIENT, SERVER, DOCUMENTATION, etc.). (This is obtained via the module instance's getTag() method.) 3. The module's declared name
     
     
   • The module's named parameters, referred to as "client options" in SCG and OAG libraries (instances of class io.swagger.codegen.CliOption and org.openapitools.codegen.CliOption, respectively). For each parameter we record its name and its description. Default values and requiredness are supported in the module info data, but they are not presently part of the client option abstraction in either SCG or OAG, so all parameters end up as optional and with no default value after discovery. (These can be overridden by manually editing the corresponding CSV file, but we have never done this to-date).
     
     
6. Next, we compute a derived name for the module, making use of the above information, and we store that into the info record as well, in the DerivedDisplayName property. This is the name your application should display in its UI, unless a manual override is supplied for the module.
   
   
7. Additional fields are set as follows in each module info record:
   
   
   • DisplayName - an empty string by default; if it's present, it is used as a name for the wrapped GenTemplate instead of the derived name computed during discovery.
     
     
   • Suppressed flag - a boolean indicating whether this module should be suppressed (i.e. not wrapped as a GenTemplate). This flag is always set false by discovery but can be manually overridden in the CSV file.
     
     
   • Deleted flag - a boolean that is set by discovery to indicate that this module appeared in the base info but was not encountered during the ServiceLoader scan performed in discovery. This means that the module was decommissioned in the SCG/OAG library. Deleted modules always have their Changed flag set and their New and Vetted flags cleared. The setting of the Deleted flag has no impact on code behavior outside discovery; it is intended to make it easy to identify decommissioned modules when writing release notes, etc.
     
     
   • New flag - a boolean that is set by discovery to indicate that this module was encountered during the ServiceLoader scan but was not present in the base info set. This means that the module was added to teh SCG/OAG library sometime between the base version and the current version. New modules always have their Changed flags set as well.
     
     
   • Changed flag - a boolean that is set by discovery whenever any of the information copied from the base info set differs from what was obtained from the same module during discovery. The flag is set for all new and deleted modules, but also any modules whose reported type or name has changed, or whose derived name has changed for some other reason (e.g. a change in the derived name algorithm).
     
     
   • Vetted flag - a boolean that is never set by discovery but is carried forward from the base info set and is intended to be manually edited in the CSV file. It indicates that the module has been reviewed and approved for inclusion in GenFlow applications. The primary purpose of the review is to ensure that a reasonable name will be used in the wrapped GenTemplate - either the automatically derived name or a manually specified DisplayName.
     
     N.B. Previously vetted modules will show up as vetted in the new module info, even if some of the captured info has changed. In this case, the module's Changed flag will also be set. Reviewers should check such records to ensure that display name will still be appropriate.

Two files constitute the module info for a given codegen library version: modulesInfo_M.m.p.csv and moduleParams_M.m.p.csv, where M.m.p is the codegen library version to which the info applies.

Each module is represented by a single record in the modulesInfo file, and by multiple records (one for each discovered parameter) in the moduleParams file.

The files appear in the GenFlow swagger-codegen and openapi-generator projects, in src/main/resources/com/reprezen/genflow/swagger/codegen and src/main/resources/com/reprezen/genflow/openapi/generator, respectively (i.e. in the same "package" as the discovery classes, but in the resources tree).

The module info files can be edited with Excel or LibreOffice Calc (or probably Google Sheets and other spreadsheet programs). After making changes, the files should be saved out as CSV files again.

Columns for flags (e.g. Suppressed and Vetted) should be left empty to indicate false values, and should contain a single asterisk ("*") for true values. 

The following procedure details how to make changes to GenFlow sources to upgrade to a new version of SCG and/or OAG libraries. These changes should be done in the context of the general change process for GenFlow.

1. Modify the io.swagger:swagger-codegen or org.openapitools:openapi-generator versions specified in the root-level GenFlow pom.xml file. These are managed dependencies that will affect all the GenFlow modules. Do not get confused by dependencies with the same artifact ids but in the com.reprezen.genflow group; those are the GenFlow projects that adapt the codegen modules as GenTemplates.
   
   
2. Perform a local GenFlow build (generally: mvn clean install from the GenFlow working tree root)
   
   
3. Edit the module info files that were generated for your new version(s), and alter DisplayName, Suppressed, and Vetted column values as desired.
   
   You should leave all the other columns unchanged, except that if you wish, you may also make changes in the DefaultValue and Required columns of the modelParams file. The SCG and OAG configuration objects don't support these options, but altering them here will affect GenTarget files as if they were supported.
   
   
4. Rebuild GenFlow and test locally in your application to ensure that all expected GenTemplates appear with the expected names.
   
   
5. Check changes into an appropriate GenFlow repo branch, create a PR, create a test build of your application, merge your PR, etc.

The review in step 3 need not be performed immediately when a library upgrade is performed, though this is recommended since new generator modules will not be available in applications until they are marked as vetted. CSV files can be edited and checked in at any time in the future, and will be used in subsequent builds.