1. Overview of auditing
   1. Using audit types
   2. Structure of audit-entity types
   3. Structure of synthetic audit-entity types
   4. Evolution of audit-entity types (type versions)
      1. Evolutionary scenarios
         1. An edge case
2. Application configuration
   1. Auditing Modes
3. Generating audit sources
4. Web UI configuration for audit types
   1. Standalone centre
   2. Embedded centre
5. Security Tokens
6. Auditing external entity types

Annotating an entity type with @Audited makes the platform recognise it as such, which enables automatic auditing (but see "Auditing Modes" below). Saving an audited entity instance creates an audit record, which is represented by an instance of a corresponding audit-entity type and one or more instances of an audit-prop type.

This procedure is comprised of the following steps:

1. E, instance of an audited entity type, is saved.
2. AE, audit-entity instance, is created, filled with the contents of E, and saved.
3. For each dirty property of E, an audit-prop instance is created, its many-to-one relationship to AE is captured, and it is saved.

The source code for audit types should be generated as described in this document.

For each audited entity type, there are the following audit types:

• Persistent audit-entity type (or just audit-entity type), which reflects the structure of the audited entity type at the time of generation.

  It contains common service properties and audit-properties that correspond to persistent properties of the audited entity type.

  The naming scheme for such types is {Entity}_a3t_{n}, where {Entity} is the audited entity type's name and {n} is the type version (more on this later).

  The abstract type is AbstractAuditEntity.

• Persistent audit-prop type (or just audit-prop type), which is the "many" side of the one-to-many relationship with a corresponding audit-entity type.

  This entity type is used to model changes to values of audited properties.

  The naming scheme for such types is {Entity}_a3t_{n}_Prop.

  The abstract type is AbstractAuditProp.

• Synthetic audit-entity type, which combines all versions of persistent audit-entity types.

  This entity type is used to aggregate all audit data. Web UI configuration can be dynamically generated for such types.

  The naming scheme for such types is Re{Entity}_a3t.

  The abstract type is AbstractSynAuditEntity.

• Synthetic audit-prop type, which combines all versions of audit-prop types.

  This type is the "many" side of the one-to-many relationship with a corresponding synthetic audit-entity type.

  The naming scheme for such types is Re{Entity}_a3t_Prop.

  The abstract type is AbstractSynAuditProp.

Audit types should not be used directly, as generated code is prone to changes. Instead, abstract types should be used.

Audit types can be obtained using IAuditTypeFinder. That is, Class objects that represent audit types.

The only exception to this rule is synthetic audit-entity types. Sometimes one cannot avoid using them. For example, as a class reference in "open-master-action" entity types or menu items, or in EQL queries, or when generating code with IDE plugins. Meta-models are also generated for synthetic audit-entity types. Nevertheless, prefer the abstract type if possible.

ISynAuditEntityDao is an interface for audit-entity companions that provides various operations on audit-entities.

AuditUtils is a collection of utilities that may come in handy when working with auditing.

Effective and correct evolution of audit types requires an understanding of their structure. In this section, the structure of persistent audit-entity types is described in detail.

Persistent audit-entity types form a hierarchy, where the latest audit-entity type (with the greatest version) is at the bottom (is the most specific type).

For an audited entity type E with n audit type versions, its persistent audit-entity types satisfy AE_1 > ... > AE_n, where a > b means "a is a superclass of b".

To support removal of properties, an audit-entity type declares the removed property as inactive, effectively hiding the inherited active one.

When speaking of audit-entity types of some entity E, the term "current audit-entity type" or "latest audit-entity type" may be used to denote the audit-entity type with the greatest version, but one may also simply say "audit-entity type". The term "prior" should be used to refer to audit type versions that are less than the latest one. The same applies to persistent audit-prop types, which are versioned just like audit-entity types are. Therefore, for each audit type version there is a separate one-to-many association.

From the perspective of actual data auditing, only the latest audit type version is relevant. That is, when creating an audit record, only the latest audit-entity type and its audit-prop are used. Prior audit types are never used to create new data, so the existing data associated with them can be said to be historical.

Synthetic audit types are not versioned. Their role lies in accumulating data from all versions of persistent audit-entity types.

The structure of a synthetic audit-entity type is a combination of all versions of persistent audit-entity types. Audit-properties from all versions of persistent audit-entity types are declared. In terms of the data model, a synthetic audit-entity is a union. The nature of union requires structural differences between versions of persistent audit-entity types to be resolved so that a rectangular structure is obtained.

An example of a structural difference is when a new audit-entity type is generated with a new property, meaning that the new property is absent in all prior audit-entity types.

To satisfy the requirements of union, "holes" are filled in with empty values -- typically, null, but for boolean properties -- false.

Audit-entity types should evolve along their audited entity types. The responsibility for this lies with application developers.

For a TG application to be in a correct state, each audit-entity type's structure should accurately reflect that of its corresponding audited entity type. Therefore, when an audited entity type's structure is changed, the structure of its audit-entity types should be synchronised. At the same time, destructive changes that could lead to a loss of audit data should be avoided.

For example, if Vehicle is audited and its property checkupDate is removed, then removing a corresponding property from its audit-entity type would lead to the loss of audit data for that property.

This challenge is addressed by having multiple versions of an audit-entity type exist simultaneously. This model also introduces the concept of a current audit-entity type, which is an audit-entity type with the greatest (current) version.

For each audited entity type, its current audit-entity type must accurately reflects its structure.

In practice, this means that in most cases application developers should generate a new audit-entity type when the structure of a corresponding audited entity type is changed. However, in some cases the change may be insignificant, and it may be more practical to modify existing audit types.

Evolution of audit types may occur in several forms:

1. A new audit type version.

   An application developer runs the generator, which results in:

   • persistent audit-entity type with version n + 1 is generated;
   • persistent audit-prop type with version n + 1 is generated;
   • synthetic audit-entity type may be regenerated;
   • synthetic audit-prop type may be regenerated.

   Afterwards, the standard procedure for adding new persistent entity types should be followed: generate a piece of DDL for new persistent audit types and prepare corresponding SQL scripts.

2. Refactoring of existing audit types.

   An application developer modifies existing audit types to synchronise their structure with that of the audited entity type. Given the hierarchical model of audit-entity types, modification of a property definition in the audit-entity type that declares it propagates to all subsequent type versions (its subclasses). However, one should also be aware of edge cases, some of which are described in the evolutionary scenarios section of this document.

Below, several common evolutionary scenarios are presented with a recommended plan of action for each. Each scenario begins in some evolutionary stage of audited entity Vehicle. Assume that Vehicle already exists, declares no properties, and has a single audit type version (Vehicle_a3t_1). Each scenario contains a brief outcome description that takes into account only changes to persistent audit-entity types.

1. A new property is added.

   Evolution of Vehicle:

   1. Add license : String. (Vehicle_a3t_2)

   Action: generate a new audit-entity type.

   Outcome: Vehicle_a3t_2 is generated, declaring audit-property for license.

2. A property is removed.

   Evolution of Vehicle:

   1. Add license : String. (Vehicle_a3t_2)
   2. Remove license : String.

   Action: generate a new audit-entity type.

   Outcome: Vehicle_a3t_3 is generated, hiding audit-property for license.

3. Property name is changed.

   Evolution of Vehicle:

   1. Add license : String. (Vehicle_a3t_2)
   2. Rename license to permit.

   In this scenario, there are 2 actions:

   • Action A: generate a new audit-entity type.

     Outcome: Vehicle_a3t_3 is generated, hiding audit-property for license and declaring audit-property for permit.

     This scenario can be viewed as removing property license and adding property permit.

   • Action B: refactor existing audit types.

     Outcome: Vehicle_a3t_2 and ReVehicle_a3t are modified by renaming the corresponding properties.

4. Property type is changed.

   Evolution of Vehicle:

   1. Add license : String. (Vehicle_a3t_2)
   2. Change type of license to License.

   • Action: refactor existing audit types.

     Outcome: Vehicle_a3t_2 and ReVehicle_a3t are modified by changing the type of the corresponding properties.

     Note that it would be incorrect to generate a new audit-entity type as that would lead to the existence of two properties with the same name (an invalid class definition).

5. Property title is changed.

   This scenario is similar to changing property name, and it is recommended to refactor existing audit types.

When refactoring audit-entity types, be aware of hidden audit-properties. Specifically, when modifying an audit-property p in audit-entity type version n, be aware of properties that hide p in versions > n.

For example, consider the following evolution of audited entity Vehicle:

1. Add license : String.
2. Remove license.
3. Add license : String.

In the resulting audit-entity types, there are 3 separate properties named license:

1. Vehicle_a3t_1.license.
2. Vehicle_a3t_2.license hides the previous one to effectively remove it.
3. Vehicle_a3t_3.license represents a new property, effectively hiding the previous one.

Any modification to license (e.g., renaming) must be performed in all 3 audit entity-types.

1. Create and configure a source root for generated audit sources.

   The standard location for generated audit sources is a standalone source root in the pojo-bl module of a TG application. A source root that is separate from src/main/java is used for convenience (logical separation of generated and handwritten sources).

   In Maven, extra source roots can be configured with a plugin. This configuration should be picked up automatically by IntelliJ IDEA.

   The POM file in the pojo-bl of a TG application should include the following:

   <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>3.6.0</version>
    <executions>
        <execution>
            <id>audit-sources</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <!-- Location of generated audit sources. -->
                    <source>src/audit/java/</source>
                </sources>
            </configuration>
        </execution>
    </executions>
   </plugin>

2. Set the audit path.

   Application property audit.path defines the location of class files for audit types. Its value depends on the location of generated audit sources. The default is app-pojo-bl/target/classes/. It is used to discover audit types during application startup.

   This property is required and must be specified in all application configurations. For example:

   • application.properties file for the web server.
   • Test configuration (specified by implementations of ua.com.fielden.platform.test.IDomainDrivenTestCaseConfiguration).

   As a rule of thumb, wherever tokens.path is specified, audit.path should be specified as well.

   For deployed applications, the value of this property should be set to the path to a corresponding JAR file, similarly to tokens.path.

TG applications can be launched in one of the auditing modes defined in AuditingMode. The default mode enables auditing.

There are several ways of configuring the auditing mode, ordered from highest priority to lowest:

1. System property audit.mode.
2. Application property audit.mode.

Auditing can be disabled simply by setting the corresponding auditing mode. In this mode, audit records are not created and auditing facilities (all the relevant interfaces) cannot be used (refer to the documentation for more details).

For generation of audit sources, a standalone utility class, akin to GenDdl, should be created in TG applications.

An example of such a class is presented below.

IAuditEntityGenerator is the interface providing source generation capabilities. Refer to its documentation for details.

The generator offers several strategies for choosing the version of generated audit types, which are described in the documentation.

Application developers using this utility should ensure that the destination for generated sources (../ttgams-pojo-bl/src/audit/java in the example) is compatible with the value of the audit.path application property. In other words, sources of audit types should be generated in the same Maven module where their class files are expected to be found at runtime.

package fielden.dev_mod.util;

import org.apache.logging.log4j.Logger;
import ua.com.fielden.platform.audit.IAuditEntityGenerator;

import java.io.FileInputStream;
import java.nio.file.Path;
import java.util.Properties;
import java.util.Set;

import static java.lang.String.format;
import static java.util.stream.Collectors.joining;
import static org.apache.logging.log4j.LogManager.getLogger;
import static ua.com.fielden.platform.audit.IAuditEntityGenerator.VersionStrategy.NEW;

/**
 * This is a helper class that generates audit sources.
 */
public class GenAudit {
    private static final Logger LOGGER = getLogger();

    public static void main(final String[] args) throws Exception {
        LOGGER.info("Initialising ...");
        final String configFileName = args.length == 1 ? args[0] : "src/main/resources/application.properties";
        final var props = new Properties();
        try (final FileInputStream in = new FileInputStream(configFileName)) {
            props.load(in);
        }

        final var config = new AuditGenerationConfig(props);
        LOGGER.info("Generating audit-entities ...");

        // Generate audit types
        final var generator = config.getInstance(IAuditEntityGenerator.class);
        final var result = generator.generate(Set.of(WorkActivity.class),
                                              // Location of audit sources
                                              Path.of("../ttgams-pojo-bl/src/audit/java"),
                                              IAuditEntityGenerator.VersionStrategy.NEW);
        LOGGER.info(format("Generated audit-entities:\n%s", 
                           result.stream().map(p -> p.toAbsolutePath().normalize().toString()).collect(joining("\n"))));
    }
}

The example GenAudit uses a separate configuration -- AuditGenerationConfig, which sould be created alongside (DataPopulationConfig may be used as a reference).

package fielden.dev_mod.util;

import com.google.inject.Injector;
import fielden.config.ApplicationDomain;
import fielden.ioc.EmailSenderTestIocModule;
import fielden.ioc.WebApplicationServerIocModule;
import ua.com.fielden.platform.basic.config.Workflows;
import ua.com.fielden.platform.ioc.ApplicationInjectorFactory;
import ua.com.fielden.platform.ioc.NewUserEmailNotifierTestIocModule;
import ua.com.fielden.platform.test.IDomainDrivenTestCaseConfiguration;

import java.util.Properties;

import static ua.com.fielden.platform.audit.AuditingIocModule.withAuditingMode;
import static ua.com.fielden.platform.audit.AuditingIocModule.AUDIT_MODE;
import static ua.com.fielden.platform.audit.AuditingIocModule.AUDIT_PATH;
import static ua.com.fielden.platform.audit.AuditingMode.GENERATION;

public final class AuditGenerationConfig implements IDomainDrivenTestCaseConfiguration {
    private final Injector injector;

    public AuditGenerationConfig(final Properties props) {
        // Application properties
        // ...
        props.setProperty(AUDIT_PATH, "../ttgams-pojo-bl/target/classes");
        props.setProperty(AUDIT_MODE, GENERATION.name());

        final var module = new WebApplicationServerIocModule(
                new ApplicationDomain(),
                ApplicationDomain.domainTypes(),
                props);
        injector = ...
    }

    @Override
    public <T> T getInstance(final Class<T> type) {
        return injector.getInstance(type);
    }
}

The platform provides dynamically generated Web UI configurations for synthetic audit-entity types.

Entity centres for synthetic audit-entity types are registered as usual.

Such centres can be generated at runtime using IAuditWebUiConfigFactory.

The following example shows one way of registering an entity centre for Vehicle audit.

class WebUiConfig {
  public void initConfiguration() {
    ...
    var builder = configApp();
    ...
    var auditConfigFactory = injector().getInstance(IAuditWebUiConfigFactory.class);
    var vehicleAuditConfig = auditConfigFactory.create(Vehicle.class, builder);
    ...
    configDesktopMainMenu()
      .addModule(APP.title)
          ...
          .menu()
              .addMenuItem(makeMenuItemTitle(vehicleAuditConfig.auditType()))
                  .description(makeMenuItemDesc(vehicleAuditConfig.auditType()))
                  .centre(vehicleAuditConfig.centre())
                  .done()
    ...
  }
}

1. Following the established procedure of creating a compound master/centre, via the TG plugin, create one for the audited entity Vehicle, choosing the corresponding synthetic audit-entity type ReVehicle_a3t as a detail entity with a centre.

2. Adjust the generated code.

   • OpenVehicleMasterActionDao: refactor.

     Adjust the usage of the synthetic audit-entity's meta-model by specifying property auditedEntity.

   • ReVehicle_a3t_Producer: remove.

     Producers for synthetic audit-entities are of no use, as audit records cannot be created by users, and therefore there are no entity masters for audit types.

   • MiVehicleMaster_ReVehicle_a3t: remove.

     All necessary Mi-types for audit types are generated at runtime.

   • Replace the generated definition of an embedded centre by a call to injector.getInstance(IAuditWebUiConfigFactory.class).createEmbeddedCentre(Vehicle.class).

     The platform takes care of embedded centres for synthetic audit-entity types by generating them at runtime.

The resulting Web UI configuration should look roughly as follows:

public class VehicleWebUiConfig {

    private final Injector injector;
    public final EntityMaster<OpenVehicleMasterAction> compoundMaster;

    private VehicleWebUiConfig(final Injector injector, final IWebUiBuilder builder) {
        compoundMaster = CompoundMasterBuilder.<Vehicle, OpenVehicleMasterAction>create(injector, builder)
                // Main master
                .forEntity(OpenVehicleMasterAction.class)
                .withProducer(OpenVehicleMasterActionProducer.class)
                .addMenuItem(VehicleMaster_OpenMain_MenuItem.class)
                    .icon("icons:picture-in-picture")
                    .shortDesc(OpenVehicleMasterAction.MAIN)
                    .longDesc(Vehicle.ENTITY_TITLE + " main")
                    .withView(createVehicleMaster())
                .also()
                // Audit centre
                .addMenuItem(VehicleMaster_OpenReVehicle_a3t_MenuItem.class)
                    .icon("icons:view-module")
                    .shortDesc(OpenVehicleMasterAction.REVEHICLE_A3TS)
                    .longDesc(Vehicle.ENTITY_TITLE + " " + OpenVehicleMasterAction.REVEHICLE_A3TS)
                    .withView(injector.getInstance(IAuditWebUiConfigFactory.class)
                                      .createEmbeddedCentre(Vehicle.class))
                .done();
        builder.register(compoundMaster);
    }

}

Security tokens for audit types are generated at runtime. They are generated only for synthetic audit-entity types.

Application developers can customise the token generation process by implementing a custom ISecurityTokenProvider which should extend SecurityTokenProvider. The latter provides methods that can be overriden for the purpose of customisation.

Auditing of external entity types, such as User from the TG platform, is currently unsupported.