Usage

Connect Dependency

Configurator supports files in .properties and YAML formats.

It requires the following dependency in your project

<dependency>
    <groupId>com.aegisql</groupId>
    <artifactId>conveyor-configurator</artifactId>
    <version>1.4.3</version>
</dependency>

Initialize conveyors

Conveyor configurator will read properties in the following order:

Provided list of properties or YAML files
OS Environment Variables
JVM system variables

//Optional: change default property prefix
ConveyorConfiguration.PROPERTY_PREFIX = "conv";
//Optional: You can chose other field delimiter than "."
ConveyorConfiguration.PROPERTY_DELIMITER = ".";
//Optional: change Configurator builder timeout to avoid circular dependency dead lock
ConveyorConfiguration.DEFAULT_TIMEOUT_MSEC = 10_000;

//List of config files.
//At least one is mandatory. No default name assumptions
//Absolute or Classpath paths
ConveyorConfiguration.build("classpath:application.yml","/work/more.properties");
...
//Anywhere in your code
Conveyor<Integer,Labels,Product> myConveyor = Conveyor.byName("my_conveyor");

Conveyor Configuration

Conveyor .properties format

Configurator is compatible with Java's Properties format with some extensions.

Preserving order of lines in the properties file
All properties that do not start with conveyor. are ignored.
Same property key can appear more than once and it will be applied each time. Depending on the property nature, it will either override all previous values, or it's value will be added to the pool of similar objects. Example of such behavior can be the accceptedLabels. Each label can be defined by its own line, but, resulting conveyor will be created with all of them.

Format of the key.

Property key is a dot separated string with two or mode fields.

conveyor.[name_part_1.][name_part_2.][...]field_name

First field is always a upper or lower case word "conveyor". This default can be changed by re-assigning value of two constants

ConveyorConfiguration.PROPERTY_PREFIX = "CONVEYOR";
ConveyorConfiguration.PROPERTY_DELIMITER = ".";

Last field is always a name of the conveyor property we want to set.

All other fields are considered as a name of conveyor we create, and they are optional.

So, minimal number of key parts is two. If key has only two parts, those properties are considered as defaults, and will be applied to all conveyors.

All fields between the firs and the last will be combined in one (with dots), and assigned to a conveyor as its name.

YAML format

Organizing keys

#Use the same "dot" delimited format as properties files
conveyor.default_property: value
conveyor.name.property: value
#Or put each part of the key on its "level"
conveyor:
  default_property: value
conveyor.name:
  property: value
conveyor:
  name:
    property: value

Put properties that allow multiple values in YAML arrays

conveyor:
  name:
    acceptLabels: 
      - '"X"'
      - '"Y"'
      - '"Z"'
    staticPart: 
      - >
        label = com.aegisql.conveyor.config.harness.NameLabel.FIRST;
        value = "preffix-";
      - >
        label = com.aegisql.conveyor.config.harness.NameLabel.LAST;
        value = "-suffix";
#If you have only one value though, treat it as scalar
  other_name:
    acceptLabels: '"X"'

Parametrization

# set the timeout property earlier in the file, in OS Environment variables
# or in JVM properties
timeout=10 SECONDS
# use later
# note optional default value separated by the first found ':'
conveyor.test.defaultBuilderTimeout=${timeout:1000}

List of supported Conveyor properties

Check current list

#List of supported properties

#Names for time units must be compatible with Java's TimeUnits enum
#Time unit allows fraction
#This is an example of default parameter
conveyor.idleHeartBeat = 1.5 SECONDS
conveyor.conveyor_name.defaultBuilderTimeout = 1 SECONDS
#If time unit omitted, MILLISECONDS assumed
conveyor.conveyor_name.rejectUnexpireableCartsOlderThan = 10000

#Optional. Default is AssemblingConveyor
#If defined, this parameter overrides earlier defined 'maxQueueSize' or 'enablePriorityQueue'
conveyor.conveyor_name.supplier = new com.aegisql.conveyor.AssemblingConveyor();
#If maxQueueSize is defined, AssemblingConveyor instance with ArrayBlockingQueue will be created
#If defined, this parameter overrides earlier defined custom 'supplier' or 'enablePriorityQueue'
#If your custom conveyor needs queue size initialization, do it with the 'supplier'
conveyor.conveyor_name.maxQueueSize: 10000
#When epriority queue enabled, Assembling conveyor will be created with the PriorityBlockingQueue
#If defined and enabled, overrides  maxQueueSize and supplier
conveyor.conveyor_name.enablePriorityQueue: true

#Desirable parameter.
#Conveyor must have a builder supplier by the time it is used.  
conveyor.conveyor_name.builderSupplier = new com.aegisql.conveyor.config.harness.StringSupplier("conveyor_name");


#Applies ForwardResult result consumer.
#Can be defined several times, each time adding new forwarding consumer
#Third parameter is optional, of not defined, assumed no key conversion is necessary
conveyor.conveyor_name.forward = \
	var label = "Y";\
	var name = "other_conveyor_name";\
	var keyTransformer = function(k){return 'X'+k}; /*optional field*/
#Comma separated list of labels. 
#String must be single or double quoted
#If not quoted, considered as a public constant reference 
conveyor.conveyor_name.acceptLabels = "X_PART","Y_PART"
conveyor.conveyor_name.acceptLabels = org.my.project.Labels.FORWARD

#Static Part property can be defined as many times as needed.
#Each one will be sent to the conveyor
conveyor.conveyor_name.staticPart = \
	label = com.aegisql.conveyor.config.harness.NameLabel.FIRST;\
	value = "preffix-";
conveyor.conveyor_name.staticPart = \
	label = com.aegisql.conveyor.config.harness.NameLabel.LAST;\
	value = "-suffix";

#First Result Consumer MUST be first in chain of Result Consumers
conveyor.conveyor_name.firstResultConsumer = new com.aegisql.conveyor.consumers.result.LogResult()
#Next Result Consumers can be defined several times
conveyor.conveyor_name.nextResultConsumer = com.aegisql.conveyor.config.ConfigUtilsTest.rCounter
#First Scrap Consumer MUST be first in chain of Scrap Consumers
conveyor.conveyor_name.firstScrapConsumer = new com.aegisql.conveyor.consumers.scrap.LogScrap()
#Next Scrap Consumers can be defined several times
conveyor.conveyor_name.nextScrapConsumer = com.aegisql.conveyor.config.ConfigUtilsTest.sCounter
#Lambda
conveyor.conveyor_name.onTimeoutAction = com.aegisql.conveyor.config.ConfigUtilsTest.timeoutAction
#Lambda
conveyor.conveyor_name.defaultCartConsumer = com.aegisql.conveyor.config.ConfigUtilsTest.lvc
#Readiness evaluator is either a Predicate or a BiPredicate, with additional Status parameter
#Both types supported
#But only one will be finally applied
conveyor.conveyor_name.readinessEvaluator = com.aegisql.conveyor.config.ConfigUtilsTest.biPredRE; //BiPredicate
conveyor.conveyor_name.readinessEvaluator = com.aegisql.conveyor.config.ConfigUtilsTest.predRE; //Predicate
#Chainable lambda
conveyor.conveyor_name.addCartBeforePlacementValidator = com.aegisql.conveyor.config.ConfigUtilsTest.cartValidator1
conveyor.conveyor_name.addCartBeforePlacementValidator = com.aegisql.conveyor.config.ConfigUtilsTest.cartValidator2
#Chainable lambda
conveyor.conveyor_name.addBeforeKeyEvictionAction = com.aegisql.conveyor.config.ConfigUtilsTest.beforeEviction
#Chainable lambda
conveyor.conveyor_name.addBeforeKeyReschedulingAction = com.aegisql.conveyor.config.ConfigUtilsTest.beforeReschedule
#Boolean
conveyor.conveyor_name.enablePostponeExpiration = false
#Boolean
conveyor.enablePostponeExpirationOnTimeout = false
#Time Unit
conveyor.conveyor_name.expirationPostponeTime = 1000
#Boolean
conveyor.conveyor_name.autoAcknowledge = true
#Lambda
conveyor.conveyor_name.acknowledgeAction = com.aegisql.conveyor.config.ConfigUtilsTest.acknowledgeAction
#Comma separated list of Status enum values.
conveyor.conveyor_name.autoAcknowledgeOnStatus = READY,TIMED_OUT,CANCELED
#Function
conveyor.conveyor_name.cartPayloadAccessor = com.aegisql.conveyor.config.ConfigUtilsTest.payloadFunction

#Adding Parallelism
#There are two mutually exclusive ways to set parallelism

#If parallel parameter contains a comma separated list of conveyors, 
#then it is treated as a L-Balanced conveyor
conveyor.conveyor_name.parallel = conveyor_name_1, conveyor_name_2
#If parallel parameter contains a number > 1, then it is treated as 
#a K-Balanced conveyor
#Value <=1 cancels parallelism
conveyor.conveyor_name.parallel = 2

#Adding persistence
#In 1.3.10 Persistence must be already available by its name.
#In future releases initialization of Persistence will be supported by configurator.
conveyor.conveyor_name.persistence = com.aegisql.conveyor.persistence.derby.testConv:type=conveyor_name


#Dependency parameter affects order of conveyor initialization
#In this case, 'conveyor_name' will be created after 'conveyor_name_1'
#Can contain a comma separated list; L-Balanced parallelism effectively creates a dependency
conveyor.conveyor_name.dependency = conveyor_name_1
#circle dependency example
#Configurator will be blocked until ConveyorConfigurator.DEFAULT_TIMEOUT_MSEC will happen
#Be advised that default timeout is INFINITY. 
conveyor.a.dependency = b
conveyor.b.dependency = a

#Conveyor with "dot" delimited name
conveyor.conveyor.name.defaultBuilderTimeout = 1 SECONDS

#User defined ResultConsumer for the CONFIGURATOR conveyor
#To get access to the conveyor right after it were created and initialized with provided parameters
#Allows to customize conveyors in a way which is not covered by configurator itself.
conveyor.conveyor_name.postInit = com.aegisql.conveyor.config.ConfigUtilsTest.conveyor_namePostCreation; //custom post-creation processor
#Next param can be applied only as default.
#conveyor.postFailure = <some ref or function>

#Configurator provides limited support for JavaScript Functions
#In place of Conveyor Functionsl interfaces
conveyor.postFailure = function(bin){print('FAILED '+bin.key);}
conveyor.conveyor.name.postInit = function(bin){print('CREATED CONVEYOR '+bin.product.getName());}

#Readiness Tester
#Optional number of times the label must be accepted before ready will fire.
#Default is one
#Can be combined with Readiness Evaluators. 
conveyor.conveyor_name.readyWhenAccepted = 2 com.aegisql.conveyor.config.harness.NameLabel.FIRST,com.aegisql.conveyor.config.harness.NameLabel.LAST
conveyor.conveyor_name.readyWhenAccepted = com.aegisql.conveyor.config.harness.NameLabel.END

Persistence Configuration

Persistence .properties format

Similar to Conveyor .properties format
All properties that do not start with persistence. are ignored.

Format of the persistence key.

Property key is a dot separated string with two or mode fields.

persistence.[type].[schema].[parts].[field[.subField]] = value

First field is always a upper or lower case word "persistence". This default can be changed by re-assigning value of two constants

ConveyorConfiguration.PERSISTENCE_PREFIX = "PERSISTENCE";
ConveyorConfiguration.PROPERTY_DELIMITER = ".";

Second field - type of persistence. Now only those pre-configured types supported

derby
derby-client
derby-memory (1.4.2)
sqlite
mysql
mariadb
postgres

Third field - schema name. In databases that require separate database and schema configuration 'database' must be passes via property 'database'

Fourth field - name of table or collection to store parts

All other fields - name of the property

So, minimal number of key parts is two. If key has only two parts, those properties are considered as defaults, and will be applied to all conveyors.

Yaml Format

Key structure for YAML property keys is similar to the Conveyor YAML keys.

persistence can be added directly to the conveyor.persistence property, like in example below

conveyor:
  conv_name:
    builderSupplier:  new com.aegisql.conveyor.config.harness.StringSupplier("c9-1");
    defaultBuilderTimeout:  1 SECONDS
    firstResultConsumer:  new com.aegisql.conveyor.consumers.result.LogResult()
    firstScrapConsumer:  new com.aegisql.conveyor.consumers.scrap.LogScrap()
    readyWhenAccepted:  
      - com.aegisql.conveyor.config.harness.NameLabel.FIRST
      - com.aegisql.conveyor.config.harness.NameLabel.LAST
    persistence:
      derby.conv_name.parts:
        keyClass: java.lang.Integer
        archiveStrategy:
          path: ./
          maxFileSize: 20KB
          bucketSize: 50
          zip: true

Or it can be defined separately, then Conveyor can refer it by name, which must include [type].[schema].[parts]

conveyor:
  conv_name:
    builderSupplier:  new com.aegisql.conveyor.config.harness.StringSupplier("c9-1");
    defaultBuilderTimeout:  1 SECONDS
    firstResultConsumer:  new com.aegisql.conveyor.consumers.result.LogResult()
    firstScrapConsumer:  new com.aegisql.conveyor.consumers.scrap.LogScrap()
    readyWhenAccepted:  
      - com.aegisql.conveyor.config.harness.NameLabel.FIRST
      - com.aegisql.conveyor.config.harness.NameLabel.LAST
    persistence: derby.conv_name.parts
persistence:
  derby.conv_name.parts:
    keyClass: java.lang.Integer
    archiveStrategy:
      path: ./
      maxFileSize: 20KB
      bucketSize: 50
      zip: true

List of supported Persistence properties

Check current list

#List of supported persistence properties
#Second field - one of:
# derby
# derby-client
# derby-memory
# sqlite
# mysql
# mariadb
# postgres

#Conveyor can refer any persistence by its name
#Like: conveyor.conv_name.persistence = some.persistence.name

#The only mandatory property for JDBC persistence is type of the Conveyor key
#All other parameters are optional 
persistence.derby.test.parts2.keyClass = java.lang.String
persistence.derby.test.parts.keyClass = java.lang.Integer

#Enable/disable auto initialization for the database
#when database is missing, it can be created automatically
persistence.derby.test.autoInit=true

#Database
#Some DB engines require dtabase AND schema parameters
#in example below - 
#database=pg_database
#schema=test
persistence.postgres.test.database=pg_database

#Name of table or collection for complete log
persistence.derby.test.parts.completedLogTable=parts_complete

#Converts labels into storeable format 
persistence.derby.test.parts.labelConverter = com.aegisql.conveyor.config.harness.NameLabel

#Username for DB Network Server
persistence.mysql.test.parts.username = user
#Password for DB Network Server
persistence.mysql.test.parts.password = pass
#DB Server host name 
#localhost is default for derby-client, mysql and postgres
persistence.mysql.port = localhost
#Port for DB Network Server
#Default value depends on concrete DB engine
persistence.derby-client.port = 1527
#Long Id supplier
persistence.derby.test.parts.idSupplier = com.aegisql.conveyor.config.ConveyorConfigurationTest.idSource.getId()

#Encryption parameters
persistence.derby.encryptionAlgorithm = AES
persistence.derby.encryptionTransformation = AES/ECB/PKCS5Padding
persistence.derby.encryptionKeyLength = 16
persistence.derby.test.parts.encryptionSecret = erfq834hfq8723royf87341r

#How many completed parts will be buffered, before removing from storage
persistence.derby.test.parts.maxBatchSize = 100
#Max time completed parts will be buffered, before removing from storage
persistence.derby.test.parts.maxBatchTime = 10 SECONDS
#Exclude some metadata keys and values from persistence
persistence.derby.test.parts.doNotSaveProperties = property1,property2

#Archiving Strategies
#By ArchiveStrategy enum
persistence.derby.test.parts.archiveStrategy = NO_ACTION
persistence.derby.test.parts.archiveStrategy = DELETE
persistence.derby.test.parts.archiveStrategy = SET_ARCHIVED

#Custom Strategy
persistence.derby.test.parts.archiveStrategy = CUSTOM
persistence.derby.test.parts.archiveStrategy.archiver = com.aegisql.conveyor.config.ConveyorConfigurationTest.archiver

#Move to other persistence (derby.test.parts2 defined above)
persistence.derby.test.parts.archiveStrategy = MOVE_TO_PERSISTENCE
persistence.derby.test.parts.archiveStrategy.persistence = derby.test.parts2

#Move to file
persistence.derby.test.parts.archiveStrategy = MOVE_TO_FILE
persistence.derby.test.parts.archiveStrategy.path = /test/
persistence.derby.test.parts.archiveStrategy.moveTo = /backup/
persistence.derby.test.parts.archiveStrategy.maxFileSize = 10MB
persistence.derby.test.parts.archiveStrategy.bucketSize = 1000
persistence.derby.test.parts.archiveStrategy.zip = true

#Add Binary converters
persistence.derby.test.parts.addBinaryConverter = java.lang.Long, new com.aegisql.conveyor.persistence.converters.LongToBytesConverter();
persistence.derby.test.parts.addBinaryConverter = java.lang.Boolean, new com.aegisql.conveyor.persistence.converters.BooleanToBytesConverter();
persistence.derby.test.parts.addBinaryConverter = com.aegisql.conveyor.config.harness.NameLabel.FIRST, new com.aegisql.conveyor.persistence.converters.BooleanToBytesConverter();

#Add Label Converters
persistence.derby.test.parts.labelConverter = new com.aegisql.conveyor.persistence.jdbc.converters.EnumConverter(com.aegisql.conveyor.config.harness.NameLabel.class);
persistence.derby.test.parts.labelConverter = com.aegisql.conveyor.config.harness.NameLabel

#minCompactSize > 0 enables auto compaction
#when number of stored parts reaches minCompactSize compact() method is called for corresponding build ID
persistence.derby.test.parts.minCompactSize = 10000
#One of the RestoreOrder enum constants
#Supports no ordering, order by ID ascending, order by priority descending and ID ascending.
persistence.derby.test.parts.restoreOrder = BY_ID
#Additional field for test.parts table; default accesstor will lookup field in the cart properties
persistence.derby.test.parts.addField = java.lang.Long, TRANSACTION_ID
#Additional field for test.parts table; accessor can be specified as a full path to the Function static instance or JavaScript function
persistence.derby.test.parts.addField = java.lang.Long, TRANSACTION_ID, function(cart){return cart.getLabel();}
#when initialize parts table, will create unique index on provided colum names
persistence.derby.test.parts.addUniqueFields = CART_KEY, CART_LABEL

1. Configuration Files - aegisql/conveyor GitHub Wiki

Usage

Connect Dependency

Initialize conveyors

Conveyor Configuration

Conveyor .properties format

Format of the key.

YAML format

Parametrization

List of supported Conveyor properties

Persistence Configuration

Persistence .properties format

Format of the persistence key.

Yaml Format

List of supported Persistence properties

⚠️ GitHub.com Fallback ⚠️

1. Configuration Files - aegisql/conveyor GitHub Wiki

Usage

Connect Dependency

Initialize conveyors

Conveyor Configuration

Conveyor .properties format

Format of the key.

YAML format

Parametrization

List of supported Conveyor properties

Persistence Configuration

Persistence .properties format

Format of the persistence key.

Yaml Format

List of supported Persistence properties

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️