08. Coding Your Service

Now that you've worked out your workflow, your delivery and storage technologies, and how you'll branch management and versioning, you can build a service! You're likely to build three general types of service:

1. REST-based Services - services that expose HTTP/REST endpoints, typically with a JSON workload, for use as app-to-app APIs or as a destination for a modern HTML5 user interface.
2. Asynchronous Event-Based Services - services that react to event received asynchronously, generally over a queue or topic-based broker like Apache Kafka.
3. Batch/Scheduling Based Services - services that perform some action based on a pre-determined schedule. These typically perform some type of bulk operation like moving data to an external system, doing some form of daily data reconcilation, etc.

Coding REST-based Services

This section defines the expected development practices for creating and supporting RESTful services. Different technologies (Java, Python, Node.js, etc) may have different implementations of the same concepts, but the concepts are critical. In some cases, AT&T may have established standards for specific technologies (such as Java). This page does not explain REpresentational State Transfer (REST), or how to code interfaces in specific technologies; Rather, it explains the required implementation to satisfy machine-generated documentation, usability, security, enhancements, and support for the services.

Standards

AT&T has established several standards that you must follow when building and deploying microServices.

The standards that apply to a service’s API include:

- - Implement all java RESTful microService APIs using the JAX-RS V2 standard, JSR-339.
  - All RESTful microService APIs, regardless of technology, must follow theAT&T TSS standard: RESTful Web Service Standard
  - All RESTful microService APIs, regardless of technology, must adhere to the Microsoft RESTful API Guidelines

Documentation

Use the “machine-generated documentation” approach to generate documentation on how to use various services and the operations they expose. We don’t rely on manually-maintained documentation because it’s difficult and expensive to maintain, and not as reliable as the code itself.

Machine-generated documentation can easily determine a lot from the code.
It can identify the:

- - method/operation names
  - return values
  - argument types and order
  - exceptions thrown
  - data structures used for arguments and return types
  - structure of the service itself.

However, machine-generated documentation cannot supply many usability details, like:

- - what happens when a specific value is specified in an argument?
  - what causes the various exceptions that might come from an operation?
  - what does a null for an argument mean?

We can’t get these types of details from structural analysis of the code. Instead, many implementation technologies allow for some form of meta-data or markup (like java annotations) to insert into the code. In the case of JAX-RS, we can use annotations to supply information for code generation as well. Different technologies may use a similar approach.

CDP uses the Swagger product to generate documentation from the source code. Swagger relies on both determining the code structure, and on annotations or metadata inserted into the code. Swagger can generate documentation about the service, its operations and the data types interchanged with them (both arguments and return values). To assist with generating documentation, we can insert annotations into the method AND classes that represent the data types used as arguments and return values. Different technologies have similar capabilities.

{width="1050"}

Swagger can utilize the existing JAX-RS annotations in a java-based service implementation. It can augment these annotations using its own set to supply more information to the user.
It also supports "indirect" arguments, like those you might find in a framework that packages the original arguments together into a containing object (such as Apache Camel).
The example above shows the use of the "ApiImplicitParams" annotation to handle this case. If using JAX-RS natively, use the ApiParam annotation instead.

{width="1050"}

Coding Asynchronous Event-Based Services

Eventual consistency and loose coupling in Domain-Driven Design apply through separate, asynchronous events. Produced and consumed by microServices (or other application components), these events convey meaning about outcomes or conditions that might trigger more processing in other services. They break the synchronous coupling between the action’s initiator that generated the event, and the event’s consumers downstream. Isolated from the originator, the consumers operate on different timelines.

When using event-based processing, it’s important to isolate the event producers from the event consumers. If the producers needed to know where to send events any change, extensions, or additional consumers to the interaction would require extensive code changes. This would result in a many-to-many connectivity problem where all producers and consumers know each other, and it does not scale!

So, in order to gain isolation, they use an intermediate “broker.” All producers and consumers know this software component, but they don’t know each other. Instead, a producer creates an event and presents it to the broker, then the broker distributes it to all interested consumers. The broker manages the acceptance, persistence, and delivery of events to all services that want to receive them.

This type of interchange has a formal messaging model called “Publish-Subscribe” (also called pub/sub). The broker that manages it is called a “Topic.” A service registers its interest in receiving events by “subscribing” to a topic. All events published to that topic are sent to its subscribers. A publisher can publish events into multiple topics, and a subscriber can subscribe to multiple topics.

All events presented to a specific topic are delivered to all of its subscribers. If they need different event “streams” (different types of events to different subscribers), then they can use multiple topics.

Say for example you want to send notifications of orders placed and customers billed. Not all subscribers would want both types of events, so you would create separate topics, segregating events by “type.” Any subscribers who want to know when someone was billed would subscribe to the billing topic, and those interested in when a shipment sent would subscribe to the shipping topic. Put simply, a publisher can publish into more than one topic, and a subscriber can subscribe to more than one topic; so you use different topics to segregate different types of events.

{.unresolved}

Events

The events themselves are data objects that should contain enough information for a subscriber to process it. For example, if an event indicates a created payment, it should also include info about the account, product(s), customer, amount, etc. so that the downstream systems have enough context to understand. We want to minimize the amount of interaction between microServices, so it’s counterproductive for consumers to contact the producer for information they could have sent in the event.

Topics

The publisher should distribute event mechanisms through topics, not queues (Queues use point-to-point messaging and don’t provide for multiple consumers). Topics allow additional consumers to subscribe to them and receive events, allowing enhancement and extension of the enterprise system, without requiring code changes.

For a new service to react to new payments, it just needs to subscribe to the topic. This would not modify or impact the payment system in any way.

DMaaP

Producers and subscribers MUST use the DMaaP (Data Movement as a Platform) product to implement all topics and manage all event flows. DMaaP, a sophisticated messaging system built on top of Kafka, provides data analysis, visibility, and control. It supports “durable” subscribers, meaning events won’t get lost if multiple subscribers register with the topic and one isn’t “up” at the time.

DMaaP contains three significantly different capabilities. The data router facilitates large data set movement, like file transfers, but isn’t useful in event processing. The replication router replicates databases and data stores. The DMaaP Message Router implements the publish/subscribe semantics needed for event processing. This feature should be used.

Refer to DMaaPfor more information.

Coding Batch/Scheduling Based Services

Batch processing involves bulk data processing (such as large amounts of inserts, updates, deletions, extractions, triggering time-initiated operations, etc.). These activities differ from the interactive workload, mostly due to the scale of the data and operations involved. A batch process may often generate as much (or more) access to the data store and processing logic than all of the day’s interactive traffic. Batch processes, triggered mostly by either an event or a time-of-day alarm (like Chron), usually have restricted time windows in which to operate. Each domain’s batch processing requirements will vary widely.

Batch processes exist to perform several common functions:

- extract data for processing elsewhere
- update data to synchronize with an upstream or downstream system
- perform data loads of new data
- trigger additional processing based on time
- perform archival and database cleanup

Batch processes often deal with diverse, homogeneous data sets. They can extract data from multiple different tables or processes, and direct imported/updated data to multiple destinations as needed. For example, an extract from another system may need to process multiple different transaction types, such as updates, inserts, or others that might trigger additional processing. And while batch processes share many similarities, each one likely has differences in its exact nature.

When you refactor a monolithic application to microServices, many of these functions will often move into different ones. If processed in the same way, the batch containing heterogeneous data would need to interact with multiple microServices, causing interaction between services that leads to high overhead and delays. Instead, they should refactor and split up the batch so that any operations the microServices need to perform are isolated to each mS. Using a “splitter” pre-process that extracts different processing types from the batch (and can create multiple homogeneous batch processes) would direct each stream to individual microServices for processing.

"Using a 'splitter' pre-process... would direct each stream to individual microServices for processing."
The following figure shows how to accomplish this:

Extracts can also accomplish the reverse of this. Multiple homogeneous extract streams can combine, creating a heterogeneous data stream for transmittal.

Time-initiated processing can happen using batch as well. For example, someone could schedule a batch “job” at some time of day, and initiate some operation on a microService to process pending data.
The processing should move into the microService, not stay maintained in the “job.” The job only initiates the process.

Patterns

Batch processing has two different patterns: 8A and 8B. Both strive to move all of the batch workload processing into the microService. Additionally, the microService employs batch algorithms that should be written to process large batches of data, and optimized to operate on vectors (sets of data) and access the data store in a batch mode. Using interactive, scalar-based approaches to batch won’t get the high throughput required and will consume too many resources, generating large overhead.

MicroServices that interact with additional microServices during batch processing should be optimized to work with vector data (lists and sets) rather than scalar (single value) requests. This also applies to other record systems that might perform batch processing. In many cases, existing batch processing simply “hits the database” directly. This violates key mS principles when interacting between microServices and must not be allowed. Batch processing within the microService that owns the data can access its own database directly, but not the databases of other microServices or applications.

Pattern 8A

In this pattern, a single microService performs all of the processing of a single batch process. The batch initiation actually delegates the batch processing to the microService. If in the past, a chron “job” executed a batch “job,” that “job” would now delegate processing to the microService. All of the batch job’s original processing logic is now part of the microService.

This pattern ingests the entire batch process into a single microService instance and requires some of the refactoring mentioned earlier. Also, separate microService instances may be reserved and designated only for batch processing, eliminating the impact that interactive traffic would otherwise suffer if they included them

Pattern 8B

Pattern 8B, a more sophisticated pattern, allows for a "batch controller" to integrate into a microService. This batch control function could implement scheduling, splitting, combining, and triggering functions. It could also provide services like chunking where the batch could break up into blocks and distribute across multiple replicas. This pattern requires more effort in design and development, but could be used for very complex batch processing requirements.

Authentication Integration - Camunda and AJSC

Authentication for the AJSC framework is performed using the CADI filter (servlet filter, deployed into the web server container). The CADI filter can be used as a standalone component if the user wishes to perform only authentication. It is also integrated within the AAF filter (also a servlet filter), as a pre-requisite to the authorization process. This wiki will describe details on how authentication is performed for your service using both the CADI and AAF filter.

- - CADI Filter
  - AAF Filter
  - Testing

CADI Filter

To integrate the CADI filter in your AJSC app, follow the steps below.

**Step 1:
**
You need to add the following three dependencies to your project pom.xml.

**pom.xml
**

Please check for the latest non-SNAPSHOT version available of these three components at http://mavencentral.it.att.com:8084/nexus/.

<dependency>
    <groupId>com.att.cadi</groupId>
    <artifactId>cadi-aaf</artifactId>
    <version>1.x.x.x</version>
</dependency>
<dependency>
    <groupId>com.att.cadi</groupId>
    <artifactId>cadi-client</artifactId>
    <version>1.x.x.x</version>
</dependency>
<dependency>
    <groupId>com.att.cadi</groupId>
    <artifactId>cadi-core</artifactId>
   <version>1.x.x.x</version>
</dependency>

**Step 2: **

You need to place the cadi.properties file inside the project directory, where it can be loaded during execution (such as a resource file). A sample cadi.properties file is provided below.

cadi.properties Expand source

Most of the properties above can be configured to your need, such as the log level and aaf_url depending on the aaf environment you want to target. There are five key properties inside cadi.properties-

1. 1. cadi_keyfile
  2. cadi_keystore
  3. cadi_keystore_password
  4. aaf_id
  5. aaf_password

cadi_keystore can be retrieved using the following wget command, and should go inside the project directory, just like the cadi.properties file.

wget:  http://aaf.dev.att.com:8096/truststore2016.jks

The aaf_id and aaf_password are the mechid and its encrypted password, used to pull down the user's permissions in the namespaces of which the mechid is an admin. The cadi_keyfile is the key used to encrypt the password.

To generate that key and to encrypt, you will need cadi-core jar. This should already be available in your local maven repository at com/att/cadi directory. Follow the commands below.

To generate the key called 'keyfile'-

java -jar <path>/cadi-core- 1 .x.x.x.jar keygen keyfile

To encrypt the password using that key-

java -jar cadi-core- 1 .x.x.x.jar digest <MechId Password> <Path of the keyfile>/keyfile

Once encrypted, put the encrypted password, with the phrase "enc:" preceding it , as the value for aaf_password property. For example, if the encrypted password is "abcdef", the value for aaf_password property would be "enc:abcdef". Place the keyfile inside the project directory, and specify its location as the value for cadi_keyfile property.

More information regarding all available configurable properties inside cadi.properties can be found at CADI Configuration Options.

Step 3:

After setting up the configuration for CADI filter, you need to register the CADI filter within your application. To do so, place the following three methods in any class with a @Configuration annotation. The @Configuration annotation defines the class as a Spring configuration class, and the methods are used to define instantiation, configuration, and initialization logic for objects that the Spring IoC container will manage.

Note that you need to identify the location of cadi.properties file inside the propAccess() method above.

For more information regarding CADI filter, please visit its official documentation page.

@Bean
public PropAccess propAccess() {
     PropAccess access =  new PropAccess( new String[]{ "cadi_prop_files=<PATH>/cadi.properties" });
     //PropAccess access = new PropAccess(new String[]{"cadi_prop_files=etc/config/cadi.properties"});
     return access;
}
@Bean (name =  "cadiFilter" )
public Filter cadiFilter()  throws ServletException {
     return new CadiFilter( true ,propAccess());
}
@Bean
public FilterRegistrationBean cadiFilterRegistration()  throws ServletException {
     FilterRegistrationBean registration =  new FilterRegistrationBean();
     registration.setFilter(cadiFilter());
     registration.addUrlPatterns( "/*" );
     registration.setName( "cadiFilter" );
     registration.setOrder( 0 );
     return registration;
}

AAF Filter

As mentioned before, the AAF Filter integrates the CADI Filter to perform authentication before doing authorization. Therefore, using the AAF filter to do authentication is the same as using the CADI filter. However, you will need to configure authorization rules when using the AAF Filter, but not to use the CADI Filter. Look for more information about the AAF Filter on the AJSC wiki page.

To set up the AAF Filter for authentication, follow the steps below.

Step 1: Include the AAF Filter as part of project, either through ECO at project creation time, or by manually including the aaf-filter dependency in the project pom.xml.

Check the latest version on mavencentral.

pom.xml

<dependency>     
     <groupId>com.att.api.framework</groupId>     
     <artifactId>ajsc-aaf-filter</artifactId>     
     <version> 200 .x.x</version>
</dependency>

****Step 2: ****

Follow step 2 of CADI Filter setup from above.

**Step 3: **

You will need to tell the AAF filter where to find the cadi.properties through the JVM argument with the name cadi.properties.location . For example, if you have placed cadi.properties file in directory src/main/resources, you would pass the JVM argument as follows:

-Dcadi.properties.location=src/main/resources/cadi.properties

**Step 4: ** Set up Authorization rules in a property file. Instructions on setting up authorization rules for the AAF filter can be found on the Authorization Integration documentation page.

Testing

Now the service is all set to use CADI filter/AAF Filter for authentication.

- To test through a rest client, you should pass in your aaf credentials ([email protected] and global login password) as Basic Auth Header. If authentication succeeds, you can access the requested resource. If not, the transaction will stop and you get a 401 error.
- To test through a browser, if you're set up to use CSP Global logon and your host domain is AT&T-specific (att.com, sbc.com, etc.), it will redirect you to global logon if you haven't already logged in. If you have, it will authenticate you using the CSP cookie stored inside your browser.

{width="700"}

Still testing through the browser, if you access through a non-AT&T specific host (such as localhost), then a pop up screen will appear where you can enter your aaf credentials.

Authentication Integration - ANSC

In ANSC, AAF is enabled by default. All of the required packages get generated along with the service. All the required code base for AAF is bundled as npm packages.

- - @com.att.ajsc/enforcerjs
  - @com.att.ajsc/aaf
  - @com.att.ajsc/gatekeeperhapijs

The enforcerjs package includes the gatekeeperjs and aaf packages that do the actual authentication/authorization.

See a flow chart of the process on the right. (Click the image to view full-size.)

In ANSC, AAF is enabled by default on two of the sample routes (helloworld.js and login.js) that get generated along with the service.

When you scaffold your project for ANSC, you need to look for two files that use Basic Auth and CSP.
By default these routes come enabled to use CSP or Basic Auth

In routes/helloworld.js

This route uses the enforce method from enforcerjs package, along with onPreAuth, one of the hapi.js lifecycle hooks. It is called, regardless of authentication iperformance.

In routes/login.js

This route uses aafRunner function in the post route for login, required from

npm package @com.att.ajsc/aaf.

Disable CSP and use Basic Auth

To disable CSP, just comment out config.ext.onPreAuth, which uses the enforce function.

Commenting this out will no longer make use of the enforce method, which checks for the CSP cookies that set when you use global logon, and checks if you're using a registered domain name (example att.com).

How to Use Basic Auth With JWT Token

In ANSC, the JWT token strategy applies for basic authentication.

To use it in config.auth, change 'false' to 'token' in strings. This will tell hapi to use the token strategy.

In order to use this, you will either:

- - Hit the /login route from the browser
  - POST to it with authorization header set in postman to Basic Auth and pass in your username and global logon password.

If authenticated, you'll get a token in the header that you can then pass as authorization and value of token.

This will only apply to routes that have 'token' set in config.auth in the routes.
If config.auth is set to false for a given route, then that route will have one authentication check.

If you set 'token' for a given route and pass the token's authorization and value in the header during login, the token strategy will verify it.
If valid you can access the route.

Authorization Integration - AJSC

The AAF servlet filter performs authentication for the AJSC framework using the CADI filter. It accesses the user’s permissions in the namespaces for the mechid, specified in the cadi.properties. This mechid must be a namespace administrator to perform authorization. Follow the steps below to set up authorization rules for your AJSC service using the AAF Filter.

Complete documentation on the AAF filter can be found on the AJSC site, at AAF Filter

Step 1: Perform Step 1 and Step 2 under Authentication Integration for AJSC.

Step 2:

You will need to define a mapping of URLs and permissions for AAF to check in the file named AAFUserRoles.properties.

You can place the file anywhere within your project structure, but you have to define its location as a JVM argument with the name, “aaf.roles.properties.”

For example, if you placed the “AAFUserRoles.properties” file in etc/aaf directory as shown to the right, you would add the following argument in the list of JVM argumemts:

-Daaf.roles.properties=etc/aaf/AAFUserRoles.properties

{height="400"}

AAFUserRoles.properties

The file's contents will be one or more lines in the format described below:

${URL_PATTERN} (${METHODS})=${PERMISSION_1},${PERMISSION_2},...,${PERMISSION_N}

- - URL_PATTERN is the pattern AAF will match against the servlet path of the request.
    - For example, if the request URL is http://zltv9999.vci.att.com:9800/restservices/v1/service/hello, and
    - the base url is: http://zltv9999.vci.att.com:9800
    - the context path is: /restservices/v1
    - the servlet path is: /service/hello, the URL_PATTERN will be matched against /service/hello only.
  - METHODS can be either
    - ALL, or
    - one or more of GET,PUT,POST,DELETE

Each PERMISSION is in the format described below:

{TYPE}|{INSTANCE}|{ACTION}

Type, instance, and action components make up an AAF permission instance. See below for demonstration