Gateway Custom Resource - CAAPIM/layer7-operator GitHub Wiki

Gateway Custom Resource

This section contains more contextual information about the Gateway Custom Resource.

API Docs

These API Docs are autogenerated using Layer7 Operator's CRDs using crdoc

Index

This section is complimentary to the API Docs above providing a comprehensive view of configuration options.

Gateway App Level Configuration

This configuration is comprised of common gateway configuration. It is closely aligned with the Gateway Helm Chart to make transitioning between the two more seamless.

License

A Gateway license corresponding to the major version of the Gateway that you're using is required. You can request a license via support.broadcom.com.

Create a secret with your Gateway v11.x license

This can be achieved using kubectl

Linux/MacOS ``` kubectl create secret generic gateway-license --from-file=license.xml=./path/to/license.xml ```
Windows 
kubectl create secret generic gateway-license --from-file=license.xml=path\to\license.xml

Using Kustomize is a better option for idempotence. It will also make creating a CI/CD deployment pipeline significantly easier.

Kustomize is built directly into kubectl. Kustomize is used throughout the examples in the Layer7 Operator, you can find a kustomization.yaml here.

Update: Gateway v11.1.1 now supports a file mount for Gateway Configuration. This uses node.properties, the same mechanism available on the Appliance/Software Gateway. Read more here

Back to Index

Management Configuration

These are the Configuration items and utilities that fall under management

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      secretName: gateway-secret
      # Management port requires a separate service.
      service:
        enabled: false
        annotations: {}
        type: ClusterIP
        ports:
        - name: management
          port: 9443
          targetPort: 9443
          protocol: TCP   
      restman:
        enabled: false
      graphman:
        enabled: true
        initContainerImage: docker.io/caapim/graphman-static-init:1.0.3
      cluster:
        hostname: gateway.brcmlabs.com
      database:
        enabled: false # this runs the gateway in dbbacked/ephemeral mode
      #  jdbcUrl: "jdbc:mysql://cluster1-haproxy.pxc.svc.cluster.local:3306/ssg"

The following management topics are expanded in more detail below

Back to Index

Admin User Credentials

We strongly recommend creating a Kubernetes Secret with your Gateway Credentials. The default uses the following key/value pairs.

It should contain the following, this takes precendence over any other usernames/passwords specified in the management section

  • SSG_ADMIN_USERNAME
  • SSG_ADMIN_PASSWORD
  • SSG_CLUSTER_PASSPHRASE optionally for mysql backed gateway clusters
  • SSG_DATABASE_USER
  • SSG_DATABASE_PASSWORD

Config example using an existing Kubernetes secret

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      secretName: gateway-secret

The example gateways here have more context on hardcoding plaintext credentials for development purposes.

Back to Index

Diskless Configuration

Refer to TechDocs for more info. Running without Diskless config is supported from Gateway v11.1.1 onwards.

DISKLESS_CONFIG is a new environment variable that was introduced in Gateway v11.1.1, that allows switching between configuration sources.

This is exposed in the Gateway Custom Resource via spec.app.management.disklessConfig

  • disklessConfig.disabled: false is the default
    • Default, No changes.
apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      secretName: gateway-secret
      disklessConfig:
        disabled: false
        csi: {}
  • disklessConfig.disabled: true
    • The Gateway will be read its configuration from node.properties which is mounted to the container gateway.
      • This facilitates the use of the secret store csi driver to mount configuration.
      • Creates a secret with node.properties by default
        • We strongly recommend you create your own node.properties file and make use of disklessConfig.existingSecret configuration.
apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      secretName: gateway-secret
      disklessConfig:
        disabled: true
        csi: {}

Creating a node.properties file

See Techdocs for more info on setting l7.mysql.connection.url. JDBC URLs like the value provided in database.jdbcUrl can be used as the value of l7.mysql.connection.url in node.properties.

Gateway running in Ephemeral Mode (no external MySQL)
  • To run the Gateway in Ephemeral mode, node.db.type=derby needs to be added to node.properties

Example: node.properties with Derby configuration

node.cluster.pass=7layer
admin.user=admin
admin.pass=7layer
node.db.type=derby
node.db.config.main.user=gateway
Using an External MySQL Database
  • Make sure the database configuration matches what is in node.properties

Example: node.properties with MySQL database configuration

node.cluster.pass=7layer
admin.user=admin
admin.pass=7layer
node.db.config.main.user=gateway
node.db.config.main.pass=7layer
l7.mysql.connection.url=jdbc:mysql://myprimaryserver:3306,mysecondaryserver:3306/ssg?failOverReadOnly=false&useSSL=true&requireSSL=true

Back to Index

Cluster Configuration

If using an existing secret for admin user credentials the passphrase field will be omitted.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      # this references an existing secret that contains the mysql username and password.
      secretName: gateway-secret
      cluster:
        hostname: mygw.localdomain
        # password: 7layer

Back to Index

Database Configuration

The Layer7 Operator was designed for the Gateway running in Ephemeral mode.

  • All of our examples are configured without an external MySQL database.
  • Making the move to Ephemeral will give you the best value out of using the Layer7 Operator.
  • Requires familiarity with Graphman and the Graphman Client
  • Gateway v11.1.1 introduces a new variable called liquibaseLogLevel this defaults to "off". The following values are valid
    • "off"
    • "fine"
    • "info"
    • "warning"
    • "severe"

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      # this references an existing secret that contains the mysql username and password.
      secretName: gateway-secret
      database:
        enabled: false # this runs the gateway in dbbacked/ephemeral mode
        liquibaseLogLevel: "off"
      #  jdbcUrl: "jdbc:mysql://mysqlserver:3306/ssg"
      #  username: "gateway"
      #  password: "ACm8BDr3Rfk2Flx9V"

Back to Index

Graphman Configuration

The Layer7 Operator uses Graphman to sync Layer7 Repositories, External Keys/Secrets and various other entities.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      graphman:
        enabled: true
        dynamicSyncPort: 9443
        initContainerImage: docker.io/caapim/graphman-static-init:1.0.3
        initContainerImagePullPolicy: IfNotPresent
        initContainerSecurityContext: {}

Back to Index

Restman Configuration

This configuration bootstraps the Gateway Rest Management API (Restman). If using an external MySQL database where the Gateway database already exists this setting will have no effect and the service will need to be created manually.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      restman:
        enabled: true

Back to Index

Management Service

The management service is only suited to external MySQL backed gateway clusters. The management service provides a stable PM connection, the Layer7 Operator tags one of the Gateway pods in your deployment with a special annotation that is used as a selector for the service. This means this service will only ever route to one gateway pod.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    management:
      service:
        enabled: false
        #annotations:
          # cloud.google.com/load-balancer-type: "Internal"
        type: ClusterIP
        ports:
        - name: management
          port: 9443
          targetPort: 9443
          protocol: TCP

NOTE

Make sure that app.service does not contain the same ports specified here.

Back to Index

Bootstrap Script

The bootstrap script was introduced to reduce reliance on requiring a custom gateway image for custom and modular assertions, scripts and restman/graphman bundles. The script works with the /opt/docker/custom folder. The best way to populate this folder is with an initContainer where files can be copied directly across or dynamically loaded from an external source.

Here is a breakdown of the files that the bootstrap script looks for and where they are placed.

  • Restman Bundles (.bundle)
    • Source /opt/docker/custom/bundle
    • Target /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle
  • Graphman Bundles (.json)
    • Source /opt/docker/custom/bundle
    • Target /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle
  • Custom Assertions (.jar)
    • Source /opt/docker/custom/custom-assertions
    • Target /opt/SecureSpan/Gateway/runtime/modules/lib/
  • Modular Assertions (.aar)
    • Source /opt/docker/custom/modular-assertions
    • Target /opt/SecureSpan/Gateway/runtime/modules/assertions
  • External Libraries (.jar)
    • Source /opt/docker/custom/external-libraries
    • Target /opt/SecureSpan/Gateway/runtime/lib/ext
  • Custom Properties (.properties)
    • Source /opt/docker/custom/custom-properties
    • Target /opt/SecureSpan/Gateway/node/default/etc/conf/
  • Custom Health Check (.sh)
    • Source /opt/docker/custom/health-checks
    • Target /opt/docker/rc.d/diagnostic/health_check
  • Custom Scripts (.sh)
    • Source /opt/docker/scripts
    • Target automatically run

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    bootstrap:
      script:
        enabled: true

Back to Index

Init Containers

The primary use of initContainers in the Layer7 API Gateway is for bootstrapping configuration. That includes files (like custom/modular assertions/large bundles) that are too large to store as Kubernetes Configmaps or Secrets. Refer to these examples for more context on what can be configured and how.

The InitContainer configuration is not constrained to that functionality. You can define your own initContainers that do not use the default bootstrap functionality.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    initContainers:
    - name: gateway-init
      image: docker.io/layer7api/gateway-init:1.0.0
      imagePullPolicy: IfNotPresent
      volumeMounts:
      - name: config-directory
        mountPath: /opt/docker/custom

Back to Index

Gateway Bundles

The Layer7 Operator makes use of Git repositories to create and synchronize Graphman bundles to Gateway Deployments.

In some cases you might want to reference an existing bundle (Kubernetes Configmap or Secret) in Restman or Graphman format outside of the repository functionality. The referenced secret or configmap should contain keys that end with .json for Graphman or .bundle for restman. This kustomization.yaml in the example folder includes simple cluster wide property bundles for Restman and Graphman respectively.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    bundle:
      - type: restman
        source: secret
        name: restman-bootstrap-bundle
      - type: graphman
        source: secret
        name: graphman-bootstrap-bundle

Limitations

  • Configmaps and Secrets are limited to 1mb in size
    • If you have large bundles, using an initContainer is the best approach.
    • Dynamic repositories use the Layer7 Operators ephemeral filesystem so this is also a suitable approach.

Back to Index

Gateway Application Ports

This configuration allows you to customise the Gateway's Application level ports. If nothing is set the Gateway will create a default set of ports

  • 2124 (Internode Communication)
  • 8080 (HTTP)
  • 8443 (HTTPS)
  • 9443 (HTTPS)

The harden flag will do the following if you do not wish to manually configure ports

  • 2124 (Inter-Node Communication) - Not created - if using an existing database 2124 will not be modified
  • 8080 (HTTP) - Not created - if using an existing database 8080 will not be modified
  • 8443 (HTTPS) - Remove Management Features (no Policy Manager Access) - Enables TLSv1.2,TLS1.3 only - Disables insecure Cipher Suites
  • 9443 (HTTPS) - Enables TLSv1.2,TLS1.3 only - Disables insecure Cipher Suites

Note: Gateway v11.1.1 onwards The new refreshOnKeyChanges advanced property will automatically get added to 8443 and 9443 when listenPorts.harden is true. This facilitates automatically refreshing the listen port when the referenced private key changes. If you have listenPorts.custom.enabled and wish to add this to your configuration it can be done by configuring properties in one or more ports. The example below has this configured for port 8443. This is a preview feature

Note: This new feature also includes refreshing the key used in the route via https(s) assertion you will need to add the following global configuration to system.properties to enable this

com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion.refreshOnKeyChanges=true

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    listenPorts:
      harden: false
      custom:
        enabled: true
      ports:
      - name: Default HTTPS (8443)
        port: "8443"
        enabled: true
        protocol: HTTPS
        managementFeatures:
        - Published service message input
        # - Administrative access
        # - Browser-based administration
        # - Built-in services
        properties:
        - name: server
          value: A
        - name: refreshOnKeyChanges
          values: true
        tls:
          enabled: true
          #privateKey: 00000000000000000000000000000002:ssl
          clientAuthentication: Optional
          versions:
          - TLSv1.2
          - TLSv1.3
          useCipherSuitesOrder: true
          cipherSuites:
          - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
          - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
          - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
          - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
          - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
          - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
          - TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
          - TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
          - TLS_DHE_RSA_WITH_AES_256_CBC_SHA
          - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
          - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
          - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
          - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
          - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
          - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
          - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
          - TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
          - TLS_DHE_RSA_WITH_AES_128_CBC_SHA
          - TLS_AES_256_GCM_SHA384
          - TLS_AES_128_GCM_SHA256
      - name: Default HTTPS (9443)
        port: "9443"
        enabled: true
        protocol: HTTPS
        managementFeatures:
        - Published service message input
        - Administrative access
        - Browser-based administration
        - Built-in services
        properties: []
        # - name: server
        #   value: B
        # - name: refreshOnKeyChanges
        #   values: true
        tls:
          enabled: true
        # privateKey: 00000000000000000000000000000002:ssl
          clientAuthentication: Optional
          versions:
          - TLSv1.2
          - TLSv1.3
          useCipherSuitesOrder: true
          cipherSuites:
          - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
          - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
          - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
          - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
          - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
          - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
          - TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
          - TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
          - TLS_DHE_RSA_WITH_AES_256_CBC_SHA
          - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
          - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
          - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
          - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
          - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
          - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
          - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
          - TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
          - TLS_DHE_RSA_WITH_AES_128_CBC_SHA
          - TLS_AES_256_GCM_SHA384
          - TLS_AES_128_GCM_SHA256
      - name: Default HTTP (8080)
        port: "8080"
        enabled: false
        protocol: HTTP
        managementFeatures:
        - Published service message input
        # - Administrative access
        # - Browser-based administration
        # - Built-in services
        tls:
          enabled: false

Note

  • This configuration is static and will not automatically redeploy the Gateway if you make and apply changes
  • You can also use bundles or graphman repositories to apply your own custom port configuration.

Back to Index

Service Configuration

The Gateway Service, also referred to as the standard traffic service.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    service:
      annotations: {}
      type: LoadBalancer
      ports:
      - name: https
        port: 8443
        targetPort: 8443
        protocol: TCP
      - name: management
        port: 9443
        targetPort: 9443
        protocol: TCP

Back to Index

Ingress Configuration

Configure an Ingress Record for your Gateway Deployment.

Note

This implicitly routes to the https port defined in your service configuration.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      tls:
      - hosts: 
        - gateway.brcmlabs.com
        secretName: default
      rules:
      - host: gateway.brcmlabs.com

Back to Index

Java Args

Additional Java Args to pass to the container gateway.

Note

The JVM Heap size can be automatically calculated. This requires that resources requests and limits are set, if they are not set the default value of 2g will be used. As of Gateway v11.1.1 Java min and max heap size can be set, see below for more details.

The default is 75%, this should be set no lower than 50%

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    java:
      jvmHeap:
        calculate: true
      # percentage: 75
        minPercentage: 25
        maxPercentage: 75
      # default: 2g
        minDefault: 1g
        maxDefault: 3g
      extraArgs:
      - -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER
      - -Dcom.l7tech.server.audit.message.saveToInternal=false
      - -Dcom.l7tech.server.audit.admin.saveToInternal=false
      - -Dcom.l7tech.server.audit.system.saveToInternal=false
      - -Dcom.l7tech.server.audit.log.format=json
      - -Djava.util.logging.config.file=/opt/SecureSpan/Gateway/node/default/etc/conf/log-override.properties
      - -Dcom.l7tech.server.pkix.useDefaultTrustAnchors=true 
      - -Dcom.l7tech.security.ssl.hostAllowWildcard=true

Back to Index

System Properties

System properties for the Layer7 Gateway.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    system:
      properties: |-
        # Default Gateway system properties
        # Configuration properties for shared state extensions.
        com.l7tech.server.extension.sharedKeyValueStoreProvider=embeddedhazelcast
        com.l7tech.server.extension.sharedCounterProvider=ssgdb
        com.l7tech.server.extension.sharedClusterInfoProvider=ssgdb
        # By default, FIPS module will block an RSA modulus from being used for encryption if it has been used for
        # signing, or visa-versa. Set true to disable this default behaviour and remain backwards compatible.
        om.l7tech.org.bouncycastle.rsa.allow_multi_use=true
        # Specifies the type of Trust Store (JKS/PKCS12) provided by AdoptOpenJDK that is used by Gateway.
        # Must be set correctly when Gateway is running in FIPS mode. If not specified it will default to PKCS12.
        javax.net.ssl.trustStoreType=jks
        com.l7tech.server.clusterStaleNodeCleanupTimeoutSeconds=86400
        # Additional properties go here

Back to Index

Cluster-Wide Properties

This configuration allows you to declare gateway cluster-wide properties.

Note

  • This configuration is static and will not automatically redeploy the Gateway if you make and apply changes
  • You can also use bundles or graphman repositories to apply your own cluster-wide properties.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    cwp:
      enabled: true
      properties:
        - name: io.httpsHostAllowWildcard
          value: "true"
        - name: log.levels
          value: |
            com.l7tech.level = CONFIG
            com.l7tech.server.policy.variable.ServerVariables.level = SEVERE
            com.l7tech.external.assertions.odata.server.producer.jdbc.GenerateSqlQuery.level = SEVERE
            com.l7tech.server.policy.assertion.ServerSetVariableAssertion.level = SEVERE
            com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion = SEVERE
        - name: audit.setDetailLevel.FINE
          value: 152 7101 7103 9648 9645 7026 7027 4155 150 4716 4114 6306 4100 9655 150 151 11000 4104

Back to Index

Log Override Properties

Override the standard log configuration. This configuration enables logs in json format and works with the following java args above

  • -Dcom.l7tech.server.audit.log.format=json
  • -Djava.util.logging.config.file=/opt/SecureSpan/Gateway/node/default/etc/conf/log-override.properties

If you wish to use Traffic Logger add the following Java Arg

  • -Dcom.l7tech.server.log.console.extraCats=AUDIT,LOG,TRAFFIC

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    log:
      override: true
      properties: |-
        com.l7tech.server.log.ConsoleMessageSink$L7ConsoleHandler.formatter = com.l7tech.util.JsonLogFormatter
        com.l7tech.server.log.ConsoleMessageSink$L7ConsoleHandler.level = CONFIG

Back to Index

Custom Configuration Files

This configuration allows you to mount existing configMap/Secret keys to specific paths on the Gateway without the need for an initContainer or custom/derived image. This is useful for files like ssg.security and or custom assertion properties.

Note

The Kubernetes 1mb configMap/secret size limit applies here.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    customConfig:
      enabled: true
      mounts:
      - name: custom-mount-secret
        mountPath: /opt/docker/secure.properties
        subPath: secure.properties
        ref:
          name: custom-mount-secret
          type: secret
          item: 
            key: secure.properties
            path: secure.properties
      - name: custom-mount-configmap 
        mountPath: /opt/docker/test.properties
        ref:
          item:
            key: test.properties
            path: test.properties
          name: custom-mount-configmap
          type: configmap
        subPath: test.properties

Back to Index

Graceful Termination

During upgrades and other events where Gateway pods are replaced you may have APIs/Services that have long running connections open.

This functionality delays Kubernetes sending a SIGTERM to the container gateway while connections remain open. This works in conjunction with terminationGracePeriodSeconds which should always be higher than preStopScript.timeoutSeconds. If preStopScript.timeoutSeconds is exceeded, the script will exit 0 and normal pod termination will resume.

The preStop script will monitor connections to inbound (not outbound) Gateway Application TCP ports (i.e. inbound listener ports opened by the Gateway Application and not some other process) except those that are explicitly excluded.

The following ports are excluded from monitoring by default.

8777 (Hazelcast) - Embedded Hazelcast.
2124 (Internode-Communication) - not utilised by the Container Gateway.

If there are no open connections, the preStop script will exit immediately ignoring preStopScript.timeoutSeconds to avoid unnecessary resource utilisation (pod stuck in terminating state) during upgrades.

While there aren't any explicit limits on preStopScript.timeoutSeconds and terminationGracePeriodSeconds running these for extended periods of time (i.e. more than 5 minutes) may be less reliable where other Kubernetes processes may remove the pod before terminationGracePeriodSeconds is reached. If you do run services like this we recommend testing before any real life implementation or better, creating a dedicated workload without autoscaling enabled (HPA) where you have more control over when/how pods are replaced.

The graceful termination (preStop script) is disabled by default.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    preStopScript:
      enabled: false
      excludedPorts:
      - 2124
      - 8777
      periodSeconds: 5
      timeoutSeconds: 60

Note

  • If this is enabled, the Layer7 Operator takes over terminationGracePeriodSeconds
  • Lifecycle hooks take precedence

Back to Index

Lifecycle Hooks

Lifecycle describes actions that the management system should take in response to container lifecycle events. For the PostStart and PreStop lifecycle handlers, management of the container blocks until the action is complete, unless the container process fails, in which case the handler is aborted.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    lifecycleHooks:
      postStart:
        exec:
          command:
          - /bin/bash
          - some-script.sh
          - arg1
          - arg2
      preStop:
        exec:
          command:
          - /bin/bash
          - some-script.sh
          - arg1
          - arg2

Note

Lifecycle hooks take precedence over the preStopScript.

Back to Index

Kubernetes Service Account Token

You can optionally mount the Gateway Pods Kubernetes Service Account Token. This is useful if you intend to call the Kubernetes API in Layer7 Policy.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    autoMountServiceAccountToken: true

Back to Index

Automated Features

There are several automations that the Layer7 Operator provides, the primary objective of these automations is to reduce the amount of time and resource required to perform day2 activities from dev to production environments. These features all rely on the Graphman Management API (GraphQL), the Operator has limited support for Restman.

Repository References

Repository References tell the Gateway Controller which Graphman repositories to sync with your Gateway Deployment. It's best to familiarize yourself with the Repository Custom Resource before proceeding.

The configuration below contains 3 repository references, these are part of our default examples

The 'type' defines how a repository is applied to the Container Gateway.

  • Dynamic repositories are applied directly to the Graphman endpoint on the Gateway which does not require a gateway restart
    • The Layer7 Operator will annotate Gateway Pods with the latest commit that has been applied
      • If the commit changes, the latest commit will get applied automatically
  • Static repositories are bootstrapped to the Container Gateway with an initContainer which requires a gateway restart.
    • The Layer7 Operator creates a hash with all of the static repositories and applies that to the initContainer
      • if the hash changes, the Gateway deployment is rolled out automatically and the latest changes will be bootstrapped.

Changes to the external repository are automatically synchronized in one of the above methods. In a production environment we strongly recommend using tags/releases which are immutable as opposed to branches which may change often.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    repositoryReferences:
      - name: l7-gw-myframework
        enabled: true
        type: static
        encryption:
          existingSecret: graphman-encryption-secret
          key: FRAMEWORK_ENCRYPTION_PASSPHRASE
          # passphrase: 7layer
      - name: l7-gw-myapis
        enabled: true
        type: dynamic
        encryption:
          existingSecret: graphman-encryption-secret
          key: APIS_ENCRYPTION_PASSPHRASE
          # passphrase: 7layer
      - name: l7-gw-mysubscriptions
        enabled: true
        type: dynamic
        encryption:
          existingSecret: graphman-encryption-secret
          key: SUBSCRIPTIONS_ENCRYPTION_PASSPHRASE
          # passphrase: 7layer

Back to Index

External Secrets

The Layer7 Operator reconciles existing Kubernetes Secrets with the Gateways Stored Passwords. The easiest way to integrate this functionality with your KMS provider in Kubernetes is using the external secrets operator. The external secrets operator supports these providers

Note

  • An application level integration between your gateway and the KMS provider would be the most secure option given that Kubernetes Secrets are base64 encoded.
  • Secrets can be stored encrypted using OpenSSL
    • echo -n "myothersupersecretvalue" | openssl enc -aes-256-cbc -md sha256 -pass pass:7layer -a
      • use the encryption section below to reference an existing secret or plaintext decryption passphrase

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    externalSecrets:
      - name: mysupersecrets
        enabled: true
        description: top secret
        variableReferencable: true
        encryption:
          passphrase: 7layer
          existingSecret: ""
          key: ""
      - name: private-key-secret
        enabled: true
        description: a private key
        variableReferencable: false

Back to Index

External Keys

External Keys reference existing TLS Secrets in Kubernetes, the Layer7 Operator will attempt to convert this secret to a Graphman bundle that can be applied dynamically keeping any referenced keys up-to-date. You can bring in external secrets using tools like cert-manager. This functionality is currently experimental and carries no support. Please see Gateway Application Ports for more details on how to use these keys on designated listenPorts

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    externalKeys:
      - name: brcmlabs
        alias: brcmlabs
        keyUsageType: SSL
        enabled: true

Back to Index

External Certs

The Layer7 Operator reconciles existing Kubernetes Secrets with the Gateway's Trusted Certificates; keys must end in .crt. If the Secret contains multiple certs, each will be loaded into the Gateway.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    externalCerts:
    - name: multi-cert-secret
      enabled: true
      trustAnchor: false
      revocationCheckPolicyType: USE_DEFAULT
      trustedFor:
      - "SSL"
      - "SIGNING_SERVER_CERTS"
      verifyHostname: false
    - name: single-cert-secret
      enabled: true
      trustAnchor: true
      revocationCheckPolicyType: USE_DEFAULT
      trustedFor:
      - "SSL"
      - "SIGNING_SERVER_CERTS"
      verifyHostname: true

Back to Index

Singleton Extraction

Singleton Extraction works with the Gateway in Ephemeral mode. this enables scheduled tasks that are set to execute on a single node and jms destinations that are outbound to be applied to one ephemeral gateway only. This works inconjunction with repository references and only supports dynamic repository references.

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    singletonExtraction: true

Note

This only works with repositoryReferences that are dynamic.

Back to Index

OAuth Toolkit

The Layer7 Operator can deploy and manage the Layer7 OAuth Toolkit on your Gateway. This uses the same initContainer as the Gateway Helm Chart. This integration was added with the Ephemeral Gateway in mind, while external MySQL backed Gateways will work it would be better to use the Gateway Helm Chart. This functionality is currently experimental and carries no support.

This experimental integration covers the following install modes. There are several features that will come in subsequent releases that will make day2 activities much simpler where the OTK Gateway is fully Operator managed.

See the Gateway examples here

  • Single
  • Dual
    • DMZ
    • Internal

In Dual mode the DMZ and Internal Gateways can be scaled independently.

Config (single gateway) - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    otk:
      enabled: true
      initContainerImage: layer7api/otk-install:latest
      type: single
      database:
        type: mysql
        create: true
        connectionName: OAuth
        auth:
          # A single secret containing all of the values defined here will be created
          # if existingSecret is set the corresponding gateway, readOnly or admin will be omitted from the secret
          # if no values are set, a secret will not be created or referenced and the deployment will be invalidated.
          # existingSecret: otk-db-secret
          gateway:
            username: otk_user
            password: otkUserPass
          readOnly:
           # username: readonly_user
            username: readonly_user
            password: readonly_userPass
          admin:
           # username: admin
            username: admin
            password: adminPass
        properties:
          minimumPoolSize: 3
          maximumPoolSize: 15
        sql:
          databaseName: otk_db
          #jdbcUrl: jdbc:mysql://<host>:<port>/<database>
          jdbcUrl: jdbc:mysql://mysql.brcmlabs.com:3306/otk_db_init
          jdbcDriverClass: com.mysql.cj.jdbc.Driver
          connectionProperties:
            c3p0.maxConnectionAge: "100"
            c3p0.maxIdleTime: "1000"
          manageSchema: true
          databaseWaitTimeout: 60

Back to Index

Deployment Configuration

Annotations

Annotations that apply to resources created by the Gateway Controller

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    annotations:
      someannotationkey: "someannotationvalue"

Back to Index

Pod Annotations

Pod Annotations that apply to the Gateway Deployment.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    podAnnotations:
      sidecar.opentelemetry.io/inject: "ssg"
      instrumentation.opentelemetry.io/inject-java: "true"
      instrumentation.opentelemetry.io/container-names: "gateway"

Back to Index

Labels

Labels that apply to resources created by the Gateway Controller.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    labels:
      somelabelkey: "somelabelvalue"

Back to Index

Pod Labels

Pod Labels that apply to the Gateway Deployment.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    podLabels:
      somepodlabelkey: "somepodlabelvalue"

Back to Index

Service Account

The Service Account to use for the Gateway Deployment. If an existing service account is not provided, one will be created. By default, the Gateway does not require any Kubernetes Roles, if you wish to call the Kubernetes API via the Gateway you will need to assign roles to the Gateway Service Account for the resources the Gateway requires access to. This is not done implicitly.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    serviceAccount:
      name: gateway-service-account
      create: false

Back to Index

Resources

Gateway Pod Resource limits and requests.

Note

If using app.java.jvmHeap.calculate the value is calculated from resources.limits.memory

  • Based on the configuration below the calculation would be 4Gi or 50% of resources.limits.memory

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    resources:
      requests:
        memory: 8Gi
        cpu: 4
      limits:
        memory: 8Gi
        cpu: 4

Back to Index

Autoscaling

A Horizontal Pod Autoscaler can be configured to scale your Gateway deployment.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    autoscaling:
      enabled: true
      hpa:
        minReplicas: 2
        maxReplicas: 3
        metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 60
        behavior:
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
            - type: Pods
              value: 1
              periodSeconds: 60
          scaleUp:
            stabilizationWindowSeconds: 0
            policies:
            - type: Percent
              value: 100
              periodSeconds: 15

Back to Index

Sidecars

Add sidecars to run along side the Gateway container

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    sidecars: []

Back to Index

Custom Hosts

Custom hosts allows you to configure /etc/hosts on the Gateway Container. This is useful if there downstream systems you wish to call that are not registered in DNS.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    customHosts:
      enabled: false
      hostAliases: []
      # - hostnames:
      #   - "dev.ca.com"
      #   - "dev1.ca.com"
      #   ip: "0.0.0.0"
      # - hostnames:
      #   - "example.ca.com"
      #   ip: "0.0.0.0"

Back to Index

Pod Disruption Budget

Pod Disruption Budgets limit the number of concurrent disruptions that your application experiences, allowing for higher availability while permitting the cluster administrator to manage the clusters nodes.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    pdb:
      enabled: false
      minAvailable: 1

Back to Index

Liveness and Readiness Probes

The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs. Read more here

Config - Read more liveness probe readiness probe

This is default configuration used if no liveness/readiness probes are configured

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    livenessProbe:
      exec:
        command:
        - /bin/bash
        - /opt/docker/rc.d/diagnostic/health_check.sh
      initialDelaySeconds: 45
      periodSeconds: 15
      timeoutSeconds: 1
      failureThreshold: 25
      successThreshold: 1
    readinessProbe:
      exec:
        command:
        - /bin/bash
        - /opt/docker/rc.d/diagnostic/health_check.sh
      initialDelaySeconds: 45
      periodSeconds: 15
      timeoutSeconds: 1
      failureThreshold: 25
      successThreshold: 1

Note

Liveness and Readiness Probes are required and cannot be disabled

Back to Index

TopologySpreadConstraints

You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. Read more here

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          foo: bar
    - maxSkew: 1
      topologyKey: node
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          foo: bar

Back to Index

Node Selector

nodeSelector is the simplest recommended form of node selection constraint. You can add the nodeSelector field to your Pod specification and specify the node labels you want the target node to have. Kubernetes only schedules the Pod onto nodes that have each of the labels you specify. Read more here

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    nodeSelector:
      disktype: ssd

Back to Index

Tolerations

Tolerations are applied to pods. Tolerations allow the scheduler to schedule pods with matching taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also evaluates other parameters as part of its function. Read more here

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    tolerations:
    - key: "example-key"
      operator: "Exists"
      effect: "NoSchedule"

Back to Index

Affinity

Affinity and anti-affinity expands the types of constraints you can define. Read more here

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - antarctica-east1
              - antarctica-west1
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: another-node-label-key
              operator: In
              values:
              - another-node-label-value

Back to Index

Security Context

A security context defines privilege and access control settings for a Pod or Container. Read more here

Config - Read more about Pod Security Context here and Container Security Context here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    containerSecurityContext:
      runAsNonRoot: true
      runAsUser: 1001
      capabilities:
        drop:
        - ALL
      allowPrivilegeEscalation: false
    podSecurityContext:
      runAsUser: 1001
      runAsGroup: 1001

Back to Index

Third Party Integrations

Hazelcast

The Layer7 API Gateway uses an embedded Hazelcast for sharing state of rate limits, counters and message replays. The embedded Hazelcast requires a shared policy database (external MySQL), externalising Hazelcast means that this state can be shared with Gateways running in Ephemeral Mode.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    hazelcast:
      external: false
      endpoint: myhazelcast:5601

Back to Index

Redis

This configuration enables integration with Redis Standalone or Sentinel. This is a preview feature on the Layer7 API Gateway.

Config - Read more here

apiVersion: security.brcmlabs.com/v1
kind: Gateway
metadata:
  name: ssg
spec:
  app:
    redis:
      enabled: false
    # existingSecret: ssg-shared-state-client-configuration
    # certs:
    # - enabled: true
    #   secretName: redis-ssl-cert
    #   key: redis.crt
    # - enabled: true
    #   secretName: ssg-shared-state-client-configuration
    #   key: redis123-redis.crt
      default:
        auth:
          enabled: true
          username: admin
          passwordPlaintext: password!£$^%^!!!
        # passwordEncoded: esfksepofksepofksopef12312po3%$(%$($))
        type: standalone
        standalone:
          host: 192.168.1.10
          port: 30036
      # type: sentinel
      # sentinel:
      #   masterSet: mymaster
      #   nodes:
      #   - host: 192.168.1.11
      #     port: 26379
      #   - host: 192.168.1.12
      #     port: 26379
      #   - host: 192.168.1.13
      #     port: 26379
        ssl:
          enabled: true
          existingSecretName: redis-ssl-cert
          existingSecretKey: redis.crt
          verifyPeer: true
          # crt: |+
          #   -----BEGIN CERTIFICATE-----
          #   MIIEPzCCAyegAwIBAgIQeU+e4GXMcNWtPvtYtGsxejANBgkqhkiG9w0BAQsFADAT
          #   MREwDwYDVQQDEwhyZWRpcy1jYTAeFw0yMzEyMDgxNzA0MjRaFw0yNDEyMDcxNzA0
          #   MjRaMBkxFzAVBgNVBAMTDnNlbnRpbmVsLXJlZGlzMIIBIjANBgkqhkiG9w0BAQEF
          #   AAOCAQ8AMIIBCgKCAQEAsWKC5hOfxscAxaqnkz/u4zCImNNhxrBn6mJUIsDQk+0r
          #   FZ46ACK1ViYCwezV/6rKOGm4zQYorTn/lUGXWX23cc4YbKHFPneLPW7ZSCHd65/V
          #   TUPIpxGiwbzrWMq7RWGMPrXlxISIqQsHYH+1To5PvWUn0CQUpcOWTv7NpYql7ugy
          #   GrO77y2PLa0KXsJ/Z2q6VSAoBV51nGmVMOhp0dWuKmHJKYZ4R6Xkj19FrtusZFDh
          #   02mLWIwgZlnAU1egy7LXdkWhkrD1QoVQoY+OvEVzyeAmOz1mxzuwnGlE5LE5B/xa
          #   dhL8wGwRNOxizOrKD46UQN1gOPf3qt4Qt4qAOIUcOwIDAQABo4IBhzCCAYMwDgYD
          #   VR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNV
          #   HRMBAf8EAjAAMB8GA1UdIwQYMBaAFPyHIhBglky8kYnBsdPBhb81SrnFMIIBIQYD
          #   VR0RBIIBGDCCARSCKCouc2VudGluZWwtcmVkaXMucmVkaXMuc3ZjLmNsdXN0ZXIu
          #   bG9jYWyCLXNlbnRpbmVsLXJlZGlzLW1hc3Rlci5yZWRpcy5zdmMuY2x1c3Rlci5s
          #   b2NhbIIvKi5zZW50aW5lbC1yZWRpcy1tYXN0ZXIucmVkaXMuc3ZjLmNsdXN0ZXIu
          #   bG9jYWyCMSouc2VudGluZWwtcmVkaXMtaGVhZGxlc3MucmVkaXMuc3ZjLmNsdXN0
          #   ZXIubG9jYWyCL3NlbnRpbmVsLXJlZGlzLWhlYWRsZXNzLnJlZGlzLnN2Yy5jbHVz
          #   dGVyLmxvY2FsggkxMjcuMC4wLjGCCWxvY2FsaG9zdIIOc2VudGluZWwtcmVkaXMw
          #   DQYJKoZIhvcNAQELBQADggEBAAyYb2dtzhJaVe3H9NaYxBne98wB+TQ0sfQ7S5xd
          #   LOMBOYShutQ4AYTR2z/lBApuM6rGEXJH+kWheNJ3XYwfcazYdUVF4TYbvgXwlAWX
          #   c51Bv2pvN8d3Gm/snSkEHfwXwdtPxr0JXoJ6aFpCzIpkcWYgYdGWL9C+YnYV7CSQ
          #   diDTCBfjJUD6znxaSTHzrJSGxF10dQIZk7WckzKiCmaCKMtx9T0x2YHNQfHVIRUz
          #   l5ANpWC8RhBaq9klUmMDn3Sk9huL+cxSnOK5wAxk8gmZxr0Xupbb6hRG6QJ+Wv4P
          #   1C6GAAf5Obxk6iFTvVGg+dpE5XuSK132MQAVnglXQ/9iV60=
          #   -----END CERTIFICATE-----
      additionalConfigs: []

Additional Gateway Config for Redis

You will need to set the following system-properties to take advantage of the Redis integration

# com.l7tech.server.extension.sharedKeyValueStoreProvider=redis
# com.l7tech.server.extension.sharedCounterProvider=redis
# com.l7tech.server.extension.sharedRateLimiterProvider=redis

Back to Index

Shared State Preview Features

There are two preview features that you may choose to enable with Gateway v11.1.1 onwards.

To use the Apply Distributed Rate Limit Assertion (Preview), set the following in system-properties to redis or externalhazelcast

# com.l7tech.server.extension.sharedRateLimiterProvider=redis

To use the Key Value Storage Assertions (Preview), set sharedKeyValueStoreProvider in system-properties to redis or externalhazelcast

# com.l7tech.external.assertions.keyvaluestore.sharedKeyValueStoreProvider=redis
# com.l7tech.external.assertions.keyvaluestore.storeIdList=GW_STORE_ID

MySQL

The Layer7 Operator was designed with running the Gateway in Ephemeral Mode in mind, while Gateways can still be configured to use an external MySQL database, we strongly recommend considering running in Ephemeral Mode to get the full benefit of using the Layer7 Operator. See management configuration for more details.

Back to Index

⚠️ **GitHub.com Fallback** ⚠️