Session Recordings - Twingate/kubernetes-access-gateway GitHub Wiki

Use Vector to Sync Recordings to GCS

Overview of Vector

Vector is an enterprise-grade, open-source observability pipeline platform that enables efficient collection, transformation, and routing of observability data. This robust solution provides vendor-independent functionality with extensive configuration options, facilitating seamless integration of logs, metrics, and traces across diverse source systems and destination endpoints.

Using the vector helm chart. To install Vector using Helm, first add the Vector repository and update your local repositories:

Prerequisites

Prior to configuring Vector, ensure the following prerequisites are met:

1. Create a Google Cloud Storage bucket for log storage.
2. Establish a Service Account in Google Cloud Platform with appropriate bucket write permissions. We'll be using this user with Workload Identity to authenticate the Vector pods.
3. Apply the vector.dev/twingate-gateway=true label to the target gateway pods by applying the gateway's podLabels setting in its values.yaml (see podLabels)

Vector Installation and Configuration

Create a configuration file named values.yaml that defines the Vector logs processing pipeline. This configuration implements the following workflow:

• Collects structured JSON logs from Kubernetes pods labeled with vector.dev/twingate-gateway=true
• Processes and filters logs containing session recordings (identified by the asciicast field)
• Exports the filtered recordings as .cast files to the designated Google Cloud Storage bucket

role: Agent
logLevel: "info"
env:
  - name: VECTOR_SELF_NODE_NAME  # TODO: replace this
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

rbac:
  create: true

serviceAccount:
  create: true
  annotations:
    iam.gke.io/gcp-service-account: <SERVICE_ACCOUNT_EMAIL>  # TODO: replace this
  name: vector

customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
    playground: false
  sources:
    gateway_logs:
      type: kubernetes_logs
      extra_label_selector: vector.dev/twingate-gateway=true
    internal_metrics:
      type: internal_metrics
  transforms:
    gateway_json_logs:
      type: remap
      inputs:
        - gateway_logs
      drop_on_abort: true
      metric_tag_values: single
      source: |-
        parsed_json = parse_json!(.message)
        if parsed_json == null {
          abort
        }
        if parsed_json.logger != "gateway.audit" {
          abort
        }
        . = parsed_json
      timezone: local
    session_logging:
      type: remap
      inputs:
        - gateway_json_logs
      drop_on_abort: true
      metric_tag_values: single
      source: |-
        if .asciicast == null {
          abort
        }
        parsed_ts = parse_timestamp!(.ts, "%Y-%m-%dT%H:%M:%S%.3fZ")
        . = {
          "filename_ts_part": format_timestamp!(parsed_ts, "%Y%m%d%H%M%S"),
          "user_id": replace!(.user.username, r'@|\.', "_"),
          "conn_id": .conn_id,
          "message": .asciicast,
          "asciicast_sequence_num": .asciicast_sequence_num
        }
      timezone: local
  sinks:
    gcs:
      type: gcp_cloud_storage
      inputs:
        - session_logging
      bucket: YOU_GCS_BUCKET_NAME
      key_prefix: sessionrecordings/{{"{{"}} .user_id {{"}}"}}_{{"{{"}} .filename_ts_part {{"}}"}}_{{"{{"}} .asciicast_sequence_num {{"}}"}}
      filename_extension: cast
      filename_append_uuid: false
      compression: none
      encoding:
        codec: text
      batch:
        max_events: 1
        timeout_secs: 1

Add the vector helm repository:

helm repo add vector https://helm.vector.dev
helm repo update

Then install Vector in a dedicated namespace:

helm install vector vector/vector -n vector --create-namespace -f values.yaml