Building Datadog Agent - linux-on-ibm-z/docs GitHub Wiki

Building Datadog Agent

This file has been modified with the assistance of AI Tool [claude 4.6]

The instructions provided below specify the steps to build Datadog Agent version 7.79.2 on Linux on IBM Z for the following distributions:

  • RHEL (9.6, 9.8, 10.0, 10.2)
  • SLES 16
  • Ubuntu (22.04, 24.04)

RHEL, Ubuntu, and SLES were validated on bare-metal/LPAR installs with the kernels listed above.

1. Building Datadog Agent

1.1 Build using script

If you want to build Datadog Agent using manual steps, go to Step 1.2.

Use the following commands to build Datadog Agent using the build script.

export SOURCE_ROOT=/<source_root>/
cd $SOURCE_ROOT

# Use wget to download build script. Please make sure wget is installed
wget -q 'https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/build_datadog-agent.sh'

# Build Datadog Agent
bash build_datadog-agent.sh -y   # Provide -t option for executing build with tests

In case of error, check logs for more details or go to Step 1.2 to follow manual build steps.

1.2 Install dependencies

export SOURCE_ROOT=/<source_root>/
  • RHEL (9.6, 9.8, 10.0, 10.2)
sudo yum install -y gcc gcc-c++ make wget curl tar git patch diffutils file
  • SLES 16
sudo zypper install -y gcc gcc-c++ make wget curl tar git patch file
  • Ubuntu (22.04, 24.04)
sudo apt-get update
sudo apt-get install -y gcc g++ make wget curl ca-certificates tar git patch file

1.3 Install Go 1.26.3

cd $SOURCE_ROOT
wget -q https://go.dev/dl/go1.26.3.linux-s390x.tar.gz
sudo tar -C /usr/local -xzf go1.26.3.linux-s390x.tar.gz
rm -f go1.26.3.linux-s390x.tar.gz
export PATH=/usr/local/go/bin:$PATH
go version

1.4 Download source and apply patches

cd $SOURCE_ROOT
git clone --depth 1 -b 7.79.2 https://github.com/DataDog/datadog-agent.git datadog-agent-7.79.2
cd datadog-agent-7.79.2

Apply the s390x patches. Use the provided apply-patch.sh script, which auto-detects each patch's strip level, places new source files before the diffs that modify them, and skips patches already present in the shipped source (019 dedup, 042 redundant with 023):

apply-patch.sh takes the patch location as an https:// URL and downloads each patch itself, so nothing needs to be checked out locally:

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/apply-patch.sh
bash apply-patch.sh https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/patch

Expect 31 applied, 2 SKIP, 0 FAIL and no *.rej files. Fuzz on a few hunks is expected against the 7.79.2 tag. Confirm patch 044 landed — the test suite depends on it:

head -8 pkg/ebpf/ebpftest/log.go | grep '//go:build test'

Alternatively, apply patches manually from the source root. Note the strip level differs by patch: datadog-agent.patch, 001–030, 036, 044, and 045 use -p1; 031–035 and 037–043 use -p4 (they carry absolute source paths):

# Phase 1: Core Agent
patch -p1 < <path-to>/patch/datadog-agent.patch

# Phase 2: eBPF Security Probes — diff patches
patch -p1 < <path-to>/patch/001-syscall-macro-s390x.patch
patch -p1 < <path-to>/patch/002-bpf-metadata-s390x.patch
patch -p1 < <path-to>/patch/003-bpf-tracing-typo.patch
patch -p1 < <path-to>/patch/016-syscalls-unsupported-build-constraint.patch
patch -p1 < <path-to>/patch/017-mount-linux-statfs-type-cast.patch
patch -p1 < <path-to>/patch/019-syscalls-string-dedup.patch
patch -p1 < <path-to>/patch/020-probes-s390x-syscall-prefix.patch
patch -p1 < <path-to>/patch/021-rc-btf-s390x.patch

# Phase 2: eBPF Security Probes — new source files
cp <path-to>/patch/008-ptracer-syscalls-s390x.go    pkg/security/ptracer/syscalls_s390x.go
cp <path-to>/patch/009-tests-syscalls-s390x.go      pkg/security/tests/syscalls_s390x.go
cp <path-to>/patch/010-kernel-bugs-s390x.go         pkg/ebpf/kernelbugs/kernel_bugs_s390x.go
cp <path-to>/patch/011-consts-linux-s390x.go        pkg/security/secl/model/consts_linux_s390x.go
cp <path-to>/patch/012-syscalls-linux-s390x.go      pkg/security/secl/model/syscalls_linux_s390x.go
cp <path-to>/patch/013-syscalls-string-linux-s390x.go pkg/security/secl/model/syscalls_string_linux_s390x.go
cp <path-to>/patch/014-btfhub-constants-s390x.json  pkg/security/probe/constantfetch/btfhub/constants_s390x.json
cp <path-to>/patch/015-btfhub-s390x.go              pkg/security/probe/constantfetch/btfhub_s390x.go

# Phase 3: Network Monitoring eBPF
patch -p1 < <path-to>/patch/022-usm-get-tls-base-s390x.patch
patch -p1 < <path-to>/patch/023-go-tls-location-s390x.patch

# Phase 4: Dynamic Instrumentation eBPF
patch -p1 < <path-to>/patch/024-bpf-tracing-custom-s390x.patch
patch -p1 < <path-to>/patch/025-cfa-s390x.patch
patch -p1 < <path-to>/patch/026-event-s390x.patch
patch -p1 < <path-to>/patch/027-abi-s390x.patch
patch -p1 < <path-to>/patch/028-irgen-s390x.patch
patch -p1 < <path-to>/patch/029-bininspect-s390x.patch
patch -p1 < <path-to>/patch/030-safeelf-s390x.patch

# Phase 5: Cross-Architecture Bug Fix
patch -p1 < <path-to>/patch/045-dyninst-decode-typeid-endianness.patch

# Phase 6: Dynamic Instrumentation Big-Endian Byte Order (031-035, 037-043 use -p4)
patch -p4 < <path-to>/patch/031-dyninst-object-big-endian.patch
patch -p4 < <path-to>/patch/032-dyninst-gotype-byteorder.patch
patch -p4 < <path-to>/patch/033-dyninst-gosym-byteorder.patch
patch -p4 < <path-to>/patch/034-dyninst-decode-nativeendian.patch
patch -p4 < <path-to>/patch/035-dyninst-dwarf-loclist-byteorder.patch
patch -p1 < <path-to>/patch/036-dyninst-irgen-loclist-bo.patch

# Phase 7: eBPF Telemetry
patch -p4 < <path-to>/patch/037-ebpf-probes-test-s390x.patch
patch -p4 < <path-to>/patch/038-bpf-telemetry-bounded-negation.patch
patch -p4 < <path-to>/patch/039-error-telemetry-probe-read-kernel.patch
patch -p4 < <path-to>/patch/040-errors-telemetry-dword-atomic.patch
patch -p4 < <path-to>/patch/041-errors-telemetry-test-s390x.patch

# Phase 8: Additional Fixes
patch -p4 < <path-to>/patch/042-go-tls-location-core-s390x.patch
patch -p4 < <path-to>/patch/043-sample-yaml-s390x-tags.patch

# Phase 9: B1 test-tag alignment (gate ebpftest/dyninsttest behind the `test`
# build tag so they match the test-only log.Default/slog helper APIs)
patch -p1 < <path-to>/patch/044-b1-ebpftest-dyninsttest-test-tag.patch

Patch summary by phase:

Phase Description Patches Key Changes
1 Core Agent 4 s390x arch definition (ARCH_S390X), container platform mapping, TestPlatform fix (redhat-release as file), runtime.Version() sanitization for vendor Go
2 eBPF Security Probes 17 Syscall macros (GPR2-GPR7), ptracer register mapping, BTFHub constants, full syscall table (350+ entries), build constraint fixes, Statfs_t.Type uint32 cast, SYS_ACCEPT4 (no standalone accept on s390x), syscall prefix __s390x_sys_, BTF remote config
3 Network Monitoring eBPF 2 TLS base pointer via access register pair (acrs[0:1]), Go TLS register access via gprs[] loop, read_register_indirect(), read_stack() via PT_REGS_SP
4 Dynamic Instrumentation eBPF 7 DWARF register macros (GPR0-15, BP=R11, SP=R15, PC=psw.addr), CFA calculation (frameless=SP, framed=SP+8), goroutine pointer via R13, Go ABI config (stack-based, maxIntRegs=0), ELF machine type EM_S390 mapping
5 Cross-Architecture Bug Fix 1 Fix TypeID collision in TestFormatStruct — latent endianness bug where duplicate TypeID 0 caused int32 decoder to overwrite int64 decoder (masked on little-endian, exposed on big-endian)
6 Dynamic Instrumentation Big-Endian 6 Replace ~40 hardcoded binary.LittleEndian calls with proper byte-order threading across object, gotype, gosym, decode, DWARF loclist, and irgen
7 eBPF Telemetry 5 Fix clang optimization breaking errno negation pattern, bpf_probe_read_kernel for kernel >= 5.x, 64-bit atomics (asm.DWord), test adjustments
8 Additional Fixes 2 Go TLS location core register reading via ctx->gprs[], sample YAML ConditionVariableUnavailable tags for return probes
9 B1 test-tag alignment 1 Gate ebpftest (log.go, fail_log.go) and dyninsttest (util.go) behind the test build tag so they match the test-only log.Default/slog helper APIs in pkg/util/log (patch 044)

Golden-file snapshots for dyninst tests (compiler, gotype, irgen) are also included for s390x.

1.5 Build the agent

export CGO_LDFLAGS_ALLOW="-Wl,--wrap=.*"
export CGO_CFLAGS="-Werror -Wno-deprecated-declarations"
export GO111MODULE=on

BUILD_TAGS="bundle_agent cel consul containerd cri datadog.no_waf ec2 etcd"
BUILD_TAGS="$BUILD_TAGS fargateprocess grpcnotrace jmx kubeapiserver kubelet"
BUILD_TAGS="$BUILD_TAGS linux_bpf"
BUILD_TAGS="$BUILD_TAGS ncm netcgo no_dynamic_plugins oracle orchestrator"
BUILD_TAGS="$BUILD_TAGS otlp retrynotrace sharedlibrarycheck systemprobechecks"
BUILD_TAGS="$BUILD_TAGS zk zlib zstd"

COMMIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")

LDFLAGS="-X github.com/DataDog/datadog-agent/pkg/version.Commit=${COMMIT_SHA}"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentVersion=7.79.2"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentPayloadVersion=v5.0.193"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentPackageVersion=7.79.2"

go build \
    -mod=readonly \
    -tags "$BUILD_TAGS" \
    -o ./bin/agent/agent \
    -ldflags="$LDFLAGS" \
    -trimpath \
    github.com/DataDog/datadog-agent/cmd/agent

To build without eBPF support, remove linux_bpf from BUILD_TAGS.

1.6 Verify the build (Optional)

./bin/agent/agent version

Output should be similar to:

Agent 7.79.2 - Commit: <sha> - Serialization version: v5.0.193 - Go version: go1.26.3
file ./bin/agent/agent

Output should be similar to:

ELF 64-bit MSB executable, IBM S/390, version 1 (SYSV), dynamically linked

MSB is the check that matters — it confirms a big-endian s390x binary.

1.7 Install (Optional)

sudo cp ./bin/agent/agent /usr/local/bin/datadog-agent

Note::_The following build tags are excluded because they require x86/arm64-only native code, container runtime dependencies, or CGO/rtloader: systemd, docker, jetson, nvml, pcap, crio, podman, trivy, python.

2. Execute Test Suite (Optional)

2.1 Using the shipped runner (recommended)

scripts/7.79.2/test/run-tests-s390x.sh runs the validation suite with the correct build tags, ldflags and package groups, and tallies the results. Two modes:

cd /<source_root>/datadog-agent-7.79.2

# the packages the port modifies -- ~1731 cases, ~1 minute
bash <path-to>/scripts/7.79.2/test/run-tests-s390x.sh --targeted . ./testlogs

# every package that compiles for s390x -- ~19268 cases, ~90 minutes
bash <path-to>/scripts/7.79.2/test/run-tests-s390x.sh --full . ./testlogs

Results land as go test -json in the output directory. Tally them with:

python3 -I <path-to>/scripts/7.79.2/test/tally_tests.py ./testlogs/*.json

# per-failure assertion text
python3 -I <path-to>/scripts/7.79.2/test/tally_tests.py ./testlogs/*.json --detail

2.2 Test environment requirements

These are test dependencies, not build dependencies. Each one, if missing, produces failures that look like port defects but are not. The runner preflights for all of them and warns before starting.

Requirement Without it
xz pkg/ebpf TestRemoteConfigBTF* fail — the helper shells out to tar -cvJ
tzdata pkg/collector/runner/expvars fails with unknown time zone America/New_York
iproute2, conntrack-tools pkg/network/netlink conntrack tests fail — they shell out to ip netns add and conntrack -F
RLIMIT_MEMLOCK unlimited every eBPF map test fails with failed to set memlock rlimit: operation not permitted

When running in a container, that last one means:

podman run --privileged --ulimit memlock=-1 -v /sys/kernel/debug:/sys/kernel/debug ...

2.3 Targeted results (the ported code)

34 packages, three groups, run on RHEL 10.2 (s390x, kernel 6.12, Go 1.26.3):

Group PASS FAIL SKIP
Correctness (20 package patterns) 1589 1 1
B1-unblocked (4 pkgs, un-buildable before patch 044) 58 19 1
Fixture-dependent (6 dyninst pkgs) 29 33 0
Total 1676 53 2

All 53 failures map to known infrastructure blockers: 33 dyninst test fixtures, 18 CO-RE .o assets, 1 clang-bpf, 1 the ebpf-manager arch="x64" hardcode. Zero s390x logic failures.

Reproduced across RHEL 9.6, RHEL 10.2, Ubuntu 24.04, SLES 16.0, Rocky Linux 9.8 / 10.2, Debian 12 / 13 and Fedora 42. The same suite on x86_64 (Ubuntu 24.04, UBI 10.2, SLES BCI 16.0 containers) gives 1650 PASS / 61 FAIL, with the dyninst-fixture, CO-RE and clang-bpf failure counts byte-identical to s390x — confirming those blockers are architecture-independent.

2.4 Full-repository results

Every package under pkg/, comp/ and cmd/ that compiles for s390x — run with --full on RHEL 10.2:

Group PASS FAIL SKIP
pkg/ (6 chunks) 14803 134 27
comp/ 3627 2 1
cmd/ 671 1 2
Total 19101 137 30

99.1% pass. Of the 137 failures, 87 are the same B5/B6/B7 blockers as above, 17 are parent aggregators of those, and 31 are container-environment artifacts (missing ip/conntrack, an ungenerated system-probe-lite helper, docker runtime, host networking).

Exactly two genuine s390x findings in the whole tree:

  1. pkg/network/netlink :: TestDecodeAndReleaseEvent — a non-portable test fixture, not a product defect. The test hardcodes a netlink message captured on a little-endian host and carries no architecture gate. Netlink attribute headers are native-endian by kernel convention and the decoder correctly uses nlenc.Uint16, so on s390x the fixture is misread: header bytes 0x34 0x00 0x01 0x80 parse as length=52 type=0x8001 on little-endian but length=13312 type=0x0180 on big-endian, overrunning the buffer so zero attributes decode. The product code is correct on s390x. Upstream fix: build the fixture with binary.NativeEndian, or gate the test to little-endian. (Secondary: decoding_test.go:34 uses assert.Len where require.Len is needed, so it panics instead of failing cleanly.)

  2. pkg/gohai/cpu :: TestCollectCPU — a genuine, low-impact gap. Collection of Mhz, Family, Model, Stepping and the cache-size fields returns an error that is not utils.ErrNotCollectable, because s390x /proc/cpuinfo has a different layout. Affects host metadata only.

2.5 Coverage

Packages Test functions
Repository total 1117 11599
Compile for s390x 686 6355
Exercised by --targeted 34 434
Exercised by --full 686 6355

94 packages do not build for s390x. Almost all fail on undefined: runtime.<Asset> — the generated eBPF runtime-compilation assets (RuntimeSecurity, OomKill, TcpQueueLength, SharedLibraries, Gpu), which require a BPF-target clang for s390x. The exception is opentelemetry-ebpf-profiler, which has no s390x support at all (undefined: pfelf.CurrentMachine).

3. Build a Container Image (Optional)

scripts/7.79.2/image/ builds a Datadog Agent container image for s390x without omnibus. Built and verified on real IBM Z hardware (RHEL 10.2, kernel 6.12, native s390x — no emulation): Agent 7.79.2, Build arch: s390x, Go 1.26.3, 16 healthy components, agent/process/trace supervised by s6.

3.1 Why not omnibus

The upstream Dockerfiles/agent/Dockerfile does not compile the Agent — it globs for datadog-agent*-$TARGETARCH.tar.xz and untars it. Omnibus is only one way to produce that tarball, and its chain (embedded CPython, ~100 integration wheels, a prebuilt Agent Data Plane binary) has no s390x build. This kit produces an equivalent tarball by hand, so that chain is bypassed.

3.2 Prerequisites

podman --version                       # 4.x+
command -v git curl sha256sum zstd tar

# on an x86_64 build host, register the emulator once (needs root):
podman run --rm --privileged docker.io/tonistiigi/binfmt --install s390x
ls /proc/sys/fs/binfmt_misc/qemu-s390x # must exist

~25 GB free disk. No Go toolchain needed — the build fetches a checksum-pinned one.

3.3 Build

From a patched source tree (sections 1.4 onwards):

bash <path-to>/scripts/7.79.2/image/build.sh /<source_root>/datadog-agent-7.79.2

10–15 minutes, most of it the first-time Go module download. It ends with:

### binaries OK (ELF64 MSB, IBM S/390)
Agent 7.79.2 - Commit: 11e5a46e - Serialization version: v5.0.193 - Go version: go1.26.3
built datadog-agent-s390x:7.79.2   (artifact sha256 <digest>)

The three stages, if you want to run them separately:

Script Does
image/crossbuild.sh cross-compiles agent, trace-agent, process-agent for s390x
image/make-artifact.sh lays out /opt/datadog-agent and tars it
dockerfiles/DatadogAgent.dockerfile builds the image from that tarball

Cross-compilation runs inside an ubuntu:24.04 amd64 container with crossbuild-essential-s390x. cgo is mandatory (DataDog/zstd is imported unconditionally), and that base gives an s390x cross-glibc 2.39 matching the ubuntu:24.04 s390x runtime image exactly. It takes ~2 minutes; building natively under emulation would take hours.

3.4 Verify

podman run --rm --platform linux/s390x datadog-agent-s390x:7.79.2 agent version

Full acceptance suite on a real Z host — transfers the image, then runs 6 tests and 18 assertions:

bash <path-to>/scripts/7.79.2/image/test-on-z.sh root@<your-z-host>

It refuses to run anywhere uname -m != s390x, so it cannot pass by accident on x86 or under emulation. Expect ALL CHECKS PASSED.

3.5 Run

podman run -d --name dd-agent \
  -e DD_API_KEY=<your-key> -e DD_SITE=datadoghq.com -e DD_HOSTNAME=<host> \
  datadog-agent-s390x:7.79.2
podman exec dd-agent agent status

With an invalid API key the forwarder component reports unhealthy and everything else is fine — expected, not a port defect.

3.6 What the image contains

Included — agent, trace-agent, process-agent (ELF64 MSB, IBM S/390, Go 1.26.3); Go core checks; DogStatsD on 8125/udp; APM intake on 8126/tcp; s6 supervision; the upstream entrypoint, cont-init.d scripts and healthcheck; dd-agent uid 100; nosys.so via LD_PRELOAD. ~445 MB.

Omitted — embedded CPython and all Python integrations; JMX; FIPS; the OTel agent; system-probe and the eBPF CO-RE assets. These are the omnibus-only pieces. Python Version: n/a in agent status is expected.

3.7 Deltas from the upstream Dockerfile

Ten changes, each marked DELTA Dn in dockerfiles/DatadogAgent.dockerfile with the failure it fixes. The root one is s6-overlay v3.2.3.2 instead of v2.2.0.3, because v2.2.0.3 publishes no s390x asset and upstream's arch selector has no s390x branch. Five further deltas follow from that v2→v3 move (/var/run symlink handling, cont-init.d execute bits, /command vs /usr/bin binary paths, the s6-test→eltest rename, and the legacy scandir path). The rest come from dropping omnibus: no JMX/FIPS stages, CA bundle from the distro, and no secret-generic-connector/private-action-runner.

Note that the v2→v3 deltas are not s390x-specific — if upstream moves to s6-overlay v3, the same adaptations are needed on every architecture.

See scripts/7.79.2/image/README.md for the full table and the supply-chain controls (all remote inputs are checksum-pinned and every gate fails closed).

4. Known Limitations

  1. ebpf-manager v0.7.16 hardcodes arch = "x64" in getSyscallFnNameWithKallsyms() — needs an upstream case "s390x". Affects runtime syscall probe resolution.
  2. CFA calculation for framed functions is approximate (SP+8 vs SP+framesize) — full accuracy needs DWARF frame size.
  3. Stack walking limited to 2 frames; Go does not emit frame pointers on s390x.
  4. Go register ABI unavailable on s390x — stack-based calling convention (maxIntRegs=0).
  5. CO-RE eBPF assets not pre-built for s390x.
  6. Dyninst test fixtures (pre-built Go ELF binaries) do not exist for s390x.
  7. pkg/network/netlink test fixture is a little-endian capture with no architecture gate — the product code is correct, the test is not portable.
  8. pkg/gohai/cpu cannot collect Mhz / Family / Model / Stepping / cache sizes on s390x — host metadata only.
  9. No s390x fixture for testdata/.../libssl.so.$GOARCH, so pkg/gpu/cuda TestLazySectionReader cannot run.
  10. No native-s390x image build path — image/crossbuild.sh always cross-compiles from amd64. Build on x86_64 and transfer.
  11. --pid=host container shutdown — upstream overrides s6's init-stage3 to fix this; that path does not exist in s6-overlay v3, so the fix is not carried.

5. s390x Architecture Reference

Purpose Register DWARF # pt_regs field
Syscall params GPR2-GPR7 2-7 gprs[2]-gprs[7]
Stack pointer GPR15 15 gprs[15]
Frame pointer GPR11 11 gprs[11]
Return address GPR14 14 gprs[14]
Go g pointer GPR13 13 gprs[13]
Instruction pointer PSW — psw.addr
TLS base AR0:AR1 — thread.acrs[0:1]
  • Big-endian byte order (network byte order = native order)
  • 160-byte stack frame save area
  • No standalone accept() syscall — uses accept4() or socketcall()
  • Syscall entry points prefixed __s390x_sys_
  • Statfs_t.Type is uint32 (vs int64 on amd64)

References

⚠️ **GitHub.com Fallback** ⚠️