Building Datadog Agent - linux-on-ibm-z/docs GitHub Wiki
The instructions provided below specify the steps to build Datadog Agent version 7.79.2 on Linux on IBM Z for the following distributions:
- RHEL (9.6, 9.8, 10.0, 10.2)
- SLES 16
- Ubuntu (22.04, 24.04)
RHEL, Ubuntu, and SLES were validated on bare-metal/LPAR installs with the kernels listed above.
If you want to build Datadog Agent using manual steps, go to Step 1.2.
Use the following commands to build Datadog Agent using the build script.
export SOURCE_ROOT=/<source_root>/
cd $SOURCE_ROOT
# Use wget to download build script. Please make sure wget is installed
wget -q 'https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/build_datadog-agent.sh'
# Build Datadog Agent
bash build_datadog-agent.sh -y # Provide -t option for executing build with testsIn case of error, check logs for more details or go to Step 1.2 to follow manual build steps.
export SOURCE_ROOT=/<source_root>/- RHEL (9.6, 9.8, 10.0, 10.2)
sudo yum install -y gcc gcc-c++ make wget curl tar git patch diffutils file- SLES 16
sudo zypper install -y gcc gcc-c++ make wget curl tar git patch file- Ubuntu (22.04, 24.04)
sudo apt-get update
sudo apt-get install -y gcc g++ make wget curl ca-certificates tar git patch filecd $SOURCE_ROOT
wget -q https://go.dev/dl/go1.26.3.linux-s390x.tar.gz
sudo tar -C /usr/local -xzf go1.26.3.linux-s390x.tar.gz
rm -f go1.26.3.linux-s390x.tar.gz
export PATH=/usr/local/go/bin:$PATH
go versioncd $SOURCE_ROOT
git clone --depth 1 -b 7.79.2 https://github.com/DataDog/datadog-agent.git datadog-agent-7.79.2
cd datadog-agent-7.79.2Apply the s390x patches. Use the provided apply-patch.sh script, which
auto-detects each patch's strip level, places new source files before the diffs
that modify them, and skips patches already present in the shipped source (019
dedup, 042 redundant with 023):
apply-patch.sh takes the patch location as an https:// URL and downloads
each patch itself, so nothing needs to be checked out locally:
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/apply-patch.sh
bash apply-patch.sh https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/DatadogAgent/7.79.2/patchExpect 31 applied, 2 SKIP, 0 FAIL and no *.rej files. Fuzz on a few hunks is
expected against the 7.79.2 tag. Confirm patch 044 landed — the test suite
depends on it:
head -8 pkg/ebpf/ebpftest/log.go | grep '//go:build test'Alternatively, apply patches manually from the source root. Note the strip level differs by patch: datadog-agent.patch, 001–030, 036, 044, and 045 use -p1; 031–035 and 037–043 use -p4 (they carry absolute source paths):
# Phase 1: Core Agent
patch -p1 < <path-to>/patch/datadog-agent.patch
# Phase 2: eBPF Security Probes — diff patches
patch -p1 < <path-to>/patch/001-syscall-macro-s390x.patch
patch -p1 < <path-to>/patch/002-bpf-metadata-s390x.patch
patch -p1 < <path-to>/patch/003-bpf-tracing-typo.patch
patch -p1 < <path-to>/patch/016-syscalls-unsupported-build-constraint.patch
patch -p1 < <path-to>/patch/017-mount-linux-statfs-type-cast.patch
patch -p1 < <path-to>/patch/019-syscalls-string-dedup.patch
patch -p1 < <path-to>/patch/020-probes-s390x-syscall-prefix.patch
patch -p1 < <path-to>/patch/021-rc-btf-s390x.patch
# Phase 2: eBPF Security Probes — new source files
cp <path-to>/patch/008-ptracer-syscalls-s390x.go pkg/security/ptracer/syscalls_s390x.go
cp <path-to>/patch/009-tests-syscalls-s390x.go pkg/security/tests/syscalls_s390x.go
cp <path-to>/patch/010-kernel-bugs-s390x.go pkg/ebpf/kernelbugs/kernel_bugs_s390x.go
cp <path-to>/patch/011-consts-linux-s390x.go pkg/security/secl/model/consts_linux_s390x.go
cp <path-to>/patch/012-syscalls-linux-s390x.go pkg/security/secl/model/syscalls_linux_s390x.go
cp <path-to>/patch/013-syscalls-string-linux-s390x.go pkg/security/secl/model/syscalls_string_linux_s390x.go
cp <path-to>/patch/014-btfhub-constants-s390x.json pkg/security/probe/constantfetch/btfhub/constants_s390x.json
cp <path-to>/patch/015-btfhub-s390x.go pkg/security/probe/constantfetch/btfhub_s390x.go
# Phase 3: Network Monitoring eBPF
patch -p1 < <path-to>/patch/022-usm-get-tls-base-s390x.patch
patch -p1 < <path-to>/patch/023-go-tls-location-s390x.patch
# Phase 4: Dynamic Instrumentation eBPF
patch -p1 < <path-to>/patch/024-bpf-tracing-custom-s390x.patch
patch -p1 < <path-to>/patch/025-cfa-s390x.patch
patch -p1 < <path-to>/patch/026-event-s390x.patch
patch -p1 < <path-to>/patch/027-abi-s390x.patch
patch -p1 < <path-to>/patch/028-irgen-s390x.patch
patch -p1 < <path-to>/patch/029-bininspect-s390x.patch
patch -p1 < <path-to>/patch/030-safeelf-s390x.patch
# Phase 5: Cross-Architecture Bug Fix
patch -p1 < <path-to>/patch/045-dyninst-decode-typeid-endianness.patch
# Phase 6: Dynamic Instrumentation Big-Endian Byte Order (031-035, 037-043 use -p4)
patch -p4 < <path-to>/patch/031-dyninst-object-big-endian.patch
patch -p4 < <path-to>/patch/032-dyninst-gotype-byteorder.patch
patch -p4 < <path-to>/patch/033-dyninst-gosym-byteorder.patch
patch -p4 < <path-to>/patch/034-dyninst-decode-nativeendian.patch
patch -p4 < <path-to>/patch/035-dyninst-dwarf-loclist-byteorder.patch
patch -p1 < <path-to>/patch/036-dyninst-irgen-loclist-bo.patch
# Phase 7: eBPF Telemetry
patch -p4 < <path-to>/patch/037-ebpf-probes-test-s390x.patch
patch -p4 < <path-to>/patch/038-bpf-telemetry-bounded-negation.patch
patch -p4 < <path-to>/patch/039-error-telemetry-probe-read-kernel.patch
patch -p4 < <path-to>/patch/040-errors-telemetry-dword-atomic.patch
patch -p4 < <path-to>/patch/041-errors-telemetry-test-s390x.patch
# Phase 8: Additional Fixes
patch -p4 < <path-to>/patch/042-go-tls-location-core-s390x.patch
patch -p4 < <path-to>/patch/043-sample-yaml-s390x-tags.patch
# Phase 9: B1 test-tag alignment (gate ebpftest/dyninsttest behind the `test`
# build tag so they match the test-only log.Default/slog helper APIs)
patch -p1 < <path-to>/patch/044-b1-ebpftest-dyninsttest-test-tag.patch
Patch summary by phase:
| Phase | Description | Patches | Key Changes |
|---|---|---|---|
| 1 | Core Agent | 4 | s390x arch definition (ARCH_S390X), container platform mapping, TestPlatform fix (redhat-release as file), runtime.Version() sanitization for vendor Go |
| 2 | eBPF Security Probes | 17 | Syscall macros (GPR2-GPR7), ptracer register mapping, BTFHub constants, full syscall table (350+ entries), build constraint fixes, Statfs_t.Type uint32 cast, SYS_ACCEPT4 (no standalone accept on s390x), syscall prefix __s390x_sys_, BTF remote config |
| 3 | Network Monitoring eBPF | 2 | TLS base pointer via access register pair (acrs[0:1]), Go TLS register access via gprs[] loop, read_register_indirect(), read_stack() via PT_REGS_SP |
| 4 | Dynamic Instrumentation eBPF | 7 | DWARF register macros (GPR0-15, BP=R11, SP=R15, PC=psw.addr), CFA calculation (frameless=SP, framed=SP+8), goroutine pointer via R13, Go ABI config (stack-based, maxIntRegs=0), ELF machine type EM_S390 mapping |
| 5 | Cross-Architecture Bug Fix | 1 | Fix TypeID collision in TestFormatStruct — latent endianness bug where duplicate TypeID 0 caused int32 decoder to overwrite int64 decoder (masked on little-endian, exposed on big-endian) |
| 6 | Dynamic Instrumentation Big-Endian | 6 | Replace ~40 hardcoded binary.LittleEndian calls with proper byte-order threading across object, gotype, gosym, decode, DWARF loclist, and irgen |
| 7 | eBPF Telemetry | 5 | Fix clang optimization breaking errno negation pattern, bpf_probe_read_kernel for kernel >= 5.x, 64-bit atomics (asm.DWord), test adjustments |
| 8 | Additional Fixes | 2 | Go TLS location core register reading via ctx->gprs[], sample YAML ConditionVariableUnavailable tags for return probes |
| 9 | B1 test-tag alignment | 1 | Gate ebpftest (log.go, fail_log.go) and dyninsttest (util.go) behind the test build tag so they match the test-only log.Default/slog helper APIs in pkg/util/log (patch 044) |
Golden-file snapshots for dyninst tests (compiler, gotype, irgen) are also included for s390x.
export CGO_LDFLAGS_ALLOW="-Wl,--wrap=.*"
export CGO_CFLAGS="-Werror -Wno-deprecated-declarations"
export GO111MODULE=on
BUILD_TAGS="bundle_agent cel consul containerd cri datadog.no_waf ec2 etcd"
BUILD_TAGS="$BUILD_TAGS fargateprocess grpcnotrace jmx kubeapiserver kubelet"
BUILD_TAGS="$BUILD_TAGS linux_bpf"
BUILD_TAGS="$BUILD_TAGS ncm netcgo no_dynamic_plugins oracle orchestrator"
BUILD_TAGS="$BUILD_TAGS otlp retrynotrace sharedlibrarycheck systemprobechecks"
BUILD_TAGS="$BUILD_TAGS zk zlib zstd"
COMMIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
LDFLAGS="-X github.com/DataDog/datadog-agent/pkg/version.Commit=${COMMIT_SHA}"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentVersion=7.79.2"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentPayloadVersion=v5.0.193"
LDFLAGS="$LDFLAGS -X github.com/DataDog/datadog-agent/pkg/version.AgentPackageVersion=7.79.2"
go build \
-mod=readonly \
-tags "$BUILD_TAGS" \
-o ./bin/agent/agent \
-ldflags="$LDFLAGS" \
-trimpath \
github.com/DataDog/datadog-agent/cmd/agentTo build without eBPF support, remove linux_bpf from BUILD_TAGS.
./bin/agent/agent versionOutput should be similar to:
Agent 7.79.2 - Commit: <sha> - Serialization version: v5.0.193 - Go version: go1.26.3
file ./bin/agent/agentOutput should be similar to:
ELF 64-bit MSB executable, IBM S/390, version 1 (SYSV), dynamically linked
MSB is the check that matters — it confirms a big-endian s390x binary.
sudo cp ./bin/agent/agent /usr/local/bin/datadog-agentNote::_The following build tags are excluded because they require x86/arm64-only native code, container runtime dependencies, or CGO/rtloader: systemd, docker, jetson, nvml, pcap, crio, podman, trivy, python.
scripts/7.79.2/test/run-tests-s390x.sh runs the validation suite with the
correct build tags, ldflags and package groups, and tallies the results. Two
modes:
cd /<source_root>/datadog-agent-7.79.2
# the packages the port modifies -- ~1731 cases, ~1 minute
bash <path-to>/scripts/7.79.2/test/run-tests-s390x.sh --targeted . ./testlogs
# every package that compiles for s390x -- ~19268 cases, ~90 minutes
bash <path-to>/scripts/7.79.2/test/run-tests-s390x.sh --full . ./testlogsResults land as go test -json in the output directory. Tally them with:
python3 -I <path-to>/scripts/7.79.2/test/tally_tests.py ./testlogs/*.json
# per-failure assertion text
python3 -I <path-to>/scripts/7.79.2/test/tally_tests.py ./testlogs/*.json --detailThese are test dependencies, not build dependencies. Each one, if missing, produces failures that look like port defects but are not. The runner preflights for all of them and warns before starting.
| Requirement | Without it |
|---|---|
xz |
pkg/ebpf TestRemoteConfigBTF* fail — the helper shells out to tar -cvJ
|
tzdata |
pkg/collector/runner/expvars fails with unknown time zone America/New_York
|
iproute2, conntrack-tools
|
pkg/network/netlink conntrack tests fail — they shell out to ip netns add and conntrack -F
|
RLIMIT_MEMLOCK unlimited |
every eBPF map test fails with failed to set memlock rlimit: operation not permitted
|
When running in a container, that last one means:
podman run --privileged --ulimit memlock=-1 -v /sys/kernel/debug:/sys/kernel/debug ...34 packages, three groups, run on RHEL 10.2 (s390x, kernel 6.12, Go 1.26.3):
| Group | PASS | FAIL | SKIP |
|---|---|---|---|
| Correctness (20 package patterns) | 1589 | 1 | 1 |
| B1-unblocked (4 pkgs, un-buildable before patch 044) | 58 | 19 | 1 |
| Fixture-dependent (6 dyninst pkgs) | 29 | 33 | 0 |
| Total | 1676 | 53 | 2 |
All 53 failures map to known infrastructure blockers: 33 dyninst test fixtures,
18 CO-RE .o assets, 1 clang-bpf, 1 the ebpf-manager arch="x64" hardcode.
Zero s390x logic failures.
Reproduced across RHEL 9.6, RHEL 10.2, Ubuntu 24.04, SLES 16.0, Rocky Linux
9.8 / 10.2, Debian 12 / 13 and Fedora 42. The same suite on x86_64
(Ubuntu 24.04, UBI 10.2, SLES BCI 16.0 containers) gives 1650 PASS / 61 FAIL,
with the dyninst-fixture, CO-RE and clang-bpf failure counts byte-identical
to s390x — confirming those blockers are architecture-independent.
Every package under pkg/, comp/ and cmd/ that compiles for s390x — run
with --full on RHEL 10.2:
| Group | PASS | FAIL | SKIP |
|---|---|---|---|
| pkg/ (6 chunks) | 14803 | 134 | 27 |
| comp/ | 3627 | 2 | 1 |
| cmd/ | 671 | 1 | 2 |
| Total | 19101 | 137 | 30 |
99.1% pass. Of the 137 failures, 87 are the same B5/B6/B7 blockers as
above, 17 are parent aggregators of those, and 31 are container-environment
artifacts (missing ip/conntrack, an ungenerated system-probe-lite helper,
docker runtime, host networking).
Exactly two genuine s390x findings in the whole tree:
-
pkg/network/netlink :: TestDecodeAndReleaseEvent— a non-portable test fixture, not a product defect. The test hardcodes a netlink message captured on a little-endian host and carries no architecture gate. Netlink attribute headers are native-endian by kernel convention and the decoder correctly usesnlenc.Uint16, so on s390x the fixture is misread: header bytes0x34 0x00 0x01 0x80parse as length=52 type=0x8001 on little-endian but length=13312 type=0x0180 on big-endian, overrunning the buffer so zero attributes decode. The product code is correct on s390x. Upstream fix: build the fixture withbinary.NativeEndian, or gate the test to little-endian. (Secondary:decoding_test.go:34usesassert.Lenwhererequire.Lenis needed, so it panics instead of failing cleanly.) -
pkg/gohai/cpu :: TestCollectCPU— a genuine, low-impact gap. Collection ofMhz,Family,Model,Steppingand the cache-size fields returns an error that is notutils.ErrNotCollectable, because s390x/proc/cpuinfohas a different layout. Affects host metadata only.
| Packages | Test functions | |
|---|---|---|
| Repository total | 1117 | 11599 |
| Compile for s390x | 686 | 6355 |
Exercised by --targeted
|
34 | 434 |
Exercised by --full
|
686 | 6355 |
94 packages do not build for s390x. Almost all fail on undefined: runtime.<Asset> — the generated eBPF runtime-compilation assets
(RuntimeSecurity, OomKill, TcpQueueLength, SharedLibraries, Gpu),
which require a BPF-target clang for s390x. The exception is
opentelemetry-ebpf-profiler, which has no s390x support at all
(undefined: pfelf.CurrentMachine).
scripts/7.79.2/image/ builds a Datadog Agent container image for s390x
without omnibus. Built and verified on real IBM Z hardware (RHEL 10.2,
kernel 6.12, native s390x — no emulation): Agent 7.79.2, Build arch: s390x,
Go 1.26.3, 16 healthy components, agent/process/trace supervised by s6.
The upstream Dockerfiles/agent/Dockerfile does not compile the Agent — it
globs for datadog-agent*-$TARGETARCH.tar.xz and untars it. Omnibus is only
one way to produce that tarball, and its chain (embedded CPython, ~100
integration wheels, a prebuilt Agent Data Plane binary) has no s390x build.
This kit produces an equivalent tarball by hand, so that chain is bypassed.
podman --version # 4.x+
command -v git curl sha256sum zstd tar
# on an x86_64 build host, register the emulator once (needs root):
podman run --rm --privileged docker.io/tonistiigi/binfmt --install s390x
ls /proc/sys/fs/binfmt_misc/qemu-s390x # must exist~25 GB free disk. No Go toolchain needed — the build fetches a checksum-pinned one.
From a patched source tree (sections 1.4 onwards):
bash <path-to>/scripts/7.79.2/image/build.sh /<source_root>/datadog-agent-7.79.210–15 minutes, most of it the first-time Go module download. It ends with:
### binaries OK (ELF64 MSB, IBM S/390)
Agent 7.79.2 - Commit: 11e5a46e - Serialization version: v5.0.193 - Go version: go1.26.3
built datadog-agent-s390x:7.79.2 (artifact sha256 <digest>)
The three stages, if you want to run them separately:
| Script | Does |
|---|---|
image/crossbuild.sh |
cross-compiles agent, trace-agent, process-agent for s390x |
image/make-artifact.sh |
lays out /opt/datadog-agent and tars it |
dockerfiles/DatadogAgent.dockerfile |
builds the image from that tarball |
Cross-compilation runs inside an ubuntu:24.04 amd64 container with
crossbuild-essential-s390x. cgo is mandatory (DataDog/zstd is imported
unconditionally), and that base gives an s390x cross-glibc 2.39 matching the
ubuntu:24.04 s390x runtime image exactly. It takes ~2 minutes; building
natively under emulation would take hours.
podman run --rm --platform linux/s390x datadog-agent-s390x:7.79.2 agent versionFull acceptance suite on a real Z host — transfers the image, then runs 6 tests and 18 assertions:
bash <path-to>/scripts/7.79.2/image/test-on-z.sh root@<your-z-host>It refuses to run anywhere uname -m != s390x, so it cannot pass by accident
on x86 or under emulation. Expect ALL CHECKS PASSED.
podman run -d --name dd-agent \
-e DD_API_KEY=<your-key> -e DD_SITE=datadoghq.com -e DD_HOSTNAME=<host> \
datadog-agent-s390x:7.79.2
podman exec dd-agent agent statusWith an invalid API key the forwarder component reports unhealthy and
everything else is fine — expected, not a port defect.
Included — agent, trace-agent, process-agent (ELF64 MSB, IBM S/390,
Go 1.26.3); Go core checks; DogStatsD on 8125/udp; APM intake on 8126/tcp; s6
supervision; the upstream entrypoint, cont-init.d scripts and healthcheck;
dd-agent uid 100; nosys.so via LD_PRELOAD. ~445 MB.
Omitted — embedded CPython and all Python integrations; JMX; FIPS; the OTel
agent; system-probe and the eBPF CO-RE assets. These are the omnibus-only
pieces. Python Version: n/a in agent status is expected.
Ten changes, each marked DELTA Dn in dockerfiles/DatadogAgent.dockerfile with the
failure it fixes. The root one is s6-overlay v3.2.3.2 instead of v2.2.0.3,
because v2.2.0.3 publishes no s390x asset and upstream's arch selector has no
s390x branch. Five further deltas follow from that v2→v3 move (/var/run
symlink handling, cont-init.d execute bits, /command vs /usr/bin binary
paths, the s6-test→eltest rename, and the legacy scandir path). The rest
come from dropping omnibus: no JMX/FIPS stages, CA bundle from the distro, and
no secret-generic-connector/private-action-runner.
Note that the v2→v3 deltas are not s390x-specific — if upstream moves to s6-overlay v3, the same adaptations are needed on every architecture.
See scripts/7.79.2/image/README.md for the full table and the supply-chain
controls (all remote inputs are checksum-pinned and every gate fails closed).
-
ebpf-manager v0.7.16 hardcodes
arch = "x64"ingetSyscallFnNameWithKallsyms()— needs an upstreamcase "s390x". Affects runtime syscall probe resolution. - CFA calculation for framed functions is approximate (SP+8 vs SP+framesize) — full accuracy needs DWARF frame size.
- Stack walking limited to 2 frames; Go does not emit frame pointers on s390x.
-
Go register ABI unavailable on s390x — stack-based calling convention (
maxIntRegs=0). - CO-RE eBPF assets not pre-built for s390x.
- Dyninst test fixtures (pre-built Go ELF binaries) do not exist for s390x.
-
pkg/network/netlinktest fixture is a little-endian capture with no architecture gate — the product code is correct, the test is not portable. -
pkg/gohai/cpucannot collectMhz/Family/Model/Stepping/ cache sizes on s390x — host metadata only. -
No s390x fixture for
testdata/.../libssl.so.$GOARCH, sopkg/gpu/cudaTestLazySectionReadercannot run. -
No native-s390x image build path —
image/crossbuild.shalways cross-compiles from amd64. Build on x86_64 and transfer. -
--pid=hostcontainer shutdown — upstream overrides s6'sinit-stage3to fix this; that path does not exist in s6-overlay v3, so the fix is not carried.
| Purpose | Register | DWARF # | pt_regs field |
|---|---|---|---|
| Syscall params | GPR2-GPR7 | 2-7 |
gprs[2]-gprs[7]
|
| Stack pointer | GPR15 | 15 | gprs[15] |
| Frame pointer | GPR11 | 11 | gprs[11] |
| Return address | GPR14 | 14 | gprs[14] |
| Go g pointer | GPR13 | 13 | gprs[13] |
| Instruction pointer | PSW | — | psw.addr |
| TLS base | AR0:AR1 | — | thread.acrs[0:1] |
- Big-endian byte order (network byte order = native order)
- 160-byte stack frame save area
- No standalone
accept()syscall — usesaccept4()orsocketcall() - Syscall entry points prefixed
__s390x_sys_ -
Statfs_t.Typeisuint32(vsint64on amd64)