Retry and Resilience Patterns - phax/phoss-ap GitHub Wiki

Retry and Resilience Patterns

Overview

This page describes the retry strategies and resilience patterns used by phoss-ap for outbound AS4 sending and inbound forwarding, including the circuit breakers that guard the SMP lookup, the outbound AS4 sending and the inbound forwarding.

Outbound Sending Retry

When an AS4 sending attempt fails, the outbound transaction is marked as failed and scheduled for retry by the periodic retry scheduler.

Retry Strategy

The retry strategy is configurable via Java interfaces and the following properties:

  • retry.sending.max-attempts — Maximum number of attempts before permanent failure
  • retry.sending.initial-backoff.ms — Initial backoff interval
  • retry.sending.backoff-multiplier — Exponential backoff multiplier
  • retry.sending.max-backoff.ms — Maximum backoff cap
  • retry.scheduler.interval.ms — How often the scheduler checks for eligible retries

Each failed attempt creates a new outbound_sending_attempt row. The next_retry_dt on the parent outbound_transaction is computed using the backoff strategy and stored in the database, so the scheduler can efficiently query WHERE status = 'failed' AND next_retry_dt <= NOW().

Multi-Instance Safety

The retry scheduler uses SELECT ... FOR UPDATE SKIP LOCKED when picking up eligible transactions. This ensures that when multiple AP instances run concurrently, each transaction is retried by exactly one instance. No leader election is needed — all instances run the scheduler, and the database coordinates work distribution.

phase4 Internal HTTP Retry Disabled

phase4 offers its own internal HTTP retry mechanism (HttpRetrySettings). In phoss-ap this is set to 0 (disabled), because all retry logic is handled at the application level by the phoss-ap retry scheduler. This ensures a clean separation: each phoss-ap retry attempt creates exactly one outbound_sending_attempt row with its own AS4 Message ID.

The circuit breakers (see below) also operate at the application level, gating whether a new sending attempt should be initiated at all.

Inbound Forwarding Retry

When forwarding a received document to the Receiver Backend fails, the inbound transaction is marked as forward_failed and scheduled for retry.

Retry Strategy

Analogous to outbound, configured via:

  • retry.forwarding.max-attempts — Maximum number of attempts before permanent failure
  • retry.forwarding.initial-backoff.ms — Initial backoff interval
  • retry.forwarding.backoff-multiplier — Exponential backoff multiplier
  • retry.forwarding.max-backoff.ms — Maximum backoff cap

Each failed attempt creates a new inbound_forwarding_attempt row. The next_retry_dt on the parent inbound_transaction is updated accordingly.

Deferred Verification Retry (since v0.12.0)

With verification.verifier-fail-mode=deferred, an inbound document whose verifier backend service was unavailable is parked in status verification_deferred instead of being rejected. The same scheduler cycle that drives the forwarding retries also re-verifies these documents.

  • Selection: status = verification_deferred AND next_retry_dt <= NOW(), batched with FOR UPDATE SKIP LOCKED — like the forwarding retry.
  • Schedule: a fixed interval (verification.deferred.retry-interval, default 5m) instead of exponential backoff — the AP is waiting for a service to come back, not backing off from a rejecting peer.
  • Bound: a wall-clock deadline (verification.deferred.max-duration, default 12h, relative to received_dt) instead of an attempt counter. When it is exceeded, the document is rejected and the negative MLS is sent, so C2 always gets a final answer.
  • The forwarding attempt_count is deliberately not touched by a deferral, so a long verifier outage does not consume the forwarding retry budget.

Once the verifiers accept the document, the processing continues where it was interrupted — MLS correlation, forwarding to C4 and the positive MLS to C2.

Multi-Instance Safety

Same SKIP LOCKED approach as outbound retry — safe for concurrent instances without coordination.

MLS Watchdog (since v0.13.0)

With mls.sending.trigger=api the positive MLS of a successfully forwarded document is not sent by the AP but by the Receiver Backend, via POST /api/mls/send. A backend that stays silent would leave the document without any MLS at all, so the same scheduler cycle carries a watchdog that answers C2 on its behalf.

  • Selection: status = forwarded AND mls_response_code IS NULL AND as4_timestamp < NOW() - mls.sending.api.timeout, excluding inbound MLS and MLR documents, everything with verification_result = rejected and everything with mls_type = FAILURE_ONLY. Batched with FOR UPDATE SKIP LOCKED, like the other cycles, and it needs no schema change — the existing columns carry all the information.
  • The age is measured on as4_timestamp and not on completed_dt, because MLS-1 is measured from the reception of the document. Anchoring on the forwarding would add the forwarding duration on top of the SLA budget instead of consuming it from it: a document that only reached C4 after 15 minutes of retries would then be answered at minute 20 at the earliest. A slow forwarding therefore shortens the window of the backend, and in the extreme case the fallback goes out with the first cycle after the forwarding — which is the correct answer when there is no budget left.
  • A FAILURE_ONLY transaction is excluded because it never gets a positive MLS at all. Recording a fallback response code for it would put nothing on the wire and only block the rejection that the Receiver Backend may still report.
  • Schedule: no backoff and no attempt counter. Each transaction is answered once; the claimed mls_response_code takes it out of the selection.
  • Bound: the response code of the fallback is mls.sending.api.timeout.code (default AB — the document was accepted, the delivery to C4 was never confirmed).
  • The fallback exists for compliance: MLS-1 of the Peppol Network Policy requires 99.5 % of all MLS responses within 20 minutes of the reception of the business document, so the default timeout of 5m deliberately leaves room for the AS4 sending and its retries.

Because the check runs inside the existing cycle, the effective delay is up to one retry.scheduler.interval longer than the configured timeout.

Manual Replay (since 0.11.0)

Once the automatic retries are exhausted (status permanently_failed), forwarding can be re-triggered manually via POST /api/ops/inbound/{sbdhInstanceID}/replay. The replay runs through the same forwarding pipeline as a scheduled retry — including circuit breaker, attempt recording and notification handlers — and is not restricted to failed transactions, so it can also re-deliver an already forwarded document. See API Specification.

Circuit Breakers

Motivation

When a remote system is down or unreachable, repeatedly calling it wastes resources and delays the processing of everything else. A circuit breaker prevents that by temporarily blocking calls to an unhealthy remote system, giving it time to recover.

The three circuit breakers

All of them are managed by CircuitBreakerManager in phoss-ap-core, implemented in-memory with Failsafe and identified by a key:

Key Guards Opened by
smp$<SMP host URI> The SMP lookup of an outbound transaction Failures of the SMP infrastructure only — see below
ap$<C3 endpoint URL> The outbound AS4 transmission A failed AS4 transmission
phoss-ap-forwarder The inbound forwarding to the Receiver Backend A failed forwarding

The SMP and the AP circuit breaker are keyed by the resolved host respectively endpoint URL, not by the receiver participant ID, because many participants share the same SMP and the same AP. The forwarding circuit breaker is a single one, because the forwarding target is configured per AP instance.

Design

State transitions follow the standard circuit breaker pattern:

CLOSED    ──(failure threshold reached)──>  OPEN
OPEN      ──(open duration expires)──>      HALF_OPEN
HALF_OPEN ──(probe succeeds)──>             CLOSED
HALF_OPEN ──(probe fails)──>                OPEN
  • Closed: Normal operation. Failures are counted.
  • Open: Every call is rejected immediately. The transaction stays in failed respectively forward_failed status with next_retry_dt set.
  • Half-open: circuit-breaker.half-open-max-attempts probe calls are allowed through. If they succeed, the circuit breaker closes. If they fail, it reopens.

The orchestrators use the circuit breaker directly rather than through a phase4 extension point: CircuitBreakerManager.tryAcquirePermit (key) is called before the remote system is contacted, and exactly one of recordSuccess (key) / recordFailure (key, cause) is called afterwards, on every code path.

Every acquired permit must be recorded (since v0.13.0). A permit that is acquired in the half-open state and never recorded is never released again — Failsafe releases it only in recordSuccess / recordFailure, there is no timeout. With the default circuit-breaker.half-open-max-attempts=1 a single leaked permit therefore leaves the circuit breaker stuck in HALF_OPEN forever: every call is rejected, no state change event is logged anymore, and only a restart clears it. All three usages are wrapped in a try/finally that records a failure if nothing else did.

What counts as a failure (since v0.13.0)

For the SMP circuit breaker, the outcome of a lookup is classified by SmpLookupFailureClassifier before it is recorded:

Outcome Circuit breaker Transaction
The receiver or the service is not registered (HTTP 404, or no endpoint matches the process and transport profile) recordSuccess — the SMP answered, this is a valid answer Permanent failure (or the MLS_TO fallback for an MLS)
The SMP is unavailable: connection failure, connect or read timeout, unknown host, HTTP 5xx, unparsable or badly signed response recordFailure Retried
Anything else, e.g. HTTP 400 or 403, or a certificate that cannot be extracted recordFailure Per Phase4Exception.isRetryFeasible ()

Before v0.13.0 every exception counted as a failure, so a handful of lookups for unregistered receivers suspended a large SMP for all of its participants. The retry decision is deliberately taken away from isRetryFeasible () for the first two rows: phase4 reports a "not registered" answer as retry feasible and an SMP timeout as not retry feasible, which is the wrong way round for both.

Rejections do not consume a retry attempt (since v0.13.0)

A call that is rejected by a circuit breaker never reaches the remote system, so it must not count as an attempt:

  1. No outbound_sending_attempt / inbound_forwarding_attempt row is created.
  2. The attempt_count of the transaction is left unchanged.
  3. The status is set to failed respectively forward_failed and next_retry_dt is set to now + max (remaining open duration, retry scheduler interval), so the retry happens as soon as the circuit breaker may grant a permit again.
  4. The rejection message is stored in error_details and in the sending report, so the deferral stays visible.

To keep a permanently unreachable remote system from deferring a transaction forever, the deferral only applies while the transaction is younger than circuit-breaker.defer-max-duration (default 12h). An older transaction falls back to the regular attempt counting.

Rejection messages (since v0.13.0)

CircuitBreakerManager remembers per key when the circuit breaker was opened and what the last failure was, and builds a message an operator can act on:

SMP access to 'https://smp.example.org' suspended by circuit breaker
(state OPEN since 2026-09-14T10:12:03Z, 42s remaining) after 5 failures;
last failure: SocketTimeoutException: Read timed out after 10000 ms

The same message goes into the log, the transaction error_details and the sending report. onOpen is logged at WARN including the last failure cause; onHalfOpen and onClose stay at INFO.

Configuration

See Configuration Properties for the full list, including the optional time based and rate based thresholding added in v0.13.0.

Property Description Default
circuit-breaker.failure-threshold Number of consecutive failures before opening 5
circuit-breaker.open-duration How long it stays open before transitioning to half-open 1m
circuit-breaker.half-open-max-attempts Number of probe attempts allowed in half-open state 1
circuit-breaker.defer-max-duration Maximum transaction age for which a rejection does not consume an attempt 12h
circuit-breaker.failure-executions Executions the failure threshold is measured over unset
circuit-breaker.failure-period Rolling time window the failure threshold is measured over unset
circuit-breaker.failure-rate Failure rate in percent at which it opens unset

The configuration is read when a circuit breaker is created. Changing a property therefore only affects keys that are used for the first time afterwards — existing circuit breakers keep their settings until the AP is restarted or the key is reset.

In-Memory Design

The state is held purely in-memory (ConcurrentHashMap keyed by the circuit key). This means:

  • State is lost on AP restart — this is intentional, as a restart is itself a recovery event.
  • No database persistence overhead.
  • Thread-safe for concurrent calls to different remote systems.
  • In a multi-instance deployment, each instance maintains its own independent state. This is acceptable — each instance protects itself, and different instances may have different views of a remote system's health.

Runtime visibility (since v0.13.0)

  • GET /api/ops/circuit-breakers lists every known circuit breaker with its key, state, open-since timestamp, remaining delay, failure count and last failure cause.
  • POST /api/ops/circuit-breakers/{circuitKey}/reset forgets a single circuit breaker, so the next call creates a new, closed one from the current configuration. This is also the way to apply a changed circuit-breaker.* property without a restart.
  • The counters phoss.ap.circuit_breaker.rejections and phoss.ap.circuit_breaker.state_changes are tagged with the circuit breaker key and the state — see OpenTelemetry Integration.

Both endpoints are part of the Operations API and require the X-Token header. See API Specification.

Module Placement

Concern Module
Circuit breaker manager (in-memory state, Failsafe integration) phoss-ap-core
Retry strategy interface phoss-ap-api
Retry scheduler phoss-ap-core
Circuit breaker configuration properties phoss-ap-webapp (application.properties)

The Failsafe library dependency is added to phoss-ap-core only.