Receiving Process - phax/phoss-ap GitHub Wiki

Receiving Process

Overview

The receiving process handles inbound Peppol documents arriving at the AP (C3) via AS4 from a remote sending AP (C2) and forwards them to the Receiver Backend (an internal system within C3).

Process Steps

  1. Receive via AS4 - The phase4 servlet at /as4 receives the AS4 message (see AS4 Receiving Endpoint). Duplicate detection is applied on two levels (AS4 Message ID, SBDH Instance Identifier) — configurable: reject or store-and-flag. AS4 Message ID duplicate detection is backed by the configured database (as4_duplicate_item table), so detection survives restarts and is shared across multiple AP instances.

  2. Verify receiver is serviced - Check whether this AP services the addressed receiver participant for the given document type and process. This is implemented via a pluggable SPI (IPeppolReceiverCheckSPI) — e.g., by querying the AP's own SMP registration or a local configuration. If the receiver is not serviced, an AS4 error with code EBMS:0004 / PEPPOL:NOT_SERVICED is returned to C2. No database entry is created.

  3. Store document and metadata - The document is written to disk under storage.inbound.path and the file path, metadata, phase4 Incoming ID, and MLS configuration (MLS_TO from SBDH extension, MLS_TYPE from instance config) are persisted to the inbound_transaction table in the database.

  4. If storage fails - Respond with an AS4 error to the remote sending AP (C2). The transaction is not recorded.

  5. Optional verification - The received document is validated via the pluggable verification SPI (IInboundDocumentVerifierSPI). By default, the phoss-ap-validation module calls the phorm Validation Service to validate the document against Peppol BIS rules. If verification fails, the transaction status is set to rejected, a notification is triggered, an MLS response with code RE (Rejection) is sent back to C2 asynchronously (not when the received document is itself an MLR or MLS), and no forwarding attempt is made. The AS4 receipt returned to C2 is still positive in this case. The MLS response carries the verifier's detailed issues (SV / BV / BW) — see Message Level Status#verifier-driven-issue-mapping-since-v0100. All registered verifiers are evaluated; the first explicit rejection wins immediately and always takes precedence over an unavailable verifier. Since v0.12.0, a verifier whose backend service is unavailable (connection refused, HTTP 5xx, timeout) is not treated as a rejection — see Deferred verification below. Since v0.12.0, the verdict is additionally persisted in inbound_transaction.verification_result (passed / rejected / unverified) together with the findings as a JSON array of VerificationIssue objects in verification_details. That is deliberately independent of the transaction status, which the forwarding state machine overwrites — see Inbound Verification Result. Also since v0.12.0, a rejected document can optionally still be forwarded to C4 via verification.inbound.rejection-forwarding — see Forwarding a rejected document below. The default none keeps the behaviour described above.

  6. Incoming MLS correlation - If the received document is itself an MLS, it is parsed as a UBL ApplicationResponse and correlated to the original outbound transaction via its reference ID (the SBDH Instance ID of the original business document) — see the MLS Reception Flow on the Sending Process page. The MLS then continues through the regular forwarding step below like any other document.

  7. Forward to Receiver Backend - Deliver the SBD via the forwarding mode configured by the forwarding.mode property:

    • http_post_async — HTTP POST (async reporting)
    • http_post_sync — HTTP POST with sync response (C4 country code in response)
    • s3_link — S3 + link forwarding
    • sftp — SFTP
    • filesystem — Local filesystem write
    • spi — deployment-provided forwarder via IDocumentForwarderProviderSPI (since v0.10.0)

    After a successful primary forwarding, all configured secondary forwarders additionally receive a copy asynchronously (fire-and-forget, no retry) — see Forwarding Process.

  8. Record forwarding attempt - Insert a row into inbound_forwarding_attempt with the outcome (success or failure + error details). Update the parent inbound_transaction status accordingly.

  9. If forwarding fails - The document will be retried by a periodic scheduler (only if the receiver allows a retry and the maximum attempt count is not yet reached). Each retry creates a new inbound_forwarding_attempt row. If the forwarder circuit breaker is open, no forwarding is attempted at all: no attempt row is written and the attempt_count stays unchanged, but the transaction is set to forward_failed with a next_retry_dt, so that the scheduler picks it up again once the breaker closes.

  10. Reporting - After successful forwarding, the C4 country code is taken from the synchronous HTTP POST response if present; otherwise the configured determination modes (forwarding.c4countrycode.modes: receiver participant ID, Business Card cache) are tried in order. If a C4 country code is available, the reporting record is stored immediately; otherwise the Receiver Backend reports asynchronously via the reporting API (SBDH Instance ID + C4 country code). Since v0.13.0 a transaction whose sender or receiver is listed in peppol.reporting.exclude.participant-ids creates no reporting record at all and ends in reporting status excluded - with or without a C4 country code.

  11. On permanent forwarding failure - Notification via generic interface. This also covers the case that no document forwarder is configured at all, which is a local configuration error but must still be answered towards C2. An MLS response with code AB (Acknowledging) and the text "Forwarding to C4 failed for now" is sent back to C2 asynchronously (not when the received document is an MLR or MLS). Deliberately no RE with status reason FD is sent: the PNP reserves FD for a permanent inability to deliver, and the forwarding problem is assumed to be resolvable later.

  12. Send MLS response - Based on the forwarding outcome and the MLS_TYPE configuration, the AP generates an MLS response and sends it back to C2 (unless MLS sending is globally disabled via mls.sending.enabled). The response code depends on the outcome and forwarding mode:

    • RE (Rejection) — On verification failure (status reason SV, BV and/or BW per issue)
    • AP (Approved) — On successful forwarding with delivery confirmation (the HTTP modes)
    • AB (Accepted Blind) — On successful forwarding without delivery confirmation (SFTP, S3, Filesystem; for spi mode per the provider's isWithDeliveryConfirmation()), and on permanent forwarding failure (see previous step)

    The MLS sending is recorded as an outbound_transaction with transaction_type = mls_response, linked via mls_inbound_transaction_id. The inbound transaction's mls_outbound_transaction_id is set to reference it.

    Since v0.13.0 the positive MLS of this step can be deferred to the Receiver Backend, via mls.sending.trigger=api. The AP then sends nothing here and waits for POST /api/mls/send — which is what a C3 forwarding to a middleware in front of C4 needs, because a 2xx from the middleware is not proof that the document reached C4. The RE of a verification failure and the AB of a permanent forwarding failure are unaffected and always sent automatically. See Who triggers the positive MLS.

Sequence Diagram: Receiving and Forwarding

sequenceDiagram
    participant RA as Remote AP (C2)
    participant PS as phase4 Servlet
    participant RC as Receiver Check
    participant DB as DB
    participant FM as Forwarding Module
    participant RB as Receiver Backend

    RA->>PS: AS4 UserMessage

    PS->>DB: 1. Duplicate check (AS4, SBDH)
    DB-->>PS: result
    note over PS: If duplicate and reject mode: return AS4 error

    PS->>RC: 2. Verify receiver is serviced
    RC-->>PS: result
    note over PS: If not serviced: return AS4 error PEPPOL:NOT_SERVICED

    PS->>DB: 3. Store doc to disk + insert inbound_transaction (incl. MLS config)
    note over PS: If store fails: return AS4 error
    DB-->>PS: stored OK

    PS->>PS: 4. Verify doc (optional)
    note over PS: If rejected: update DB, notify, send MLS RE (async) - AS4 receipt is still positive

    PS->>FM: 5. Forward SBD
    FM->>RB: Deliver SBD
    RB-->>FM: Response (sync mode: C4 country in response)

    PS->>DB: 6. Insert forwarding attempt row
    PS->>DB: 7. Update inbound_transaction status

    opt C4 country code known (sync response or configured resolver)
        PS->>DB: Store reporting record
    end

    PS-->>RA: AS4 Receipt (after the full synchronous pipeline)
Loading

MLS Response Sending Flow (after forwarding outcome is known)

sequenceDiagram
    participant AP as AP
    participant DB as DB
    participant P4 as phase4
    participant RA as Remote AP (C2)

    AP->>AP: Determine MLS response code (RE, AP, or AB)
    note over AP: If mls.sending.enabled=false: skip MLS entirely
    AP->>AP: Check MLS_TYPE (ALWAYS_SEND or FAILURE_ONLY)
    note over AP: If FAILURE_ONLY and outcome is success: skip MLS

    AP->>DB: Create outbound_transaction (transaction_type = mls_response)
    AP->>DB: Update inbound_transaction (mls_response_code, mls_outbound_transaction_id)

    AP->>P4: Send MLS via AS4
    P4->>RA: AS4 UserMessage (MLS)
    RA-->>P4: AS4 Receipt
    P4-->>AP: AS4 result

    AP->>DB: Insert outbound_sending_attempt
    AP->>DB: Update outbound_transaction status
Loading

Async Reporting Flow (separate API call from Receiver Backend)

sequenceDiagram
    participant RB as Receiver Backend
    participant API as Reporting API
    participant DB as DB

    RB->>API: Report message (SBDH Instance ID, C4 country code)
    API->>DB: Lookup inbound transaction by SBDH Instance ID
    DB-->>API: transaction data
    API->>DB: Store reporting record (C4 country + data from SBDH)
    API-->>RB: OK
Loading

Forwarding Retry Flow (periodic scheduler)

sequenceDiagram
    participant S as Scheduler
    participant DB as DB
    participant FM as Forwarding Module
    participant RB as Receiver Backend

    S->>DB: Query inbound_transaction where status = forward_failed and next_retry_dt <= now (batched, SKIP LOCKED)
    DB-->>S: list of pending

    loop For each failed transaction
        S->>FM: Forward
        FM->>RB: Deliver SBD
        RB-->>FM: result
        FM-->>S: result

        S->>DB: Insert forwarding attempt row
        S->>DB: Update inbound_transaction status

        opt Max retries exhausted or retry disallowed by receiver
            S->>S: Trigger notification + send MLS AB (async)
        end
    end
Loading

Since 0.11.0 the same forwarding run can also be triggered manually for a single transaction via POST /api/ops/inbound/{sbdhInstanceID}/replay — see API Specification.

Deferred Verification (since v0.12.0)

An inbound document verifier can fail in two fundamentally different ways: it can inspect the document and reject it (malware found, invalid document), or it can not inspect the document at all, because its backend service is unavailable (ICAP connection refused, phorm HTTP 5xx, timeout). Before v0.12.0 both cases were treated as a rejection, so a verifier outage produced permanent negative MLS responses for perfectly valid documents.

A verifier signals unavailability by returning VerificationOutcome.serviceUnavailable (sMessage). What happens then is controlled by verification.verifier-fail-mode:

Mode Behaviour
closed (default) Like a rejection: status rejected, negative MLS (RE) to C2. The pre-v0.12.0 behaviour.
open The unverified document is forwarded to C4; a warning is logged per document. The onInboundVerificationAccepted callback is not fired. Since v0.12.0 the transaction is recorded as verification_result = unverified, so such a document stays distinguishable from a properly verified one.
deferred Status verification_deferred, next_retry_dt set, no MLS sent yet. Nothing is recorded in verification_result yet, because the verdict is still open.

In deferred mode:

  • The retry scheduler picks the transaction up every verification.deferred.retry-interval (default 5m) and re-runs all verifiers against the stored payload.
  • Once they accept, the processing continues exactly where it was interrupted: an incoming MLS is correlated with its outbound transaction, the document is forwarded to C4 and the positive MLS is sent to C2.
  • If a verifier rejects the document on a later attempt, it is rejected as usual, including the negative MLS.
  • If the verifiers are still unavailable verification.deferred.max-duration (default 12h) after the document was received, the document is rejected and the negative MLS is sent — C2 always gets a final answer. A re-verification is never scheduled beyond that deadline, so the rejection happens at the first scheduler cycle at or after it.
  • The forwarding attempt_count is never modified by a deferral; deferred verification and forwarding retries are counted and scheduled independently.
  • A verifier reports "unavailable" not only for an outage, but also when the call could not be made at all — a misconfigured endpoint URL or token, or an unreadable stored payload. In deferred mode such a configuration error also parks the affected documents until verification.deferred.max-duration is exceeded, which is another reason to act on the deferral notification rather than to wait it out.
  • POST /api/ops/inbound/{sbdhInstanceID}/reverify-and-forward triggers the same resume manually. It only accepts transactions in status verification_deferred and answers 409 for anything else — see API Specification.
  • Every deferral fires IAPNotificationHandlerSPI.onInboundVerificationDeferred (sTransactionID, sSbdhInstanceID, sVerifierName, aNextRetryDT, sErrorDetails) and increments the phoss.ap.inbound.verification.deferred OTel counter — including every unsuccessful re-verification. Alert on this: the deferral window only helps if somebody fixes the verifier inside it. When the maximum duration is exceeded, the regular onInboundVerificationRejection (...) fires instead.

The AS4 receipt returned to C2 at reception time is positive in all three modes — the document was received correctly, it just could not be verified yet.

Verification Outcome and Retry Flow

The diagram shows how the three fail modes act on the same verifier evaluation, the re-verification loop of the deferred mode, and what verification.inbound.rejection-forwarding does with a rejected document. The same loop is entered from the initial reception and from every re-verification, so a document that was deferred is verified with exactly the same logic as a freshly received one.

flowchart TD
    RCV["Document received via AS4, stored on disk,<br/>inbound_transaction created - status = received"]
    VER["Run ALL registered inbound verifiers"]
    REJQ{"Any verifier<br/>rejected the document?"}
    UNQ{"Any verifier<br/>unavailable?"}
    MODE{"verification.<br/>verifier-fail-mode"}
    MAXQ{"received_dt + max-duration<br/>already passed?"}
    DEFER["status = verification_deferred<br/>next_retry_dt = now + retry-interval<br/>error_details = verifier name + reason<br/>NO MLS sent to C2<br/>attempt_count unchanged<br/>onInboundVerificationDeferred + OTel counter"]
    SCHED["RetryScheduler cycle picks up<br/>status = verification_deferred with<br/>next_retry_dt reached<br/>batched, FOR UPDATE SKIP LOCKED"]
    MAN["Manual trigger:<br/>POST /api/ops/inbound/.../reverify-and-forward"]
    ACC["onInboundVerificationAccepted"]
    OPENM["Warning logged per document -<br/>deliberately no 'verification accepted' callback,<br/>because nothing was verified"]
    CONT["Correlate the incoming MLS, if the document is an MLS<br/>Forward the SBD to C4<br/>Positive MLS - AP or AB - to C2"]
    REJREC["Record the rejection:<br/>verification_result = rejected<br/>verification_details = findings<br/>negative MLS - RE - to C2<br/>onInboundVerificationRejection"]
    FWDMODE{"verification.inbound.<br/>rejection-forwarding"}
    REJNONE["status = rejected<br/>error_details = verifier name + reason<br/>document never forwarded"]
    REJBE["status = rejected, as in mode none<br/>+ fire-and-forget copy to the primary and to all<br/>secondary forwarders - failures logged only,<br/>no attempt row, no retry"]
    REJFWD["Regular forwarding to C4 incl. retries<br/>status = forwarded / permanently_failed<br/>verification_result stays rejected<br/>NO further MLS to C2"]

    RCV --> VER
    VER --> REJQ
    REJQ -- yes --> REJREC
    REJQ -- no --> UNQ
    UNQ -- no --> ACC
    ACC --> CONT
    UNQ -- yes --> MODE
    MODE -- closed --> REJREC
    MODE -- open --> OPENM
    OPENM --> CONT
    MODE -- deferred --> MAXQ
    MAXQ -- yes --> REJREC
    MAXQ -- no --> DEFER
    DEFER --> SCHED
    SCHED --> VER
    MAN --> VER
    REJREC --> FWDMODE
    FWDMODE -- none --> REJNONE
    FWDMODE -- best-effort --> REJBE
    FWDMODE -- retry --> REJFWD
Loading

The re-verification is a separate loop from the forwarding retry loop shown above: it is driven by status = verification_deferred, while the forwarding retry is driven by status = forward_failed. A document therefore leaves the deferral loop as soon as the verifiers produce a verdict, and only then starts to consume forwarding attempts.

Forwarding a Rejected Document (since v0.12.0)

By default a verification rejection is a dead end inside phoss-ap: the document is never forwarded and only C2 learns about it, via the negative MLS. Some deployments need the opposite — the UAE Peppol DCTCE / TDD-TDS model requires C3 to act on an invoice that failed the content validation, by generating a Tax Data Summary instead of a Tax Data Document, which means the failed document has to reach C4 while the rejection stays recorded.

verification.inbound.rejection-forwarding selects the delivery strength for such a document. It is opt-in: the default none is byte-for-byte the pre-v0.12.0 behaviour.

Mode Transaction status Forwarding inbound_forwarding_attempt rows
none (default) rejected none none
best-effort rejected fire-and-forget copy to the primary and all secondary forwarders, failures logged only none
retry forwarding → forwarded / forward_failed / permanently_failed regular forwarding state machine incl. backoff and retries one per attempt

What is identical in all three modes:

  • The verdict verification_result = rejected and the findings in verification_details are always written. In mode retry they survive every forwarding status transition, because the forwarding only touches status, attempt_count, next_retry_dt and error_details.
  • The negative MLS (RE) is always sent to C2, and onInboundVerificationRejection (...) always fires.
  • The AS4 receipt returned to C2 stays positive.

What is specific to the forwarding modes:

  • C2 receives exactly one MLS. A rejected-but-forwarded transaction would otherwise get a second, contradicting one: the positive AP / AB after a successful forwarding, or the acknowledging MLS when the forwarding retries are exhausted. Both are suppressed, derived from the persisted verification_result rather than from a call parameter — so the retry scheduler and the operations endpoint behave exactly like the initial receive path. mls_response_code therefore stays RE.
  • Peppol Reporting counts the document. In mode retry a successful forwarding creates the inbound reporting item as for any other forwarded document — the document was received over the Peppol network. In the modes none and best-effort no reporting item is created, because none is created without a successful regular forwarding. This is a deliberate decision: enabling retry moves the reported TSR figures upwards.
  • The mode applies to every rejection verdict, not only to an explicit verifier rejection — so also to a document rejected because a verifier was unavailable and verification.verifier-fail-mode=closed was configured, and to one whose verification.deferred.max-duration was exceeded.
  • In mode best-effort the copy is dispatched through the same fire-and-forget contract as the secondary forwarders: asynchronous, no retry, no SLA, a failing or throwing forwarder is logged and does not stop the remaining ones.
  • POST /api/ops/inbound/{sbdhInstanceID}/reverify-and-forward is not widened by this. It still only accepts transactions in status verification_deferred and answers 409 for anything else, so a rejected document is terminal for the operator API. In mode retry such a document ends in forwarded or permanently_failed anyway, and a deliberate re-delivery is still possible via the replay endpoint.

Inbound Message Metadata

inbound_transaction (one row per received document)

Field Description
incoming_id Unique ID assigned by phase4 on receipt
c2_seat_id Peppol Seat ID of the sending AP (C2)
c3_seat_id Peppol Seat ID of the receiving AP (C3, this AP)
signing_cert_cn Subject CN of the signing certificate from the AS4 message
sender_id Peppol Participant ID of the sender
receiver_id Peppol Participant ID of the receiver
doc_type_id Peppol Document Type Identifier
process_id Peppol Process Identifier
document_path Path to the document file — either an absolute filesystem path or an s3://bucket/key URI depending on storage.mode
document_size Size of the received SBD in bytes
document_hash SHA-256 hash of the received SBD payload for integrity verification
as4_message_id AS4 Message ID from the inbound message
as4_timestamp AS4 MessageInfo/Timestamp from the incoming AS4 message (UTC). MLS Milestone M1 for business documents; M2 for MLS messages.
sbdh_instance_id Peppol SBDH Instance Identifier
c1_country_code Country of the original sender (C1) from the SBDH; ZZ if absent
c4_country_code Country of the final receiver (C4), set via reporting
is_duplicate_as4 Duplicate detected on AS4 Message ID level
is_duplicate_sbdh Duplicate detected on SBDH Instance Identifier level
status received, rejected, verification_deferred, forwarding, forwarded, forward_failed, permanently_failed
attempt_count Total number of forwarding attempts (default 0) — a deferred verification does not change it
received_dt When the message was received via AS4
completed_dt When the transaction reached its final successful state
reporting_status Whether reporting record has been created (pending, reported, excluded)
next_retry_dt Planned date/time of the next forwarding retry (status forward_failed) or of the next re-verification (status verification_deferred); cleared when the document was successfully forwarded
error_details Summary error from the last failed forwarding attempt, or the reason of a deferred verification / verification rejection; cleared when the document was successfully forwarded
mls_to MLS_TO target participant ID (from SBDH extension)
mls_type MLS sending strategy at time of reception (FAILURE_ONLY, ALWAYS_SEND)
mls_response_code MLS response code sent or to be sent (RE, AP, AB)
mls_outbound_transaction_id FK to the outbound_transaction representing the MLS sending

inbound_forwarding_attempt (one row per forwarding try)

Field Description
inbound_transaction_id FK to the parent inbound transaction
attempt_dt Date and time of this forwarding attempt
attempt_status Outcome of this attempt (success, failed)
error_code Machine-readable error code classifying the failure (null on success)
error_details Error message or reason for failure
⚠️ **GitHub.com Fallback** ⚠️