Web Requests Framework - decentraland/unity-explorer GitHub Wiki

General Structure

We are using a custom generic structure for performing Web Requests in an asynchronous way. It's located in DCL.WebRequests assembly.

Currently, the exposed API looks like this:

public interface IWebRequestController
    {        
        UniTask<GenericGetRequest> GetAsync(
            CommonArguments commonArguments,
            CancellationToken ct,
            string reportCategory = ReportCategory.GENERIC_WEB_REQUEST,
            WebRequestHeadersInfo? headersInfo = null,
            WebRequestSignInfo? signInfo = null);

        UniTask<GenericPostRequest> PostAsync(
            CommonArguments commonArguments,
            GenericPostArguments arguments,
            CancellationToken ct,
            string reportCategory = ReportCategory.GENERIC_WEB_REQUEST,
            WebRequestHeadersInfo? headersInfo = null,
            WebRequestSignInfo? signInfo = null);

        UniTask<GetTextureWebRequest> GetTextureAsync(
            CommonArguments commonArguments,
            GetTextureArguments args,
            CancellationToken ct,
            string reportCategory = ReportCategory.GENERIC_WEB_REQUEST,
            WebRequestHeadersInfo? headersInfo = null,
            WebRequestSignInfo? signInfo = null);
    }

Every type of request creates a strongly typed structure to define and restrain the set of possible operations. E.g. to prevent reading text from GetTexture request. This capability is not provided by Unity itself: it will just throw an exception.

The whole API is designed to provide an allocation-free way of setting parameters:

  • Everything is designed with Value Types unless it's restricted by Unity API itself
  • WebRequestHeadersInfo uses the pool and is disposed of automatically when a Web Request fails or succeeds

Common Arguments

public readonly URLAddress URL;
public readonly int AttemptsCount = 3;
public readonly int Timeout = 0;

Repetitions are handled inside the implementation of IWebRequestController. If needed (for test purposes) this behavior can be overridden.

Signing

Some requests require a signature to validate the identity of the caller. Those are sensitive APIs that should not be abused by third-party consumers.

The identity itself is provided by the IWeb3Authenticator implementation. Then with the capabilities of AuthChain and Nethereum Library x-identity-auth-chain-N headers are set in accordance.

For this to work the following structure should be provided as an argument to the request:

    public readonly struct WebRequestSignInfo
    {
        public readonly URLAddress SignUrl;

        public WebRequestSignInfo(URLAddress signUrl)
        {
            SignUrl = signUrl;
        }
    }

Common Response Parsing Scenarios

Our scenarios of handling responses are common, therefore, several utilities are presented to unify the approaches and get rid of potential boiler-plate code.

  • GenericDownloadHandlerUtils provides capabilities for generic Post and Get requests
    • OverwriteFromJson<T> - parse JSON into the existing object with Unity/Newtonsoft off/on the main thread
    • CreateFromJson<T> - create an object from JSON with Unity/Newtonsoft off/on the main thread
    • GetDataCopy() - produce a managed byte[]
    • Calling these methods will provide data and automatically release the underlying request
  • Texture creation from GetTextureWebRequest (can be further expanded to include Sprites creation)

Retry Policy

Idempotency

Idempotency is the property of an operation that can be applied multiple times without changing the result beyond the initial execution.

By default, only GET requests are idempotent and, therefore, can be repeated without an extra explicit instruction. Those include not only generic GET requests but also assets such as Textures and Audio.

Additionally, in our environment, "Signed" requests are not idempotent as the signature may expire.

Delay between retries

If a request is to be repeated, a backoff delay is applied:

  • The initial default delay is 1 second
  • With every unsuccessful attempt, it gets multiplied by the backoff multiplier, which is equal to 3 by default

Default Policy

By default, a request can be repeated up to 2 times (1 original execution + 2 retries) under the following circumstances:

  • If the network error is qualified for repetition and the request is idempotent
  • If the DNS Lookup error occurred and the request is idempotent

Error Codes Qualification

Code Name Retry? What to Do Notes
400 Bad Request ❌ Fix payload/params; validate before send. Malformed JSON, missing fields.
401 Unauthorized ❌ Refresh token once, then retry. If second fail β†’ force re‑login. Can't be applied automatically
403 Forbidden ❌ Block UI path or request proper scope. Auth OK, but not allowed.
404 Not Found ❌ Check URL/IDs; refresh local cache. Common for stale resource IDs.
405 Method Not Allowed ❌ Use correct HTTP verb. Server disallows current method.
406 Not Acceptable ❌ Change Accept header/format. Unsupported response type.
408 Request Timeout βœ… Retry with backoff. Server didn’t receive body in time.
409 Conflict ❌ Resolve state/version conflict, then manual retry. E.g., duplicate, version mismatch.
410 Gone ❌ Remove/redirect feature. Resource intentionally removed.
411 Length Required ❌ Ensure Content-Length is set. Usually auto by UnityWebRequest.
412 Precondition Failed ❌ Refresh ETag/version, retry once. Optimistic concurrency failed.
413 Payload Too Large ❌ Chunk/compress upload. Split files or use CDN.
414 URI Too Long ❌ Move params to body (POST). Overlong query string.
415 Unsupported Media Type ❌ Fix Content-Type. e.g., send JSON as application/json.
421 Misdirected Request ❌ Fix host/routing. HTTP/2 specific.
422 Unprocessable Entity ❌ Fix semantic validation errors. JSON OK syntactically, rules fail.
425 Too Early βœ… Wait briefly, retry once. Rare; HTTP/2 early data.
428 Precondition Required ❌ Add conditional headers. Similar to 412.
429 Too Many Requests βœ… "Retry-After" is respected and obligatory to be qualified for retrying Rate limiting.
431 Header Fields Too Large ❌ Trim headers/cookies. Reduce auth blobs.
451 Unavailable For Legal Reasons ❌ Inform user; disable feature. Legal/geofence block.
500 Internal Server Error βœ… Backoff; maybe switch region. Generic server crash/bug.
501 Not Implemented ❌ Downgrade client / change call. Endpoint/method not supported.
502 Bad Gateway βœ… Backoff; transient proxy/CDN issue. Upstream sent bad response.
503 Service Unavailable βœ… "Retry-After" is respected and obligatory to be qualified for retrying Overload/maintenance.
504 Gateway Timeout βœ… Retry; Upstream took too long.
505 HTTP Version Not Supported ❌ Use supported protocol. Rare with Unity.
507 Insufficient Storage ❌ Can't make any assumptions Server disk full.
508 Loop Detected ❌ Can't make any assumptions WebDAV-specific.
511 Network Auth Required ❌ Prompt captive-portal login. Intercepting proxy.
521/522/523/525/526 CDN Origin Errors βœ… Treat like 502/504; backoff. Can recover Cloudflare/Akamai variants.

Enforced policy

By calling RetryPolicy.Enforce(), it's possible to qualify the request for retries even if it's not qualified by default rules.

JS API Policy

The requests originating from SimpleFetch API are not qualified for repetitions unless "Retry-Delay" is specified in the 429 or 503 response.

ECS originated requests

Streamable Assets are not repeated directly by the WebRequestController but use their own system with promises which obey budgeting and prioritization.

Analytics

IWebRequestsAnalyticsContainer records currently ongoing requests. At the moment it's used just for counting and displaying the value in the Debug View via ShowWebRequestsCountersSystem but in the future can be expanded to hold additional information about requests.