EN Writes and Errors - AlanDevise/TsGate GitHub Wiki

Home · GitHub

English | 简体中文

2.1.3 is published through the GitHub source/tag and Maven Central; same-SHA CI and public artifact verification passed. Current coordinates: 2.1.3; see 2.1.3 release notes.

Version 2.1.3 tightens structured-filter validation, merges IoTDB physical table-name case variants before whole-batch checks, verifies backend registration in tests/CI, rejects duplicate IoTDB/InfluxDB 3 result keys, and checks direct-adapter batch counts before copying inputs. The public API, Template → SPI → adapter architecture, Java 17/client/server baselines and configuration defaults remain unchanged. 2.1.3 release notes.

Direct batchWriteDetailed(database, records) calls on IoTDB, InfluxDB 1.x, InfluxDB 3 and openGemini check records.size() against the configured max-batch-records before List.copyOf(records), then check the copied snapshot again before record validation, encoding or I/O. The inherited batchWrite entry also reaches this path. If the reported size already exceeds the limit, the adapter rejects it without invoking iterator() or toArray() to copy or inspect its records.

The failure is a TSDBBatchWriteException with ARGUMENT_ERROR, NOT_COMMITTED, zero validated/committed records and zero physical batches; no HTTP write or IoTDB session borrow occurs. Existing adapter-state checks still run first: an uninitialized or closed adapter keeps its state error, while a ready adapter keeps successful null/empty-batch behavior. TGTemplate already checked size before and after copying business objects; its behavior is unchanged.

This moves an existing record-count guard earlier. Public APIs, SPI, configuration keys/defaults and request splitting are unchanged. It does not add byte/column budgets or establish a hard heap limit for a custom Collection whose size(), isEmpty() or copying behavior is misleading or expensive. The post-copy check does not make caller-side concurrent mutations atomic. See 2.1.3 release notes for release scope and validation status.

After validating table names, IoTDB groups an application batch by Locale.ROOT lowercase physical table identity. For example, records named metrics, METRICS and Metrics share one table batch and produce Tablets named metrics; caller records and their original tag/field maps are unchanged. Row order within that table and the existing Tablet capacity remain unchanged. Compatible variants share the usual Tablet splitting, so totalBatches and committedBatches count the actual Tablets after merging, while record counts still count input records. Other physical tables retain independent column schemas.

Column roles and non-null inferred types are checked across every spelling of that physical table before acquiring any session or writing any other table. For example, a device tag in metrics and a device field in METRICS, or conflicting field types across these names, reject the entire application batch with METADATA_ERROR and NOT_COMMITTED. Confirmed record/Tablet counts are zero and there is no database I/O, even when earlier records would fill a valid Tablet. The published 2.1.2 grouping used the original table-name spelling; these broader physical-table checks are included in 2.1.3.

When a failed write identifies an attempted table, BatchWriteResult.failedMeasurement refers to its lowercase physical name. Preflight and acquisition failures that have not selected an attempted Tablet may still report null. Existing commit-state handling after I/O, the absence of a cross-Tablet transaction and the requirement to create server schema in advance remain unchanged. InfluxDB 1.x, InfluxDB 3 and openGemini retain their existing measurement-name case behavior.

An exact duplicate IoTDB output column or an exact duplicate JSON object key in a successful InfluxDB 3 query response produces TSDBException with QUERY_ERROR (100308). The entire query, count or adapter executeQuery operation fails: the caller receives neither partial rows nor a count inferred from the surviving value or an empty-result fallback. IoTDB checks metadata before iteration, and a later duplicate InfluxDB 3 row also invalidates all earlier rows from that response. Existing cleanup paths remain in use.

This is query-response validation. It does not change write payloads, BatchWriteResult, commit-state classification or native SQL text. Legitimate case-distinct output names remain distinct at decoding. Direct calls on a borrowed native client retain the SDK’s behavior. See 2.1.3 release notes for release scope and validation status.

InfluxDB 1.x, InfluxDB 3 and OpenGemini apply tsdb.<backend>.max-batch-bytes to the complete application batch's prepared UTF-8 line-protocol payloads. The default is 67108864 bytes (64 MiB); it must be positive. The budget includes payload newline separators and covers all physical requests together. It does not change the existing per-request line/byte limits, max-batch-records, query-response limits or IoTDB Tablet limits. Configuration is captured at construction.

An over-budget batch is rejected before the first request with ARGUMENT_ERROR, NOT_COMMITTED, zero committed records/batches and no I/O; it is never partly submitted to satisfy the budget. Raising the setting permits more prepared payload memory and is an explicit deployment decision. It bounds encoded payload bytes, not total JVM heap usage or an asynchronous write queue.

HTTP requests made through the common adapter APIs do not automatically follow redirects, including redirects to the same origin. Configure the final database URL instead of relying on HTTP 3xx routing. Redirect responses retain the existing backend failure/commit classification; a redirect is not proof that a write was rejected without committing. The adapter never replays an uncertain write through the redirect target. Borrowed native clients remain a separate API boundary.

import com.alandevise.tsgate.annotation.TGField;
import com.alandevise.tsgate.annotation.TGMeasurement;
import com.alandevise.tsgate.annotation.TGTag;
import com.alandevise.tsgate.annotation.TGTime;

@TGMeasurement("ACCRUE")
public class AccrueRecord {

    @TGTime
    private Long timestamp;

    @TGTag("device_code")
    private String deviceCode;

    @TGTag("point_type")
    private String pointType;

    @TGTag("source")
    private String source;

    @TGField("value")
    private Double value;

    @TGField("status")
    private Long status;
}

Annotations:

• @TGMeasurement: target measurement/table.
• @TGTime: time field, restricted to non-null Long/long values representing Unix Epoch milliseconds, without implicit timezone conversion.
• @TGTag: tag field, converted to a string on write.
• @TGField: ordinary field; at least one non-null field value is required.

A write POJO must declare one @TGTime and at least one @TGField. Query result objects need not be write POJOs and may omit TsGate annotations.

IoTDB table-model unquoted column names identify the same physical column regardless of case. Batch preflight and Tablet row values now consistently use that lowercase identity with Locale.ROOT, without modifying the caller's TSDBRecord, tag map or field map.

• A single record containing both device and DEVICE is ambiguous and rejected, even if both are tags or both fields with the same Java type. A tag and field sharing that identity are also rejected, including a null-valued field.
• Across different records, value, VALUE and VaLuE merge into one physical column when their roles and inferred non-null types agree. A null or absent value remains null, and a later non-null row can supply its field type.
• Any role/type conflict rejects the complete application batch before borrowing a session, including when an earlier measurement or full Tablet would otherwise be valid. It returns METADATA_ERROR, NOT_COMMITTED, zero committed records/batches and no database I/O.

This closes a local validation gap; it does not make multiple Tablets transactional or discover the server schema in advance. Preserve the existing commit-state handling for errors occurring after a write request begins.

Inject com.alandevise.tsgate.core.TGTemplate. When a backend is active, its starter registers the shared template under the default bean name tgTemplate:

import com.alandevise.tsgate.core.TGTemplate;
import org.springframework.beans.factory.annotation.Autowired;

@Autowired
private TGTemplate tgTemplate;

Omitting the database uses the adapter's configured default database:

boolean success = tgTemplate.write(record);

Specify a database:

boolean success = tgTemplate.write("tsdb", record);

Batch write:

boolean success = tgTemplate.batchWrite("tsdb", records);

Use the detailed result when commit boundaries matter. On failure, TSDBBatchWriteException carries the same result object:

BatchWriteResult result = tgTemplate.batchWriteDetailed("tsdb", records);

commitState is SUCCESS, NOT_COMMITTED, PARTIALLY_COMMITTED or UNKNOWN. Network errors, HTTP 408 and 5xx responses cannot prove that the current physical batch was not committed. Its state is therefore UNKNOWN, with error code BATCH_COMMIT_UNKNOWN; previously confirmed record and batch counts are retained.

With accept_partial=false, InfluxDB 3 treats HTTP 400/401/403/404/405/413/415/422/429 as a definite rejection of the current batch, reporting zero or partial commitment according to earlier successful requests. InfluxDB 1.x only treats HTTP 401/403/404/405/413/415/429 as definite rejection. HTTP 400 may have committed valid points, so its state is UNKNOWN.

retryable is independent of commit certainty: callers may retry 429, 408 and temporary 5xx responses according to an idempotency strategy; 501/505 do not suggest retry. The adapter does not automatically replay these writes. See the official write documentation for rejection semantics with accept_partial=false.

A single write becomes a one-element collection and is sent immediately as a batch. The starters do not provide asynchronous batch accumulation, unbounded buffering or a backpressure queue.

For the OpenGemini adapter, the complete batch is checked before I/O: measurement names cannot contain comma, semicolon, slash or backslash, equal . or .., or contain non-printable characters. Letters, marks, numbers, punctuation, symbols and ASCII space follow the server's printable-name rule; = is preserved. Invalid names produce ARGUMENT_ERROR with NOT_COMMITTED and zero physical batches; names are never renamed. The inherited line-protocol checks also apply. Once a request is sent, an HTTP 400 partial-write response retains UNKNOWN, as in InfluxDB 1.x.

Adapters consistently throw TSDBException or a subclass. Codes are six-digit 1003xx integers. Use exception.getErrorCode() for the enum or exception.getCode() for its integer value. These classify application errors and are distinct from HTTP status codes.

try{
        tgTemplate.write(record);
}catch(
TSDBException e){
int code = e.getCode();
String detail = e.getMessage();
Throwable rootCause = e.getCause();
}

The application's Web layer maps these codes to its response format and appropriate HTTP statuses. The starters do not register a global exception handler.

• write, batchWrite and batchWriteDetailed are synchronous. Successful return means the underlying TSDB acknowledged the write, rather than merely accepting it into an adapter memory queue.
• For the OpenGemini adapter, new measurements or tag series become query-visible after asynchronous index merging. When required, the application explicitly polls the expected timestamp, tags and values with a deadline; each adapter query sends one request without a transparent visibility retry.
• Synchronous does not mean globally serialized. Applications may call concurrently; IoTDB SessionPool or InfluxDB HTTP pools reuse connections. Prefer batch APIs for large writes to reduce network round trips.
• Slow databases or exhausted pools propagate latency, timeouts and exceptions to the caller. The adapter does not retain pending data, so there is no unflushed internal buffer to lose on application restart.
• Applications needing asynchronous burst handling, durable retries or reliable delivery should use an application executor, message queue or dedicated data pipeline. These components must choose queue capacity, rejection behavior, persistence, retry and deduplication according to business semantics.

page() and strictCursorPage() only support detail queries. Aggregation requires offset pagination through page(pageNum, pageSize). Aggregate results are stably ordered by window_start + groupByTags, or by groupByTags without a time window. Offset pagination issues an additional total-count query: detail queries count matching source rows, and grouped/aggregate queries count aggregate result rows.

• IoTDB requires the official ITableSessionPool; tsdb.iotdb.pool.enabled=false fails application startup.
• IoTDB requires a configured default database created in advance. It is set on TableSessionPoolBuilder; ordinary reads/writes avoid redundant USE statements, and the official client restores the default database when a cross-database Session closes.
• The previously tested IoTDB SDK 2.0.10 does not reliably replenish a pool slot if restoring the default database fails on Session close. The default database must remain present and the application account must retain access throughout runtime. SDK 2.0.11 still restores the default database when a borrowed Session closes; upgrading the client does not remove the requirement to keep that database available.
• IoTDB waits for a returned Session when the pool is full. Acquisition timeout is returned without replacing the physical pool; other connection exceptions during acquisition also do not trigger replacement. Query execution retains version-aware recovery for the known SDK empty-response defect and connection failures, retrying read-only execution at most once. This does not repair slot leaks during acquisition or add a semaphore across pool generations. When a fault causes pool replacement, draining the old pool and using the new pool can briefly overlap; max-size remains a per-physical-pool limit.
• IoTDB assembles batch Tablets by measurement. tsdb.iotdb.table.tablet-max-row-size controls each Tablet's maximum row count.
• IoTDB Tablet RPC compact encoding/compression is enabled by default through tsdb.iotdb.table.rpc-compression-enabled=true; omission keeps the default. Set it to false for older incompatible servers such as 2.0.2. This is not Thrift transport or disk compression. Verify at least 10 rows in one actual Tablet, rather than only the total application batch, and check the rows after writing. The same setting is used for initial and replacement pools; it does not introduce an automatic write retry or change commit-state reporting.
• For null IoTDB field values, types are inferred from non-null values in the same batch. Always-null columns are omitted. A table batch with no inferable field is rejected before I/O.
• IoTDB fields map precisely to INT32/INT64/FLOAT/DOUBLE/BOOLEAN/STRING/BLOB/DATE. Unsupported Java types are rejected before the first Tablet write.
• IoTDB / InfluxDB application batches default to at most 10000 records, controlled by each backend's max-batch-records.
• InfluxDB 3 validates and encodes the complete batch before splitting it into HTTP requests of at most 5000 lines or 1 MiB of UTF-8 payload. Writes use accept_partial=false.
• InfluxDB 1.x also validates the full batch and chunks requests, but HTTP 400 can mean some valid points were written. Such a response yields UNKNOWN; confirmed counts include only earlier successful requests. Potentially partially committed chunks are not automatically retried.
• Both InfluxDB adapters reject measurement names beginning with # to prevent silent data loss from line-protocol comment interpretation. Measurement = characters and backend-specific backslash rules are encoded appropriately.
• InfluxDB integer fields accept Byte/Short/Integer/Long and BigInteger within signed-int64 bounds. BigInteger uses the integer protocol suffix to avoid float64 rounding. Ambiguous Number subclasses are rejected.
• InfluxDB Float/Double values must be finite. InfluxDB 1.x accepts BigDecimal only when exactly representable as float64. InfluxDB 3 permits ordinary double rounding for BigDecimal but rejects overflow and nonzero underflow to zero.
• InfluxDB 1.x additionally rejects _field, _measurement and time as tag or field keys, avoiding data loss or rejection caused by reserved protocol names.
• A batch is not a database transaction across multiple Tablets or HTTP requests. Conversion or validation failure before I/O confirms zero writes for the whole batch. After I/O starts, inspect BatchWriteResult.commitState for successful, uncommitted, partially committed or unknown outcomes.
• Query/pagination limit/pageSize must be in 1–10000. Offset arithmetic overflow fails explicitly.
• Adapters write synchronously and maintain no asynchronous buffer. If incoming application traffic can exceed database or network write capacity, implement rate limiting, burst handling, retry and backpressure in the application or a dedicated data pipeline.

← Configuration and database selection · Fluent queries and aggregation →

The tested 1.4.1/1.5.2 Influx-compatible HTTP write path converts integer fields through float64 before storing them. An integer protocol suffix does not prevent this conversion: 9007199254740993i reads back as 9007199254740992. The OpenGemini adapter therefore checks the complete batch before I/O and accepts integer-typed fields only within signed-int64 bounds and when their exact value is representable by IEEE 754 float64. All integers from -2^53 to 2^53 are exact; some larger values, including 2^53+2 and Long.MIN_VALUE, are also exact. 2^53+1, Long.MAX_VALUE and Long.MIN_VALUE+1 are rejected with ARGUMENT_ERROR, NOT_COMMITTED and zero physical requests, without coercion or string encoding. Structured query/count also preflight exact integer filter literals, including every IN/BETWEEN value, because backend numeric comparisons can round them. Extreme integer-valued BigDecimal predicates are compared with signed-int64 bounds before integer expansion, so very large exponents cannot bypass validation or force unbounded allocation. Float/Double and fractional query literals retain ordinary floating-point semantics; existing BigDecimal field checks still apply. Epoch-millisecond timestamps use their separate time path. Native SQL and borrowed native writes bypass these checks and retain backend precision limitations; aggregate overflow and floating-point accumulation are not made exact by this validation. This restriction is specific to the HTTP adapter, not a claim about every OpenGemini engine or alternate ingestion protocol.

TsGate 2.1.0 includes the independent OpenGemini adapter and starter. See OpenGemini integration for configuration, default-engine limits and exact-version testing, and 2.1.0 release notes for release changes.