Available cluster options - silviucpp/erlcass GitHub Wiki

The cluster options can be set inside your app.config file under the cluster_options key:

{erlcass, [
    {log_level, 3},
    {keyspace, <<"keyspace">>},
    {cluster_options,[
        {contact_points, <<"172.17.3.129,172.17.3.130,172.17.3.131">>},       
        {latency_aware_routing, true},
        {token_aware_routing, true},
        {number_threads_io, 4},
        {queue_size_io, 128000},
        {core_connections_host, 1},
        {tcp_nodelay, true},
        {tcp_keepalive, {true, 60}},
        {connect_timeout, 5000},
        {request_timeout, 5000},
        {retry_policy, {default, true}},
        {default_consistency_level, 6}
    ]}
]}.

Tips for production environment:

  • Use token_aware_routing and latency_aware_routing

  • Don’t use number_threads_io bigger than the number of your cores.

  • Use tcp_nodelay and also enable tcp_keepalive

  • Don’t use large values for core_connections_host. The driver is system call bound and performs better with less I/O threads and connections because it can batch a larger number of writes into a single system call (the driver will naturally attempt to coallesce these operations). You may want to reduce the number of I/O threads to 2 or 3 and reduce the core connections to 1 (default).

Available options:

contact_points (Mandatory)

Example : {contact_points, <<"172.17.3.129">>}

Sets/Appends contact points. The first call sets the contact points and any subsequent calls appends additional contact points. Passing an empty string will clear the contact points. White space is striped from the contact points.

Accepted values: <<"127.0.0.1">>, <<"127.0.0.1,127.0.0.2">> or <<"server1.domain.com">>

port

Example: {port, 9042}

Sets the port.

Default: 9042

ssl

Example:

{ssl, [
            {trusted_certs, [<<"cert1">>, <<"cert2">>]},
            {cert, <<"cert_here">>},
            {private_key, {<<"private_key_here">>, <<"private_key_pwd_here">>}},
            {verify_flags, ?CASS_SSL_VERIFY_PEER_CERT}
        ]
}

Sets the SSL context and enables SSL.

Default: None

Params:
{ssl, [
        {trusted_certs, CertsList::list()},
        {cert, Cert::binary()},
        {private_key, {PrivateKey::binary(), KeyPassword::binary()}},
        {verify_flags, VerifyFlags::integer()}
       ]
}

  • trusted_certs : Adds one or more trusted certificate. This is used to verify the peer’s certificate.

  • cert : Set client-side certificate chain. This is used to authenticate the client on the server-side. This should contain the entire Certificate chain starting with the certificate itself.

  • private_key : Set client-side private key. This is used to authenticate the client on the server-side. PrivateKey is a key PEM formatted key string and KeyPassword is the password used to decrypt key

  • verify_flags : Sets verification performed on the peer’s certificate.

For verify_flags use one of the values defined in erlcass.hrl :

-define(CASS_SSL_VERIFY_NONE, 0).
-define(CASS_SSL_VERIFY_PEER_CERT, 1).
-define(CASS_SSL_VERIFY_PEER_IDENTITY, 2).

  • CASS_SSL_VERIFY_NONE - No verification is performed

  • CASS_SSL_VERIFY_PEER_CERT - Certificate is present and valid

  • CASS_SSL_VERIFY_PEER_IDENTITY - IP address matches the certificate’s common name or one of its subject alternative names. This implies the certificate is also present.

You can use also a combination like : ?CASS_SSL_VERIFY_PEER_CERT bor ?CASS_SSL_VERIFY_PEER_IDENTITY

Default: CASS_SSL_VERIFY_PEER_CERT

protocol_version

Example: {protocol_version, 2}

Sets the protocol version. This will automatically downgrade to the lowest protocol version supported.

Default: 4

number_threads_io

Example: {number_threads_io, 1}

Sets the number of IO threads. This is the number of threads that will handle query requests.

Default: 1

queue_size_io

Example: {queue_size_io, 8192}

Sets the size of the the fixed size queue that stores pending requests.

Default: 8192

queue_size_event (Removed starting v4.x)

Example: {queue_size_event, 8192}

Sets the size of the the fixed size queue that stores events.

Default: 8192

core_connections_host

Example: {core_connections_host, 1}

Sets the number of connections made to each server in each IO thread.

Default: 1

max_connections_host (Removed starting v4.x)

Example: {max_connections_host, 2}

Sets the maximum number of connections made to each server in each IO thread.

Default: 2

reconnect_wait_time (Removed starting v4.x)

Example: {reconnect_wait_time, 2000}

Sets the amount of time to wait before attempting to reconnect.

Default: 2000 milliseconds

max_concurrent_creation (Removed starting v4.x)

Example: {max_concurrent_creation, 1}

Sets the maximum number of connections that will be created concurrently. Connections are created when the current connections are unable to keep up with request throughput.

Default: 1

max_requests_threshold (Removed starting v4.x)

Example: {max_requests_threshold, 100}

Sets the threshold for the maximum number of concurrent requests in-flight on a connection before creating a new connection. The number of new connections created will not exceed max_connections_host.

Default: 100

requests_per_flush (Removed starting v4.x)

Example: {requests_per_flush, 128}

Sets the maximum number of requests processed by an IO worker per flush.

Default: 128

constant_reconnect (available starting v4.x)

Example: {constant_reconnect, 0}

Configures the cluster to use a reconnection policy that waits a constant time between each reconnection attempt. Time is specified in milliseconds. Use 0 to perform a reconnection immediately.

exponential_reconnect (available starting v4.x)

Example: {exponential_reconnect, {2000, 60000}}

Configures the cluster to use a reconnection policy that waits exponentially longer between each reconnection attempt; however will maintain a constant delay once the maximum delay is reached.

Note:

A random amount of jitter (+/- 15%) will be added to the pure exponential delay value. This helps to prevent situations where multiple connections are in the reconnection process at exactly the same time. The jitter will never cause the delay to be less than the base delay, or more than the max delay.

Params:

{exponential_reconnect, {BaseDelayMs, MaxDelayMs}}

  • BaseDelayMs - The base delay (in milliseconds) to use for scheduling reconnection attempts. (default: 2000)

  • MaxDelayMs - The maximum delay to wait between two reconnection attempts. (default: 60000)

coalesce_delay (available starting v4.x)

Example: {coalesce_delay, 200}

Sets the amount of time, in microseconds, to wait for new requests to coalesce into a single system call. This should be set to a value around the latency SLA of your application’s requests while also considering the request’s roundtrip time. Larger values should be used for throughput bound workloads and lower values should be used for latency bound workloads.

Default: 200

request_ratio (available starting v4.x)

Example: {request_ratio, 50}

Sets the ratio of time spent processing new requests versus handling the I/O and processing of outstanding requests. The range of this setting is 1 to 100, where larger values allocate more time to processing new requests and smaller values allocate more time to processing outstanding requests.

Default: 50

max_schema_wait_time (available starting v4.x)

Example: {max_schema_wait_time, 10000}

Sets the maximum time to wait for schema agreement after a schema change is made (e.g. creating, altering, dropping a table/keyspace/view/index etc).

Default: 10000

token_aware_routing_shuffle_replicas (available starting v4.x)

Example: {token_aware_routing_shuffle_replicas, true}

Configures token-aware routing to randomly shuffle replicas. This can reduce the effectiveness of server-side caching, but it can better distribute load over replicas for a given partition key.

Note:

Token-aware routing must be enabled for the setting to be applicable.

Default: true (enabled)

max_reusable_write_objects (available starting v4.x)

Example: {max_reusable_write_objects, 4294967295}

Sets the maximum number of "pending write" objects that will be saved for re-use for marshalling new requests. These objects may hold on to a significant amount of memory and reducing the number of these objects may reduce memory usage of the application. The cost of reducing the value of this setting is potentially slower marshalling of requests prior to sending.

Default: Max unsigned integer value (4294967295)

speculative_execution_policy

Example: {speculative_execution_policy, null}

Enable/disable constant speculative executions with the supplied settings.

Accepted values:

  • null will disable

  • `{ConstantDelayMs, MaxSpeculativeExecutions} will enable with this settings

connect_timeout

Example: {connect_timeout, 5000}

Sets the timeout for connecting to a node.

Default: 5000 milliseconds

heartbeat_interval

Example: {heartbeat_interval, 30}

Sets the amount of time between heartbeat messages and controls the amount of time the connection must be idle before sending heartbeat messages. This is useful for preventing intermediate network devices from dropping connections.

Default: 30 seconds

idle_timeout

Example: {idle_timeout, 60}

Sets the amount of time a connection is allowed to be without a successful heartbeat response before being terminated and scheduled for reconnection.

Default: 60 seconds

request_timeout

Example: {request_timeout, 12000}

Sets the timeout for waiting for a response from a node.

Default: 12000 milliseconds

credentials

Example: {credentials, {<<"username">>, <<"password">>}}

Sets credentials for plain text authentication.

load_balance_round_robin

Example: {load_balance_round_robin, true}

Configures the cluster to use round-robin load balancing. The driver discovers all nodes in a cluster and cycles through them per request. All are considered 'local'.

load_balance_dc_aware

Example: {load_balance_dc_aware, {"dc_name", 2, true}}

Configures the cluster to use DC-aware load balancing. For each query, all live nodes in a primary 'local' DC are tried first, followed by any node from other DCs.

Note:

This is the default, and does not need to be called unless switching an existing from another policy or changing settings. Without further configuration, a default local_dc is chosen from the first connected contact point, and no remote hosts are considered in query plans. If relying on this mechanism, be sure to use only contact points from the local DC.

Params:

{load_balance_dc_aware, {LocalDc, UsedHostsPerRemoteDc, AllowRemoteDcsForLocalCl}}*

  • LocalDc - The primary data center to try first

  • UsedHostsPerRemoteDc - The number of host used in each remote DC if no hosts are available in the local dc

  • AllowRemoteDcsForLocalCl - Allows remote hosts to be used if no local dc hosts are available and the consistency level is LOCAL_ONE or LOCAL_QUORUM

token_aware_routing

Example: {token_aware_routing, true}

Configures the cluster to use token-aware request routing, or not. This routing policy composes the base routing policy, routing requests first to replicas on nodes considered 'local' by the base load balancing policy.

Default is true (enabled).

latency_aware_routing

Example:

  • {latency_aware_routing, true}

  • {latency_aware_routing, {true, {2.0, 100, 10000, 100 , 50}}}

Configures the cluster to use latency-aware request routing, or not. This routing policy is a top-level routing policy. It uses the base routing policy to determine locality (dc-aware) and/or placement (token-aware) before considering the latency.

Params:

{Enabled, {ExclusionThreshold, ScaleMs, RetryPeriodMs, UpdateRateMs, MinMeasured}}

  • Enabled : State of the future

  • ExclusionThreshold - Controls how much worse the latency must be compared to the average latency of the best performing node before it penalized.

  • ScaleMs - Controls the weight given to older latencies when calculating the average latency of a node. A bigger scale will give more weight to older latency measurements.

  • RetryPeriodMs - The amount of time a node is penalized by the policy before being given a second chance when the current average latency exceeds the calculated threshold (ExclusionThreshold * BestAverageLatency).

  • UpdateRateMs - The rate at which the best average latency is recomputed.

  • MinMeasured - The minimum number of measurements per-host required to be considered by the policy.

Defaults: {false, {2.0, 100, 10000, 100 , 50}}

Note: In case you use only true false atom the tuning settings will not change.

tcp_nodelay

Example: {tcp_nodelay, false}

Enable/Disable Nagel’s algorithm on connections.

Default: true (disabled).

tcp_keepalive

Example: {tcp_keepalive, {true, 60}}

Enable/Disable TCP keep-alive

Default: false (disabled).

default_consistency_level

Example: {default_consistency_level, ?CASS_CONSISTENCY_LOCAL_QUORUM}

Set the default consistency level

Default: ?CASS_CONSISTENCY_LOCAL_QUORUM

retry_policy (erlcass >= 3.0)

Example: {retry_policy, {default, false}}

Retry polices allow the driver to automatically handle server-side failures when Cassandra is unable to fulfill the consistency requirement of a request.

Important: Retry policies do not handle client-side failures such as client-side timeouts or client-side connection issues. In these cases application code must handle the failure and retry the request. The driver will automatically recover requests that haven’t been written, but once a request is written the driver will return an error for in-flight requests and will not try to automatically recover. This is done because not all operations are idempotent and the driver is unable to distinguish which requests can automatically retried without side effect. It’s up to application code to make this distinction.

By default, the driver uses the default retry policy for all requests unless it is overridden.

Supported values:

default

This policy retries queries in the following cases: - On a read timeout, if enough replicas replied but data was not received. - On a write timeout, if a timeout occurs while writing the distributed batch log - On unavailable, it will move to the next host

In all other cases the error will be returned. This policy always uses the query’s original consistency level.

downgrading_consistency (Removed starting v4.x)

This policy may attempt to retry requests with a lower consistency level. Using this policy can break consistency guarantees.

This policy will retry in the same scenarios as the default policy, but it will also retry in the following cases:

  • On a read timeout, if some replicas responded but is lower than required by the current consistency level then retry with a lower consistency level.

  • On a write timeout, Retry unlogged batches at a lower consistency level if at least one replica responded. For single queries and batch if any replicas responded then consider the request successful and swallow the error.

  • On unavailable, retry at a lower consistency if at lease one replica responded.

The goal of this policy is to attempt to save a request if there’s any chance of success. A writes succeeds as long as there’s a single copy persisted and a read will succeed if there’s some data available even if it increases the risk of reading stale data.

fallthrough

This policy never retries or ignores a server-side failure. The error is always returned.

Logging retry policy

In order to log the retry decision of the picked policy using info level you can specify the policy as a tuple of {Policy::atom(), Logging :: boolean()}.

By default the logging is turned off.

cloud_secure_connection_bundle (erlcass >= v4.0.8)

Example: {cloud_secure_connection_bundle, <<"/path/to/your/bundle.zip">>}

Sets the secure connection bundle path for processing DBaaS credentials. This will pre-configure a cluster using the credentials format provided by the DBaaS cloud provider.

⚠️ **GitHub.com Fallback** ⚠️