Bean Fault Tolerance - AvanzaBank/astrix GitHub Wiki

Astrix includes a fault-tolerance layer implemented on top of Hystrix which can be used to protect invocations to an Astrix bean with fault isolation. The fault tolerance layer is activated by adding the astrix-fault-tolerance module to the classpath. If the astrix-fault-tolerance module is present, then all invocations to a service bean will be protected by the fault-tolerance layer without further configuration, and library beans might be protected by explicitly annotating the bean definition with @AstrixFaultToleranceProxy.

Astrix applies two different types of protection using either HystrixCommand or HystrixObservableCommand depending on the return type of the invoked Astrix bean method. Most invocations (so called "synchronous" invocations) are protected by a regular HystrixCommand using a thread-pool for isolation. Astrix may also handle a bean invocation as "reactive", see below for definition. For reactive bean invocations Astrix uses a HystrixObservableCommand and semaphores for protection.

Each Astrix bean has a dedicated bulk-head, i.e thread-pool or semaphore depending on the type of bean invocation as described above.

Reactive invocations

Astrix may treat a bean invocation as "reactive" depending on the return type of the invoked method. All methods returning a java.util.concurrent.CompletableFuture, rx.Observable or com.gigaspaces.async.AsyncFuture are treated as reactive by default. It's possible to extend the reactive type handling to include more types by implementing a ReactiveTypeHandlerPlugin, see Extending Astrix for information about how to apply a custom ReactiveTypeHandlerPlugin.

Configuration

Astrix registers a custom HystrixPropertiesStrategy. The custom strategy resolves all hystrix properties associated with a bean invocation by reading the AstrixBeanSettings associated with the given bean. This means that all hystrix configuration is read from AstrixBeanSettings. Note that its NOT possible to configure the Hystrix settings associated with a Astrix bean invocation using the default Hystrix configuration mechanism described in the Hystrix documentation on configuration.

AstrixBeanSettings related to Hystrix configuration

AstrixBeanSetting	Default Value	Description
TIMEOUT (astrix.bean.[beanKey].faultTolerance.timeout)	1000 [ms]	The maximum time to wait for a response from a bean invocation before throwing a ServiceUnavailableException
MAX_CONCURRENT_REQUESTS (astrix.bean.[beanKey].faultTolerance.maxConcurrentRequests)	20	Defines the "maxConcurrentRequests" when semaphore isolation is used to protect invocations to a bean, i.e. the maximum number of concurrent requests before the fault-tolerance layer starts rejecting invocations by throwig a ServiceUnavailableException
CORE_SIZE (astrix.bean.[beanKey].faultTolerance.coreSize)	10	Defines the "coreSize" when thread isolation is used to protect invocations to a bean, i.e. the number of threads in the bulkhead for a synchronous bean invocation.
QUEUE_SIZE_REJECTION_THRESHOLD (astrix.bean.[beanKey].faultTolerance.queueSizeRejectionThreshold)	10	Defines the initial "queueSizeRejectionThreshold" for the queue when thread isolation is used to protect invocations to a bean. When the number of pending bean invocations reaches the "queueSizeRejectionThreshold", any subsqeuent invocation to the bean will immediatly throw a ServiceUnavailableException until the queue has decreased below in the queue to a thread-pool (bulkhead) before starting to reject invocations.

Configuration example

# Defines maxConcurrentRequests for the unqualfied bean "se.avanzabank.trading.TradingService"
astrix.bean.se.avanzabank.trading.TradingService.faultTolerance.maxConcurrentRequests=20

# Defines maxConcurrentRequests for the bean "org.openspaces.core.GigaSpace" with qualifier "trading-space"
astrix.bean.org.openspaces.core.GigaSpace-trading-space.faultTolerance.maxConcurrentRequests=20

Enabling/disabling fault tolerance at runtime

Fault tolerance may be enabled/disabled at runtime for all beans, or individual beans, in a AstrixContext. In order for a bean invocation to be protected by a fault-tolerance proxy, fault tolerance must be enabled both globally and for the individual bean using these settings:

AstrixBeanSetting	Default Value	Description
FAULT_TOLERANCE_ENABLED (astrix.bean.[beanKey].faultTolerance.enabled)	true	Determines whether invocations on a bean should be protected by fault tolerance. Applies to all service beans, and also to library beans whose definition are annotated with @AstrixFaultToleranceProxy.

AstrixSetting	Default Value	Description
ENABLE_FAULT_TOLERANCE (AstrixContext.enableFaultTolerance)	true	Globally disables/enables the fault tolerance proxy for all beans in the associated AstrixContext

Overriding DefaultBeanSettings in service definition

The default settings in all tables above might be overriden in the bean definition for an individual Astrix bean by using the @DefaultBeanSettings annotation, see the javadoc for DefaultBeanSettings.