Active Record Connections Management
Important
GitHub Wiki is just a mirror of our online documentation.
We highly recommend using our website docs due to Github Wiki limitations. Only some illustrations, links, screencasts, and code examples will work here, and the formatting may be broken.
Please use https://karafka.io/docs.
Karafka interacts with ActiveRecord in the context of database connection management. This integration is designed to ensure efficient resource use and stability during message processing.
Rails, with ActiveRecord, employs a connection pooling mechanism to manage database connections. Connection pooling aims to reuse existing connections, avoiding the overhead of establishing new connections for every database interaction. This mechanism is particularly beneficial in web applications and background job processing, where efficient database connections can significantly impact performance and scalability.
In a typical Rails application, ActiveRecord automatically manages database connections. It checks out connections from the pool when needed and returns them after the request or job is completed. However, in the context of background processing frameworks like Karafka, especially when dealing with database replicas, additional steps, as mentioned above, might be necessary to ensure connections are properly managed.
When no database replication is involved, Karafka automatically manages ActiveRecord database connections. This means that after processing messages, Karafka releases the connections back to the ActiveRecord connection pool, preventing potential connection leakage and ensuring connections are available for other processes or threads that might need them.
In database replication scenarios, Karafka requires manual intervention to ensure connections are appropriately handled. Specifically, for each database replica, connections need to be manually released back to the pool. This is achieved by subscribing to the worker.completed
event, which signals the completion of a worker's messages processing task. Implementing a handler for this event allows for explicit connection management, ensuring that resources are correctly managed and reducing the risk of connection saturation.
::Karafka::App.monitor.subscribe('worker.completed') do
rails7plus = Rails.gem_version >= Gem::Version.new('7.0.0')
# Replace this with the proper reference to the replica connections
if rails7plus
ActiveRecord::ReplicaBase.connection_handler.clear_active_connections!
else
ActiveRecord::ReplicaBase.clear_active_connections!
end
end
Karafka provides automatic database connection management for standard setups. However, when using database replicas, it's crucial to manage those connections to maintain system performance and stability manually. This process involves subscribing to Karafka's worker.completed
event and explicitly releasing connections, ensuring they are available for subsequent use.
Home | Getting started | Components | Producing Messages | Consuming messages | Concurrency and multithreading | Error handling and back off policy
This wiki is tracked by git and publicly editable. You are welcome to fix errors and typos. Any defacing or vandalism of content will result in your changes being reverted and you being blocked.