Violet Rails: The Restarone Way - restarone/violet_rails GitHub Wiki
Monitoring and Telemetry
We use monitoring and telemetry to catch errors and triage, contributing to the general operational excellence of an organization.
Appointing system guardians
It is recommended you appoint an individual or team to receive Violet Rails error traces by registering them like so:
Dissecting an error trace
Once registered, they will receive error messages via email as they happen. A sample may look something like this:
restarone.solutionsgraphql#execute (ActiveRecord::StatementInvalid) "PG::UndefinedColumn: ERROR: column "time" does n...
An ActiveRecord::StatementInvalid occurred in graphql#execute:
PG::UndefinedColumn: ERROR: column "time" does not exist
LINE 1: ...ELECT "ahoy_visits".* FROM "ahoy_visits" ORDER BY time desc ...
^
-------------------------------
Request:
-------------------------------
* URL : https://www.restarone.solutions/graphql
* HTTP Method: POST
* IP address : 142.181.192.212
* Parameters : {"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDimension: \"time\") {\n id\n country\n city\n referringDomain\n }\n}", "variables"=>nil, "controller"=>"graphql", "action"=>"execute", "graphql"=>{"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDimension: \"time\") {\n id\n country\n city\n referringDomain\n }\n}", "variables"=>nil}}
* Timestamp : 2022-06-23 12:11:50 UTC
* Server : ip-172-31-86-57
* Rails root : /var/www/violet/releases/20220623114044
* Process: 2839150
-------------------------------
Session:
-------------------------------
* session id: [FILTERED]
* data: {"session_id"=>"4a19ecf6020e7c0feb26fa23c9c102ef",
"user_return_to"=>"/admin",
"warden.user.user.key"=>[[1], "$2a$12$89B5jU.44t5BXSkZNPRJqu"],
"warden.user.user.session"=>{"last_request_at"=>1655986310},
"site_id"=>1,
"_csrf_token"=>"sSPxmpKSVLNWc7QvjywHrvkJWSLQpGBMHJg3zeqIZVg="}
-------------------------------
Environment:
-------------------------------
* CONTENT_LENGTH : 169
* CONTENT_TYPE : application/json
* GATEWAY_INTERFACE : CGI/1.2
* HTTP_ACCEPT : */*
* HTTP_ACCEPT_ENCODING : gzip, deflate, br
* HTTP_ACCEPT_LANGUAGE : en-CA,en-US;q=0.9,en;q=0.8
* HTTP_CONNECTION : close
* HTTP_COOKIE : _r_solutions_session=WApsZfetZMdxoWS7HkHlrNqnvaaKHUSSSxn9vY1prhMfT9d%2FZz%2BXMRmxnNUCjAWsSC8AuzQmSzFQjvZi2FsrcjTgulD8jFeT6gkz55DbIRFsjfYH6nBSjjnMd7NvhuXySf%2BQtLovhZgBAhOMrJvQP0CbYGE0VxJa6ISKVBjZifjiUKbbXsy4J3QeHzBCjt%2BE1t0tYP4AlZxAEjRfK0GK5v4BxXDF49F91s1VWUD5A4EGM9crQg%2F3PJw7iIQYHaDvKcuF9gHdz7xc0RjEmVBawLlF9sDy1ehLc%2BTM0ynRNy6WTxd09Kuj%2Brc20%2Bepjy8NHk3a3i8Ab4vrPg6U1mztiHbjXhTg1ZvDCmM0ZBBiTmOdHAs5irSZcdpAGgwFa7SI8x%2BBTdS6CP6VW9l6pdJAgSXUwbL%2BXBk7JLX5k8qFMb6y4EVneyH%2BLFpHwgY8W0hiM1%2BYzhrAlp933nrFroeE5ng8K3Dy1368VICWSk5wmC9D31gL4rCawHcECu%2Fgr%2Bo5Qb2TS%2BfizPGavbN%2FNFUk2szjug0wP%2FbyxthHDrj9TSPYDTdKnjVBOIQB0dE%3D--H8TqpUcF6nQid9XH--EcvI8XO4sXCNdAepxxU0EQ%3D%3D; ahoy_visit=5baffb58-186f-4fb7-8e0a-b855245a9432; ahoy_visitor=1f6244c1-bf33-4801-b0ca-c7d89f4b8b4e
* HTTP_HOST : www.restarone.solutions
* HTTP_ORIGIN : https://www.restarone.solutions/
* HTTP_REFERER : https://www.restarone.solutions/admin/graphiql
* HTTP_USER_AGENT : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15
* HTTP_VERSION : HTTP/1.0
* HTTP_X_CSRF_TOKEN : [FILTERED]
* HTTP_X_FORWARDED_FOR : 142.181.192.212
* HTTP_X_FORWARDED_PROTO : https
* HTTP_X_REAL_IP : 142.181.192.212
* ORIGINAL_FULLPATH : /graphql
* ORIGINAL_SCRIPT_NAME :
* PATH_INFO : /graphql
* QUERY_STRING :
* RAW_POST_DATA : [FILTERED]
* REMOTE_ADDR : 127.0.0.1
* REQUEST_METHOD : POST
* REQUEST_PATH : /graphql
* REQUEST_URI : /graphql
* ROUTES_47032437798660_SCRIPT_NAME :
* SCRIPT_NAME :
* SERVER_NAME : www.restarone.solutions
* SERVER_PORT : 443
* SERVER_PROTOCOL : HTTP/1.1
* SERVER_SOFTWARE : puma 5.6.4 Birdie's Version
* action_controller.instance : #<GraphqlController:0x00007fd03c34f260>
* action_dispatch.authenticated_encrypted_cookie_salt : [FILTERED]
* action_dispatch.backtrace_cleaner : #<Rails::BacktraceCleaner:0x0000558d2c51e618>
* action_dispatch.content_security_policy :
* action_dispatch.content_security_policy_nonce_directives:
* action_dispatch.content_security_policy_nonce_generator :
* action_dispatch.content_security_policy_report_only : false
* action_dispatch.cookies : #<ActionDispatch::Cookies::CookieJar:0x00007fd03c4dacd8>
* action_dispatch.cookies_digest :
* action_dispatch.cookies_rotations : #<ActiveSupport::Messages::RotationConfiguration:0x0000558d2b244c40>
* action_dispatch.cookies_same_site_protection : #<Proc:0x0000558d2e735828@/var/www/violet/shared/bundle/ruby/2.6.0/gems/railties-6.1.5/lib/rails/application.rb:636>
* action_dispatch.cookies_serializer : json
* action_dispatch.encrypted_cookie_cipher : [FILTERED]
* action_dispatch.encrypted_cookie_salt : [FILTERED]
* action_dispatch.encrypted_signed_cookie_salt : [FILTERED]
* action_dispatch.http_auth_salt : [FILTERED]
* action_dispatch.key_generator : #<ActiveSupport::CachingKeyGenerator:0x0000558d2f160dc0>
* action_dispatch.logger : #<ActiveSupport::Logger:0x0000558d2c7c7db0>
* action_dispatch.parameter_filter : [:passw, :secret, :token, :_key, :crypt, :salt, :certificate, :otp, :ssn]
* action_dispatch.permissions_policy :
* action_dispatch.redirect_filter : []
* action_dispatch.remote_ip : 142.181.192.212
* action_dispatch.request.accepts : [#<Mime::Type:0x00007fd03c3530b8 @synonyms=[], @symbol=nil, @string="*/*", @hash=-3532471350324634667>]
* action_dispatch.request.content_type : application/json
* action_dispatch.request.formats : [#<Mime::Type:0x00007fd03c3530b8 @synonyms=[], @symbol=nil, @string="*/*", @hash=-3532471350324634667>]
* action_dispatch.request.parameters : {"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDimension: \"time\") {\n id\n country\n city\n referringDomain\n }\n}", "variables"=>nil, "controller"=>"graphql", "action"=>"execute", "graphql"=>{"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDi...
* action_dispatch.request.path_parameters : {:controller=>"graphql", :action=>"execute"}
* action_dispatch.request.query_parameters : {}
* action_dispatch.request.request_parameters : {"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDimension: \"time\") {\n id\n country\n city\n referringDomain\n }\n}", "variables"=>nil, "graphql"=>{"query"=>"{\n ahoyVisits(limit: 100, orderDirection:\"desc\", orderDimension: \"time\") {\n id\n country\n ...
* action_dispatch.request.unsigned_session_cookie : {"session_id"=>"4a19ecf6020e7c0feb26fa23c9c102ef", "user_return_to"=>"/admin", "warden.user.user.key"=>[[1], "$2a$12$89B5jU.44t5BXSkZNPRJqu"], "warden.user.user.session"=>{"last_request_at"=>1655986310}, "site_id"=>1, "_csrf_token"=>"[FILTERED]"}
* action_dispatch.request_id : 697ce9b3-2407-41a0-9dd6-3b3c553bc921
* action_dispatch.routes : #<ActionDispatch::Routing::RouteSet:0x0000558d2f7eee08>
* action_dispatch.secret_key_base : [FILTERED]
* action_dispatch.show_detailed_exceptions : false
* action_dispatch.show_exceptions : true
* action_dispatch.signed_cookie_digest :
* action_dispatch.signed_cookie_salt : [FILTERED]
* action_dispatch.use_authenticated_cookie_encryption : [FILTERED]
* action_dispatch.use_cookies_with_metadata : true
* exception_notifier.exception_data : {:current_user=>#<User id: 1, global_admin: true, created_at: "2021-04-12 23:11:27.966580000 +0000", updated_at: "2022-06-23 12:07:34.745535000 +0000", email: "[[email protected]](mailto:[email protected])", can_manage_web: true, can_manage_email: true, can_manage_users: true, can_manage_blog: true, name: "contact@restaron...
* puma.config : #<Puma::Configuration:0x0000558d29a312e8>
* puma.request_body_wait : 0
* puma.socket : #<TCPSocket:0x00007fd03c305db8>
* rack.after_reply : []
* rack.cors : #<Rack::Cors::Result:0x00007fd03c3051d8>
* rack.errors : #<IO:0x0000558d28b580c8>
* rack.hijack : #<Puma::Client:0x00007fd03c305d40>
* rack.hijack? : true
* rack.input : #<StringIO:0x00007fd03c3056d8>
* rack.multiprocess : false
* rack.multithread : true
* rack.request.cookie_hash : {"_r_solutions_session"=>"WApsZfetZMdxoWS7HkHlrNqnvaaKHUSSSxn9vY1prhMfT9d/Zz+XMRmxnNUCjAWsSC8AuzQmSzFQjvZi2FsrcjTgulD8jFeT6gkz55DbIRFsjfYH6nBSjjnMd7NvhuXySf+QtLovhZgBAhOMrJvQP0CbYGE0VxJa6ISKVBjZifjiUKbbXsy4J3QeHzBCjt+E1t0tYP4AlZxAEjRfK0GK5v4BxXDF49F91s1VWUD5A4EGM9crQg/3PJw7iIQYHaDvKcuF9gHdz7xc0RjEmV...
* rack.request.cookie_string : _r_solutions_session=WApsZfetZMdxoWS7HkHlrNqnvaaKHUSSSxn9vY1prhMfT9d%2FZz%2BXMRmxnNUCjAWsSC8AuzQmSzFQjvZi2FsrcjTgulD8jFeT6gkz55DbIRFsjfYH6nBSjjnMd7NvhuXySf%2BQtLovhZgBAhOMrJvQP0CbYGE0VxJa6ISKVBjZifjiUKbbXsy4J3QeHzBCjt%2BE1t0tYP4AlZxAEjRfK0GK5v4BxXDF49F91s1VWUD5A4EGM9crQg%2F3PJw7iIQYHaDvKcuF9gHdz7xc0RjEmVBawLlF9sDy1ehLc%2BTM0ynRNy6WTxd09Kuj%2Brc20%2Bepjy8NHk3a3i8Ab4vrPg6U1mztiHbjXhTg1ZvDCmM0ZBBiTmOdHAs5irSZcdpAGgwFa7SI8x%2BBTdS6CP6VW9l6pdJAgSXUwbL%2BXBk7JLX5k8qFMb6y4EVneyH%2BLFpHwgY8W0hiM1%2BYzhrAlp933nrFroeE5ng8K3Dy1368VICWSk5wmC9D31gL4rCawHcECu%2Fgr%2Bo5Qb2TS%2BfizPGavbN%2FNFUk2szjug0wP%2FbyxthHDrj9TSPYDTdKnjVBOIQB0dE%3D--H8TqpUcF6nQid9XH--EcvI8XO4sXCNdAepxxU0EQ%3D%3D; ahoy_visit=5baffb58-186f-4fb7-8e0a-b855245a9432; ahoy_visitor=1f6244c1-bf33-4801-b0ca-c7d89f4b8b4e
* rack.request.query_hash : {}
* rack.request.query_string :
* rack.run_once : false
* rack.session : #<ActionDispatch::Request::Session:0x00007fd03c333f88>
* rack.session.options : #<ActionDispatch::Request::Session::Options:0x00007fd03c333f38>
* rack.tempfiles : []
* rack.url_scheme : https
* rack.version : [1, 6]
* warden : Warden::Proxy:70266169958160 @config={:default_scope=>:user, :scope_defaults=>{}, :default_strategies=>{:user=>[:rememberable, :database_authenticatable]}, :intercept_401=>false, :failure_app=>#<Devise::Delegator:0x0000558d2d22ea10>}
-------------------------------
Data:
-------------------------------
* data: {:current_user=>
#<User id: 1, global_admin: true, created_at: "2021-04-12 23:11:27.966580000 +0000", updated_at: "2022-06-23 12:07:34.745535000 +0000", email: "[[email protected]](mailto:[email protected])", can_manage_web: true, can_manage_email: true, can_manage_users: true, can_manage_blog: true, name: "[[email protected]](mailto:[email protected])", moderator: true, can_view_restricted_pages: true, deliver_analytics_report: true, can_manage_api: true, can_manage_subdomain_settings: true, session_timeoutable_in: "1-week", can_access_admin: true, deliver_error_notifications: true, can_manage_analytics: true, can_manage_files: true>,
:current_visit=>
#<Ahoy::Visit:0x00007fd03c4fb190
id: 24986,
visit_token: "5baffb58-186f-4fb7-8e0a-b855245a9432",
visitor_token: "1f6244c1-bf33-4801-b0ca-c7d89f4b8b4e",
user_id: 1,
ip: "142.181.192.212",
user_agent:
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15",
referrer: nil,
referring_domain: nil,
landing_page: "https://www.restarone.solutions/",
browser: "Safari",
os: "Mac",
device_type: "Desktop",
country: "CA",
region: "Ontario",
city: "Toronto",
latitude: 43.6655,
longitude: -79.4378,
utm_source: nil,
utm_medium: nil,
utm_term: nil,
utm_content: nil,
utm_campaign: nil,
app_version: nil,
os_version: nil,
platform: nil,
started_at: Thu, 23 Jun 2022 12:07:23.782883000 UTC +00:00>}
Important error data (user + visit)
We see the user who encountered the error
* data: {:current_user=>
#<User id: 1, global_admin: true, created_at: "2021-04-12 23:11:27.966580000 +0000", updated_at: "2022-06-23 12:07:34.745535000 +0000", email: "[[email protected]](mailto:[email protected])", can_manage_web: true, can_manage_email: true, can_manage_users: true, can_manage_blog: true, name: "[[email protected]](mailto:[email protected])", moderator: true, can_view_restricted_pages: true, deliver_analytics_report: true, can_manage_api: true, can_manage_subdomain_settings: true, session_timeoutable_in: "1-week", can_access_admin: true, deliver_error_notifications: true, can_manage_analytics: true, can_manage_files: true>,
we also see their visit
#<Ahoy::Visit:0x00007fd03c4fb190
id: 24986,
visit_token: "5baffb58-186f-4fb7-8e0a-b855245a9432",
visitor_token: "1f6244c1-bf33-4801-b0ca-c7d89f4b8b4e",
user_id: 1,
ip: "142.181.192.212",
user_agent:
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15",
referrer: nil,
referring_domain: nil,
landing_page: "https://www.restarone.solutions/",
browser: "Safari",
os: "Mac",
device_type: "Desktop",
country: "CA",
region: "Ontario",
city: "Toronto",
latitude: 43.6655,
longitude: -79.4378,
utm_source: nil,
utm_medium: nil,
utm_term: nil,
utm_content: nil,
utm_campaign: nil,
app_version: nil,
os_version: nil,
platform: nil,
started_at: Thu, 23 Jun 2022 12:07:23.782883000 UTC +00:00>}
Sensible tuning for your mental health
Not all errors need to be reported, so we suppress errors that are a fact of life for a web server on the internet. See the errors we suppress here: https://github.com/restarone/violet_rails/blob/master/config/environments/production.rb#L136-L141
Change management
Change management is the process we use to create, deploy and manage changes on a Violet Rails system.
CMS
In the CMS, each page can be part of 3 releases. The current release (named after the public path), the staged release (next version, named after the public path + name + -staging) and a backup of the current release (named after the public path-release-date-etc`)
Example
Here we see for the page / (that is named root) the current release is named root and is mapped to /. A new page root-staging has been created that maps to /root-staging. Right before cutting over, we will copy the contents of the root page (in raw HTML format, not with WSIWYG encoding) and create a new page named 'root-pre-etc' that will map to /root-pre-etc.
Incident response
If you find a Violet Rails system unresponsive. Please follow these steps to debug and remediate.
Make sure Violet Images are being backed up
For customers running on AWS, we use Lifecycle manager to grab snapshots of EC2 instances to create AMI's. These AMI's can be used to launch fresh instances of an app
Customer First!
Do not try to debug on a dead/dying system. Respond to the issue by launch a fresh instance to restore service, instead of trying to debug and fix in real time.
Check if NGINX is working
Check the NGINX access log, make sure the requests are hitting the reverse proxy
tail -f /var/log/nginx/access.log
If the access log looks good, check the error log
tail -f /var/log/nginx/error.log
Check if Puma is working
sudo systemctl status puma.service
Check if Sidekiq is working
sudo systemctl status sidekiq.service
Restart services if they are in a bad state
sudo systemctl restart puma.service
sudo systemctl restart sidekiq.service
sudo systemctl restart nginx