Presto JMX on EMR - qyjohn/AWS_Tutorials GitHub Wiki
Introduction
The Java Management Extensions (JMX) framework was introduced in Java 1.5. It provides a way to observe what happens in the JVM, either running locally or on a remote computer. We recommend you to read Trail: Java Management Extensions (JMX) in The Java Tutorial to understand how JMX works.
Presto offers a JMX Connector, allowing observation on Presto via JMX.
Environment Setup
Launch an EMR cluster with PrestoSQL. SSH into the master node. Edit/Create Presto configuration file /etc/presto/conf/catalog/jmx.properties, with the following content:
connector.name=jmx
jmx.dump-tables=java.lang:type=Runtime,com.facebook.presto.execution.scheduler:name=NodeScheduler
jmx.dump-period=10s
jmx.max-entries=86400
Restart Presto:
sudo systemctl restart presto-server
Take a quick look at the JMX tables in Presto:
$ presto-cli
presto> USE jmx.current;
USE
presto:current> SHOW TABLES;
Table
-----------------------------------------------------------------------------------------------------------------------------------
com.sun.management:type=diagnosticcommand
com.sun.management:type=hotspotdiagnostic
... ...
presto.type:name=typeoperatorscache
sun.nio.ch:context=http_1_1_h2c@20d3fff4,id=0,type=serversocketchannelimpl
(275 rows)
Query 20210801_221858_00003_y89gw, FINISHED, 3 nodes
Splits: 36 total, 36 done (100.00%)
0.27 [275 rows, 23.9KB] [1.02K rows/s, 88.7KB/s]
presto:current> DESC "presto.execution:name=queryexecution";
Column | Type | Extra | Comment
-----------------------------------+---------+-------+---------
executor.activecount | bigint | |
executor.allowcorethreadtimeout | boolean | |
executor.completedtaskcount | bigint | |
executor.corepoolsize | bigint | |
executor.keepalivetime | varchar | |
executor.largestpoolsize | bigint | |
executor.maximumpoolsize | bigint | |
executor.poolsize | bigint | |
executor.queuedtaskcount | bigint | |
executor.rejectedexecutionhandler | varchar | |
executor.shutdown | boolean | |
executor.taskcount | bigint | |
executor.terminated | boolean | |
executor.terminating | boolean | |
node | varchar | |
object_name | varchar | |
(16 rows)
Query 20210801_222349_00004_y89gw, FINISHED, 3 nodes
Splits: 36 total, 36 done (100.00%)
0.42 [16 rows, 1.71KB] [38 rows/s, 4.07KB/s]
presto:current> DESC "presto.execution:name=querymanager";
Column | Type | Extra | Comment
---------------------------------------------------+---------+-------+---------
abandonedqueries.fifteenminute.count | double | |
abandonedqueries.fifteenminute.rate | double | |
... ...
wallinputbytesrate.oneminute.percentiles | varchar | |
wallinputbytesrate.oneminute.total | double | |
node | varchar | |
object_name | varchar | |
(303 rows)
Query 20210801_222653_00007_y89gw, FINISHED, 3 nodes
Splits: 36 total, 36 done (100.00%)
0.25 [303 rows, 34.7KB] [1.19K rows/s, 137KB/s]
presto:current> SELECT queuedqueries FROM "presto.execution:name=querymanager";
queuedqueries
---------------
0
(1 row)
Query 20210801_222905_00008_y89gw, FINISHED, 2 nodes
Splits: 17 total, 17 done (100.00%)
0.24 [1 rows, 8B] [4 rows/s, 33B/s]
presto:current> exit
JMX Term
JMX Term is an open source command line based interactive JMX client written in Java. It lets user access a Java MBean server in command line console without graphical environment. In this section, we briefly introduce how to use JMX Term to observe what happens in Presto.
Download JMX Term:
wget https://github.com/jiaqi/jmxterm/releases/download/v1.0.2/jmxterm-1.0.2-uber.jar
Start JMX Term:
$ java -jar jmxterm-1.0.2-uber.jar
Delete /home/hadoop/.jmxterm_history if you encounter error right after launching me.
Welcome to JMX terminal. Type "help" for available commands.
$>help
#following commands are available to use:
about - Display about page
bean - Display or set current selected MBean.
beans - List available beans under a domain or all domains
bye - Terminate console and exit
close - Close current JMX connection
domain - Display or set current selected domain.
domains - List all available domain names
exit - Terminate console and exit
get - Get value of MBean attribute(s)
help - Display available commands or usage of a command
info - Display detail information about an MBean
jvms - List all running local JVM processes
open - Open JMX session or display current connection
option - Set options for command session
quit - Terminate console and exit
run - Invoke an MBean operation
set - Set value of an MBean attribute
subscribe - Subscribe to the notifications of a bean
unsubscribe - Unsubscribe the notifications of an earlier subscribed bean
watch - Watch the value of one MBean attribute constantly
$>open localhost:9080
#Connection to localhost:9080 is opened
$>beans
#domain = JMImplementation:
JMImplementation:type=MBeanServerDelegate
#domain = com.sun.management:
com.sun.management:type=DiagnosticCommand
com.sun.management:type=HotSpotDiagnostic
#domain = io.airlift.discovery.client:
io.airlift.discovery.client:name=Announcer
io.airlift.discovery.client:name=ServiceInventory
... ...
presto.type:name=TypeOperatorsCache
#domain = sun.nio.ch:
sun.nio.ch:context=HTTP_1_1_h2c@20d3fff4,id=0,type=serversocketchannelimpl
$> bean presto.execution:name=QueryManager
#bean is set to presto.execution:name=QueryManager
$>info
#mbean = presto.execution:name=QueryManager
#class name = io.prestosql.dispatcher.DispatchManager
# attributes
%0 - AbandonedQueries.FifteenMinute.Count (double, r)
%1 - AbandonedQueries.FifteenMinute.Rate (double, r)
%2 - AbandonedQueries.FiveMinute.Count (double, r)
%3 - AbandonedQueries.FiveMinute.Rate (double, r)
%4 - AbandonedQueries.OneMinute.Count (double, r)
... ...
$>get QueuedQueries
#mbean = presto.execution:name=QueryManager:
QueuedQueries = 0;
$>exit
#bye
Syabru Nagios JMX Plugin
The Syabru Nagios JMX Plugin is a JMX monitoring plugin for Nagios. It is a tool that is very convenient if you just want to quickly get the value of a particular attribute exposed by JMX.
Download the plugin:
wget https://snippets.syabru.ch/nagios-jmx-plugin/download/nagios-jmx-plugin.zip
unzip nagios-jmx-plugin.zip
cd nagios-jmx-plugin-1.2.3
Do a simple query for HeapMemoryUsage:
$ java -jar check_jmx.jar -U service:jmx:rmi:///jndi/rmi://localhost:9080/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used
JMX OK - HeapMemoryUsage.used = 335378592 | 'HeapMemoryUsage used'=335378592;;;;
Do a simple query for QueuedQueries:
$ java -jar check_jmx.jar -U service:jmx:rmi:///jndi/rmi://localhost:9080/jmxrmi -O presto.execution:name=QueryManager -A QueuedQueries
JMX OK - QueuedQueries = 0 | 'QueuedQueries'=0;;;;
In practice, you can use JMX Term to see what is being exposed by JMX, then use Syabru Nagios JMX Plugin to develop scripts for automation.