HOWTO: Extract performance data directly from the Metric Store - ibmcb/cbtool GitHub Wiki

As discussed elsewhere, the metrics collected by CBTOOL can be grouped in three categories: Management performance metrics (management), Application performance metrics (runtime_app), and OS resource usage metrics (runtime_os). Unsurprisingly the data is stored in the Metric Store (a MongoDB data store) in collections whose names contain the management, runtime_app or runtime_os string.

The data structure of the Metric Store is the following:

> show dbs

local (empty)
metrics 5.951171875GB
test (empty)

> use metrics

switched to db metrics

> show collections

latest_management_VM_msilva
latest_runtime_os_HOST_msilva
management_HOST_msilva
management_VM_msilva
reported_management_VM_metric_names_msilva
reported_runtime_app_VM_metric_names_msilva
reported_runtime_os_HOST_metric_names_msilva
reported_runtime_os_VM_metric_names_msilva
runtime_app_VM_msilva
runtime_os_HOST_msilva
runtime_os_VM_msilva
system.indexes
trace_msilva

Please note that the name formation rule for each collection is PREFIX____CATEGORY___OBJECTUSERNAME.

Possible values for PREFIX can be either reported, latest or nothing (i.e., empty). Collections prefixed by reported_ and latest_ are used only by the CloudBench GUI, and can be safely ignored by CBTOOL users.
Possible values for CATEGORY are (as discussed), management, runtime_app or runtime_os.
Possible values for OBJECT are VM or HOST.
Finally, by suffixing the collections with the USERNAME, multiple CBTOOL users can share the same MongoDB database.

The important collections, containing the actual data that can be extract and analyzed are then the ones that do not have a prefix (i.e. prefix is an empty string). It is important to keep in mind that, each individual collection holds data pertaining multiple experiments (that is why is so important to set an experiment id early in your experiment file through the command expid).

Collection have internal arrangements that vary according to its category. For instance, the collection management_VM_msilva contains one document per VM (i.e. each VM produces a single "sample"), containing all the information regarding the VM provisioning/deprovisioning timing. An illustrative example:

> db.management_VM_msilva.find({"expid" : "mynewexperiment"})

{ "_id" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "imageid1" : "cb_nullworkload", "expid" : "mynewexperiment", "host_cloud_ip" : "9.2.223.20", "last_known_state" : "ACTIVE with ip assigned", "ai" : "336B60C4-0095-5EC0-8731-9FE4B4A90049", "mgt_001_provisioning_request_originated" : "1400017863", "vmc_name" : "RegionOne", "size" : "m1.tiny", "obj_type" : "VM", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "pattern" : "none", "mgt_002_provisioning_request_sent" : "1", "cloud_ip" : "10.10.0.5", "state" : "attached", "role" : "tinyvm", "mgt_004_network_acessible" : "15", "cloud_hostname" : "cb-msilva-TESTOPENSTACK-vm1-tinyvm", "type" : "nullworkload", "ai_name" : "ai_1", "mgt_003_provisioning_request_completed" : "11", "aidrs" : "none", "cloud_name" : "TESTOPENSTACK", "mgt_006_application_start" : "5", "aidrs_name" : "none", "name" : "vm_1", "vmc_cloud_ip" : "10.10.0.3", "mgt_005_file_transfer" : "2", "host_name" : "ngp-c1-virt-9", "model" : "osk" }
{ "_id" : "4F3A2A3F-A5EA-56B6-99AA-4013F56AC6BE", "imageid1" : "cb_netperf", "expid" : "mynewexperiment", "host_cloud_ip" : "9.2.223.21", "last_known_state" : "ACTIVE with ip assigned", "ai" : "DDD497F0-A9D2-5532-B97B-B6DAF03E3D72", "mgt_001_provisioning_request_originated" : "1400018004", "vmc_name" : "RegionOne", "size" : "m1.medium", "obj_type" : "VM", "uuid" : "4F3A2A3F-A5EA-56B6-99AA-4013F56AC6BE", "pattern" : "none", "mgt_002_provisioning_request_sent" : "1", "cloud_ip" : "10.10.0.7", "state" : "attached", "role" : "netclient", "mgt_004_network_acessible" : "5", "cloud_hostname" : "cb-msilva-TESTOPENSTACK-vm2-netclient", "type" : "netperf", "ai_name" : "ai_2", "mgt_003_provisioning_request_completed" : "11", "aidrs" : "none", "cloud_name" : "TESTOPENSTACK", "mgt_006_application_start" : "0", "aidrs_name" : "none", "name" : "vm_2", "vmc_cloud_ip" : "10.10.0.3", "mgt_005_file_transfer" : "1", "host_name" : "ngp-c1-virt-10", "model" : "osk" }
{ "_id" : "666CC54B-329F-5946-BCA7-DACF17EB495B", "imageid1" : "cb_netperf", "expid" : "mynewexperiment", "host_cloud_ip" : "9.2.223.20", "last_known_state" : "ACTIVE with ip assigned", "ai" : "DDD497F0-A9D2-5532-B97B-B6DAF03E3D72", "mgt_001_provisioning_request_originated" : "1400018004", "vmc_name" : "RegionOne", "size" : "m1.tiny", "obj_type" : "VM", "uuid" : "666CC54B-329F-5946-BCA7-DACF17EB495B", "pattern" : "none", "mgt_002_provisioning_request_sent" : "1", "cloud_ip" : "10.10.0.6", "state" : "attached", "role" : "netserver", "mgt_004_network_acessible" : "10", "cloud_hostname" : "cb-msilva-TESTOPENSTACK-vm3-netserver", "type" : "netperf", "ai_name" : "ai_2", "mgt_003_provisioning_request_completed" : "11", "aidrs" : "none", "cloud_name" : "TESTOPENSTACK", "mgt_006_application_start" : "0", "aidrs_name" : "none", "name" : "vm_3", "vmc_cloud_ip" : "10.10.0.3", "mgt_005_file_transfer" : "2", "host_name" : "ngp-c1-virt-9", "model" : "osk" }

On other hand, the collection runtime_app_VM_msilva contains multiple documents per VM (i.e. each VM produces multiple "samples"), with each document containing the performance reported by the Virtual Application's Load Manager for a single execution with a certain LOAD_LEVEL for a certain LOAD_DURATION (i.e., a LOAD_ID as shown in the leftmost part of the bottom figure here). Another illustrative example:

> db.runtime_app_VM_msilva.find({"expid" : "mynewexperiment"})
...
{ "_id" : ObjectId("53729459fbe8e91ae9542b38"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "68" }, "time" : 1400018009, "app_throughput" : { "units" : "tps", "val" : "11.12" }, "app_bandwidth" : { "units" : "mbps", "val" : "12.56" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "3" }, "app_latency" : { "units" : "msec", "val" : "2.308" }, "time_h" : "Tue May 13 21:53:29 UTC 2014" }
{ "_id" : ObjectId("53729477fbe8e91d54635c12"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "87" }, "time" : 1400018039, "app_throughput" : { "units" : "tps", "val" : "13.90" }, "app_bandwidth" : { "units" : "mbps", "val" : "15.70" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "4" }, "app_latency" : { "units" : "msec", "val" : "2.885" }, "time_h" : "Tue May 13 21:53:59 UTC 2014" }
{ "_id" : ObjectId("53729496fbe8e91fbf33f75a"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "40" }, "time" : 1400018070, "app_throughput" : { "units" : "tps", "val" : "16.68" }, "app_bandwidth" : { "units" : "mbps", "val" : "18.84" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "5" }, "app_latency" : { "units" : "msec", "val" : "3.462" }, "time_h" : "Tue May 13 21:54:30 UTC 2014" }
{ "_id" : ObjectId("537294b5fbe8e9222a258375"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "96" }, "time" : 1400018101, "app_throughput" : { "units" : "tps", "val" : "19.46" }, "app_bandwidth" : { "units" : "mbps", "val" : "21.98" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "6" }, "app_latency" : { "units" : "msec", "val" : "4.039" }, "time_h" : "Tue May 13 21:55:01 UTC 2014" }
{ "_id" : ObjectId("537294d3fbe8e924967f2cf4"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "10" }, "time" : 1400018131, "app_throughput" : { "units" : "tps", "val" : "22.24" }, "app_bandwidth" : { "units" : "mbps", "val" : "25.12" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "7" }, "app_latency" : { "units" : "msec", "val" : "4.616" }, "time_h" : "Tue May 13 21:55:31 UTC 2014" }
{ "_id" : ObjectId("537294f2fbe8e92701dbb420"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "28" }, "time" : 1400018162, "app_throughput" : { "units" : "tps", "val" : "25.02" }, "app_bandwidth" : { "units" : "mbps", "val" : "28.26" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "8" }, "app_latency" : { "units" : "msec", "val" : "5.193" }, "time_h" : "Tue May 13 21:56:02 UTC 2014" }
{ "_id" : ObjectId("53729510fbe8e9296e231696"), "expid" : "mynewexperiment", "uuid" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "app_load_duration" : { "units" : "sec", "val" : "30" }, "app_load_level" : { "units" : "load", "val" : "99" }, "time" : 1400018192, "app_throughput" : { "units" : "tps", "val" : "27.80" }, "app_bandwidth" : { "units" : "mbps", "val" : "31.40" }, "app_load_profile" : { "units" : "name", "val" : "default" }, "app_load_id" : { "units" : "seqnum", "val" : "9" }, "app_latency" : { "units" : "msec", "val" : "5.770" }, "time_h" : "Tue May 13 21:56:32 UTC 2014" }
{ "_id" : ObjectId("5372952ff6c40d19a835133f"), "expid" : "mynewexperiment", "uuid" : "4F3A2A3F-A5EA-56B6-99AA-4013F56AC6BE", "app_load_duration" : { "units" : "sec", "val" : "83" }, "app_load_level" : { "units" : "load", "val" : "1" }, "app_bandwidth" : { "units" : "Mbps", "val" : "795.46" }, "app_load_profile" : { "units" : "name", "val" : "tcp_stream" }, "app_load_id" : { "units" : "seqnum", "val" : "2" }, "time" : 1400018223, "time_h" : "Tue May 13 21:57:03 UTC 2014" }
...

Finally the collection trace_USERNAME simply contains the list of commands/API calls that were issued against this CBTOOL Orchestrator Node. Again, illustrative example using trace_msilva:

> db.trace_msilva.find({"expid" : "mynewexperiment"})
...
{ "_id" : ObjectId("53729477cad020798186bf44"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018039, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK space", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372947bcad020798186bf45"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018043, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK metricstore", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372947bcad020798186bf46"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018043, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK time", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372947bcad020798186bf47"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018043, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK time", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372947ecad020798186bf48"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "128A7C32-9DC4-52A2-8CEF-91545A22EB43", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018046, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "vmshow TESTOPENSTACK 128A7C329DC452A28CEF91545A22EB43 all", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729480cad020798186bf49"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "c216c070-7908-5d43-bfe4-b5fbfc7c5a9e", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018048, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "hostshow TESTOPENSTACK c216c07079085d43bfe4b5fbfc7c5a9e all", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729480cad020798186bf4a"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "610a27b0-5965-5ba6-b9b1-88aa616290b4", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018048, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "hostshow TESTOPENSTACK 610a27b059655ba6b9b188aa616290b4 all", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729481cad020798186bf4b"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018049, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK space", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729482cad020798186bf4c"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018050, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK metricstore", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729482cad020798186bf4d"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "1", "vm_reservations" : "3", "command_originated" : 1400018050, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK time", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("53729482cad020798186bf4e"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "2", "vm_reservations" : "3", "command_originated" : 1400018050, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK time", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372948bcad020798186bf4f"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "cloud_TESTOPENSTACK", "ai_arrived" : "2", "vm_reservations" : "3", "command_originated" : 1400018059, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "cldshow TESTOPENSTACK space", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
{ "_id" : ObjectId("5372948bcad020798186bf50"), "vm_departed" : "0", "expid" : "mynewexperiment", "vm_failed" : "0", "name" : "c216c070-7908-5d43-bfe4-b5fbfc7c5a9e", "ai_arrived" : "2", "vm_reservations" : "3", "command_originated" : 1400018059, "vmc_departed" : "0", "ai_reservations" : "2", "dashboard_polled" : false, "ai_failed" : "0", "vmc_failed" : "0", "vm_arriving" : "0", "vm_arrived" : "3", "command" : "hostshow TESTOPENSTACK c216c07079085d43bfe4b5fbfc7c5a9e all", "ai_arriving" : "0", "vmc_arrived" : "1", "ai_departed" : "0" }
...