FAQ U - ibmcb/cbtool GitHub Wiki

Use

UQ1: What is the difference between --soft_reset and --hard_reset?

UA1: "Soft Reset" kill all running CBTOOL daemons (list of daemons here) and cleans up the Object Store. "Hard Reset" goes beyond that by also removing any old log files from the Log Store, and removes all the collected performance data from the Metric Store. The cleanup of the Metric Store is kind of a big deal. If you didn't extract all values from the Metric Store to comma-separated value files using the command monextract, you will lose experiment data permanently.

UQ2: I see that CBTOOL has a GUI. Where can I learn more?

UA2: Basic information on how administrating the tool with the Graphical Environment. You can monitor the statistics collected by CBTOOL on the dashboard tab. The latter can be customized.

UQ3 How can I quickly check if performance data is being properly generated by CBTOOL?

UA3: Just deploy one or more instances of a Virtual Application (VApp) via the CLI (e.g., aiattach nullworkload). Once the prompt returns, wait a couple of minutes and type monlist VM. You should a list of VMs which reported management, runtime/OS and runtime/Application metrics. It should include the number of samples per VM, as well as the age (i.e., how long ago it was written on the Metric Store) of the latest sample.

UQ4 What is the experiment identifier?

UA4: The experiment identifier or (expid) is simply an arbitrary name that a CBTOOL user/experiment can assign to an Experiment (check the Key Concepts and Objects). By the default, the expid is simply the string EXP_ followed by the current (start) date. The current expid can be obtained via the CLI/API call expid, and new one can be set with expid <NEW EXPERIMENT IDENTIFIER>.

UQ5 How can I extract/save the performance data generated by CBTOOL?

UA5: Issue the CLI command monextract all, will instruct CB to get all the performance data from the Metric Store and dump it on four comma-separated value files (.csv), under a newly created directory named after the experiment (i.e., ~/cbtool/data/<experiment_identifier>). These files are briefly describe as follows:

VM_management_<experiment_identifier>.csv: contains information regarding the provisioning time, capture time, among others, all extracted directly from the cloud management system.
VM_runtime_app_<experiment_identifier>.csv: contains information regarding the application performance, such as latency, throughput and bandwidth, all generated directly from the VM. Note: a more detailed explanation of the columns on this file can be found here
VM_runtime_os_<experiment_identifier>.csv: contains OS metrics (CPU, memory, disk I/O, network I/O) all generated by the VM.
HOST_runtime_os_<experiment_identifier>.csv: contains OS metrics (CPU, memory, disk I/O, network I/O) all generated by the HOSTS. It can only be used on Clouds where there is direct access to the hosts (this means that this data cannot be collected on EC2, for instance), and where the Ganglia monitoring tool was manually configured on the hosts.

USEFUL TIP: in order to quickly check the sanity of the files directly on a text-only shell, install tabview (sudo pip3 install tabview) and issue (outside CB CLI, of course) a command such as tabview --width max --delimiter , ~/cbtool/data/experiment_identifier/VM_runtime_app_experiment_identifier.csv

UQ6: Do you have any utilities for automatic data plot generation?

UA6: Yes, the CLI command monextract all, in addition to the four comma-separated value files (.csv) described in the previous question, will also create a helper script, ~/cbtool/data/<experiment_identifier>/plot.sh. This script invokes the CBTOOL - experimental GNU R code - ~/cbtool/util/plot/cbplotgen.R which will produce multiple graphs in PDF format, as exemplified here.

UQ7: What is the difference between the attributes NETNAME, RUN_NETNAME and PROV_NETNAME?

UA7: The attribute NETNAME indicates the name of an already existing (virtual) network on a cloud. It is specified under the section [USER-DEFINED : CLOUDOPTION_MY<CLOUD MODEL>] on your private configuration file (usually while preparing your cloud to be driven by CBTOOL). CBTOOL will assign the value of this single parameter to both RUN_NETAME and PROV_NETNAME (specified under the section [VM_DEFAULTS : <MODEL>_CLOUDCONFIG]). The first parameter indicates which virtual network will be used by all instances which are part of a Virtual Application Instance to exchange application data. The second indicates which virtual network will be used by the instances to communicate (in/out) with the CBTOOL Orchestrator Node (ON).

An illustrative example of a situation where these two different networks can be used simultaneously in can be seen here (i.e, a Public cloud where the instances have both public and private IP addresses, and the ON is outside of the cloud). More examples of the separation between "provisioning" (PROV_NETNAME) and "running" (RUN_NETNAME) can be found on this how to

IMPORTANT : keep in mind that these parameters can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ8: Does CBTOOL support cloud-init?

UA8:Yes. The first versions of CBTOOL ignored the use of cloud-init due to its lack of maturity, but given its current status a "de-facto" standard in cloud, it is currently fully supported. A list of Cloud Adapters with support for this feature can be found here. CBTOOL will implicitly use cloud-init for the following:

Automatically and idempotently create the user specified by the VM-specific attribute LOGIN. Controlled by the attribute USERDATA under the section [VM_DEFAULTS : <MODEL>_CLOUDCONFIG] (automatically set to "True" on all cloud models/adapters where cloud-init is supported)
Automatically inject the contents of the public ssh key specified by the VM-specific attribute SSH_KEY_NAME. Controlled by the attribute USERDATA_SSH under the section [VM_DEFAULTS : <MODEL>_CLOUDCONFIG] (automatically set to "True" on all cloud models/adapters where cloud-init is supported)
Optionally, install additional packages. A list of semicolon-separated package names can be specified via the attribute CLOUDINIT_PACKAGES, under the section [VM_DEFAULTS].
Optionally execute additional commands. A list of semicolon-separated commands can be specified via the attribute CLOUDINIT_COMMANDS, under the section [VM_DEFAULTS].
Optionally, inject additional public keys. A list of semicolon-separated key contents (important, all blank spaces need to be converted into ____) can be specified via the attribute CLOUDINIT_KEYS, under the section [VM_DEFAULTS].

IMPORTANT : keep in mind that these parameters can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ9: How can I quickly check if CBTOOL is properly configured to deploy VMs in my cloud`?

UA9: Once a cloud is attached, take note of the name or identifier (imageid) for at least one base (unconfigured) Ubuntu or RHEL/Centos/Fedora image to be used later for the creation of the workloads (e.g., "ami-a9d276c9" on EC2's "us-west-2" or "ubuntu-1604-xenial-v20161221" on Google Compute Engine"). You can now try the following commands:

vmattach check:<imageid>: if successful, it will ensure that CB can interact with the cloud, a deploy a running instance.
vmattach check:<imageid>:cbuser: if successful, it will ensure that CB can interact with the cloud, deploy an running instance and then contact such instance over the network. Please note that cbuser is the name of the user which will automatically created on the instance, via cloud-init, as discussed in the previous question. If you are unsure of the proper operation of cloud-init on your cloud, or attempting to check the functionality on a cloud which does not support it, replace cbuser with the name of an user which you know exists on the image being deployed.

UQ10: I just want the instantiate VMs, but don't want them to run any application. I just want to see the cloud's provisioning performance with "zero application load". Can I do that with CBTOOL?

UA10: Yes. Just deploy one or more instances of the Virtual Application (VApp) called NullWorkload. On the CLI, type aiattach nullworkload

UQ11: How can I quickly check the number of VMs failed/deployed?

UA11: Just use the stats command, and look for the counters under the "------------------ EXPERIMENT-WIDE COUNTERS ------------------" section.

UQ12: I have a VM image that was configured to run a certain application with a username different than the username that is indicated on the VM role template. Is there a way for me to specify a per-Virtual Application-type login for each VM "role"?

UA12: Any role-specific parameter can be changed from the default (size, imageid) or added (e.g., an image-specific login) in multiple different ways:

Just re-define the VM role attributes directly on your private private configuration file, on the [VM_TEMPLATES] Configuration (Global) Object, just as described here, adding the parameter login

Example:

[VM_TEMPLATES]
CASSANDRA = size:04-16384, imageid1:cb_ycsb, login:cbuser
SEED = size:04-16384, imageid1:cb_ycsb, login:cbuser

Just use the typealter command to specify a different login for each VM "role" on your Virtual Application. Also described in further detail here

For instance, suppose that the VM with role "db2" has to be accessed using the user "db2user", while the other VM "roles" ("client_daytrader", "was") on the Virtual Application "ibm_daytrader" can be accessed by the user "causer". By issuing the command typealter ibm_daytrader db2_login db2user, CBTOOL will make sure that db2 VMs will always be contacted using this particular user.

Finally, the specific parameters can be dynamically overridden during a VApp deployment. For instance, the command aiattach ibm_daytrader default default none none none db2_login=db2user,was_login=appuser will override whatever is defined on the here or what was defined through the typealter command.

IMPORTANT : please note that these three ways are simply a particular case of the generalized parameter change methods here and here.

UQ13: Can I extract collected performance data directly from the Metric Store?

UA13: Yes. Take a look at this how to

UQ14: How can I experiment with VM image capture on a cloud?

UA14: Assuming that your cloud supports the "capture" operation from within CBTOOL - the support is clearly specified in the supported clouds list - individual Virtual Machines (i.e. VMs that do not belong to any VApp) can be captured with the command vmcapture VM_NAME, while Virtual Applications can be captured with the command aicapture AI_NAME. Please note that in the case of the Virtual Applications, CBTOOL will capture only one of the VMs that are part of the VApp, just destroying all others in the process.

UQ15: I know that CBTOOL keeps polling the cloud in order to determine when VMs become "active" (i.e., the instance started the booting process). How can I control the polling frequency?

UA15: Both the number of polling attempts (for each VM creation) and the interval between attempts are controlled by the UPDATE_FREQUENCY and UPDATE_ATTEMPTS attributes on the configuration object VM_DEFAULTS. If you want to change these, just add the following entry to your private configuration file:

[VM_DEFAULTS]
UPDATE_FREQUENCY = SECONDS BETWEEN STATUS REQUESTS FROM THE CLOUD
UPDATE_ATTEMPTS = NUMBER OF TIMES STATUS REQUESTS ARE MADE BEFORE DECLARING THAT VM FAILED TO START

Please note that a similar attribute pair exists for other configuration objects, such as AI_DEFAULTS (e.g., how many time should an application start script attempted to be executed before declaring a Virtual Application deployment as failed) and AIDRS_DEFAULTS (e.g., what is the interval between checks if a given Virtual Application already past its lifetime). The default value for all parameter pairs on each configuration object is taken for MAIN_UPDATE_FREQUENCY and MAIN_UPDATE_ATTEMPTS, specified in the USER-DEFINED section of your private configuration file.

IMPORTANT : keep in mind that this parameter can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ16: How can I control the operation (e.g., vmattach) parallelism in CBTOOL?

UA16: In CBTOOL specific operations, executed over specific objects, can be configured with different degrees of parallelism. For instance, to set the parallelism for operations against Virtual Application instances, the following parameters can be set in your private configuration file (here the default values for a generic cloud are shown) ...

[VM_DEFAULTS]
DETACH_PARALLELISM = 20
RUNSTATE_PARALLELISM = 5

[AI_DEFAULTS]
ATTACH_PARALLELISM = 1
DETACH_PARALLELISM = 20
EXECUTE_PARALLELISM = 6
RUNSTATE_PARALLELISM = 5

[AIDRS_DEFAULTS]
DETACH_PARALLELISM = 20
DAEMON_PARALLELISM = 50

...

IMPORTANT : keep in mind that this parameter can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ17: How can I debug vmattach errors in the post-instance start phase?

UA17: First of all, instruct CBTOOL to deploy the instances without attempting to establishing contact to these, by running the command vmdev. After that, just run vmattach <VM ROLE NAME> once. CBTOOL will output each command that would have been executed against the VM. At this point, you can just execute the commands yourself and check for error messages. Once done, do not forget to disable the debug mode by issuing the command vmundev

NOTE: There is a small utility, called "cbssh", that could be used to directly connect to the VMs. To try it, just run - on a bash prompt - cd ~/cbtool; ~/cbtool/cbssh vm_1, and you should be able to login on the instance.

Additional detailed debug instructions

UQ18: How can I debug aiattach errors in the post-instance start phase?

UA18: First of all, instruct CBTOOL to deploy the Virtual Application instances without attempting to run the configuration scripts on its multiple VMs, by running the command appdev. After that, just run aiattach <AI TYPE NAME> once. CBTOOL will output each command that would have been executed against each one of the VMs that compose the VApp. At this point, you can just execute the command yourself and check what could be wrong. IMPORTANT:Please note that some commands should be executed in parallel (or at least in a non-sequential manner) and therefore might require multiple prompts for overlapping execution. Once done, do not forget to disable the debug mode by issuing the command appundev

Additional detailed debug instructions

UQ19: The Virtual Application Instance deploys properly, but no application performance metrics samples are generated. How can I debug "application load" (i.e., load manager) errors in the post-Application Instance start phase?

UA19: First of all, instruct CBTOOL to fully deploy the Virtual Application instance, but do not start the actual load generation by running the command appnoload. After that, just run aiattach <AI TYPE NAME> once. At the very end, CBTOOL will output a message such as **Load Manager will NOT be automatically started on <VM NAME> during the deployment of <VAPP NAME>...**. After the Virtual Application attachment completes, just move to another terminal and login on the instance name with cd ~cbtool; ~/cbtool/cbssh <VM NAME>, and then run /usr/local/bin/cbloadman debug. Once done, do not forget to disable the debug mode by issuing the command appload

Additional detailed debug instructions

UQ20: What are the attributes IMAGE_PREFIX and IMAGE_SUFFIX?

UA20: These attributes, specified under the section [VM_DEFAULTS : <MODEL>_CLOUDCONFIG] on private configuration file will automatically append a prefix/suffix , to all image names (attribute imageid1) as defined for each VM role

IMPORTANT : keep in mind that these parameters can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ21: Can CBTOOL execute generic actions before/during/after deploying VMs or VApps ?

UA21: Yes, CBTOOL can execute generic scripts at multiple points during the deployment of VMs or Virtual Applications. The name of the script to be executed is specified by the attribute (applicable to both individual VMs or whole AIs) execute_script_name. This script is invoked with a single paramater: the name of a json file (name after the object's UUID under /tmp) containing a dump of the object's (i.e., VM or VApp) attributes (meaning that it is up to the script to parse the object for specific attribute values). The decision on the exact point in time where the script will be executed on is controlled by the (again, applicable to both individual VMs or whole AIs) staging. So, if one wants to execute an script even before a VM is created (for instance one wants to create a new virtual network even before an instance is created), a CLI/API call such as vmattach <VM ROLE> staging=execute_provision_originated will trigger the execution of the script specified by execute_script_name very early in the deployment process.

It is expected that the script return a zero exit code. Fail to do so will cause CBTOOL to abort the deployment of the VM or VApp. It should be noted that these scripts can modify the CBTOOL object's attributes values. All that is required is that the scripts standard output to be in the format <attribute>:<value>, with multiple attributes comma-separated.

The possible values for staging are execute_provision_originated, execute_provision_started, execute_provision_complete, execute_deprovision_finished. For VApps an additional possibility is the all_vms_booted, to indicate a script which has to be execute after all VMs which forms a whole AI are ready, but the application/workload configuration/startup hasn't started yet.

An illustrative example of the use of this functionality is available on the "OpenStack-specific" section of this FAQ.

IMPORTANT : keep in mind that these parameters (execute_script_name and staging) can be changed in any of the 3 ways discussed here and here (i.e., by changing the private configuration file, by issuing the CLI/API cldalter vm_defaults <parameter> <value> or by dynamically overriding the parameters during an vmattach or aiattach CLI/API call).

UQ22: How can I experiment with "fault injection" on a cloud?

UA22: TBD

UQ23: How can I access the GUI on a remote Orchestrator Node?

UA23: A very common use case for CBTOOL - especially when experimenting on Public Clouds - is the establishment of an Orchestrator Node (ON) directly on a Region/Availability Zone. In this case, it is not uncommon that only method of access to the ON is via SSH. For this case, you can still access the GUI with a simple SSH tunnel. Please keep in mind that the port where the GUI listens on is controlled by the attribute PORT under [GUI_DEFAULTS] (default is 8080), and the whole URL for the GUI is displayed during CBTOOL's startup.

From you workstation just do: ssh -L <GUI PORT>:localhost:<GUI PORT> <IP ORCHESTRATOR>. After it connects, just point your workstation's browser to http://127.0.0.1:<GUI PORT>

UQ24: How can I debug the (starting of the process which implements) the GUI?

UA24: Whenever CB CLI is invoked (i.e., simply typing ~/cbtool/cb, even without --soft_reset or --hard-reset), a message will be printed out indicating the PID of the process which implements both the CB GUI and CB API. Example:

Checking for a running API service daemon.....API Service daemon was successfully started. The process id is ['72579'] (http://192.168.122.1:7070).
Checking for a running GUI service daemon.....GUI Service daemon was successfully started. The process id is ['72645', '72646'], listening on port 8080. Full url is "http://192.168.122.1:8080".

It is important to note that the GUI process runs inside a screen virtual terminal, named cbgui<USERNAME>. During the initial start of CB (i.e. when CB was invoked with the options --soft_reset or --hard_reset) a small helper script, named /tmp/restart_cb_gui_<USERNAME> was created. In ordem to restart the GUI in debug mode just do:

sed -i 's/sudo pgrep/pgrep/g' /tmp/restart_cb_gui_<USERNAME> (there is a small bug on the generation of this script, which will be fixed soon)
/tmp/restart_cb_gui_<USERNAME> debug (you might have to execute it twice)
Leave the process running in foreground and make sure you can now access the GUI
Once the normal operation, you can restart the process inside the screen session, with /tmp/restart_cb_gui_<USERNAME>

UQ25: How can I SSH into a newly created Virtual Machine?

UA25: First obtain a list of currently deployed VM by issuing the command vmlist on the CLI. Once you selected a particular VM, exit the CLI (or just go to a separated terminal window), do cd cbtool, and issue the command ./cbssh <VM NUMBER>, where <VM_NUMBER> is in the format vm_<x>.

UQ26: Can I skip the VM snapshots? Avoid preparing workload images?

UA26: Yes: We have a (nearly) zero-configuration docker-based solution for you: Are you impatient? Use our workloads instead.

⚠️ GitHub.com Fallback ⚠️