metadata cache - dianaclarke/openstack-notes GitHub Wiki

Problem description:

Currently the metadata service retrieves data unneccessarily from the conductor in at least two separate cases:

where the same data was already retrieved and cached by another metadata service worker running on the same node

when a retrieval for that same data is already in flight

This unneccessary load on the conductor could be reduced by using a shared cache across all workers running on the same node, and recording when a retrieval for some URL is already in progress so that if that same data is requested again before the initial fetch has completed, then the subsequent request can simply await it's arrival in the shared cached as opposed to independently re-fetching it in parallel.

Initial context:
- Red Hat Bug #1302413 - optimize metadata service caching logic to avoid unnecessary data retrieval from conductor
- Red Hat Bug #1244852 - openstack-nova-api metadata service is surprisingly slow with memcached
- Bug #1549814 - Metadata caching is fundamentally broken
- Patch - Don't use memcached for caching metadata for now
- Patch - Avoid parallel DB queries of identical metadata
- Commit - Avoid lazy-loads in metadata requests
- Commit - Replace conductor get_ec2_ids() with new Instance.ec2_ids attribute
- "I think that not stashing the whole instance in the cache is probably a reasonable change to make, although I wouldn't prioritize it unless there's something compelling" --irc
What is the metadata service?

Compute uses a metadata service for virtual machine instances to retrieve instance-specific data. The metadata service supports two sets of APIs: an OpenStack metadata API and an EC2-compatible API. http://docs.openstack.org/admin-guide-cloud/compute-networking-nova.html#metadata-service

What is config-drive?

You can configure OpenStack to write metadata to a special configuration drive that attaches to the instance when it boots. The instance can mount this drive and read files from it to get information that is normally available through the metadata service. http://docs.openstack.org/user-guide/cli_config_drive.html
What is the metadata test coverage like at the moment?

Coverage for nova/api/metadata/__init__.py : 100%
Coverage for nova/api/metadata/base.py : 90%
Coverage for nova/api/metadata/handler.py : 88%
Coverage for nova/api/metadata/password.py : 90%
Coverage for nova/api/metadata/vendordata_json.py : 58%
Coverage for nova/api/openstack/compute/image_metadata.py : 92%

Misc Links (as I troubleshoot):
- http://blog.oddbit.com/2014/01/14/direct-access-to-nova-metadata/
Play with the metadata service on devstack (master) for the first time:
- No need to do anything to devstack, the metadata API is enabled by default
- Create an instance:

[diana@localhost devstack]$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID                                   | Name       | Status | Task State | Power State | Networks         |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| 5a7ad0e0-c43c-4973-9d45-db7d734b31ee | instance-1 | ACTIVE | -          | Running     | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+
[diana@localhost devstack]$ nova show 5a7ad0e0-c43c-4973-9d45-db7d734b31ee
+--------------------------------------+----------------------------------------------------------------+
| Property                             | Value                                                          |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                           |
| OS-EXT-AZ:availability_zone          | nova                                                           |
| OS-EXT-SRV-ATTR:host                 | localhost.localdomain                                          |
| OS-EXT-SRV-ATTR:hostname             | instance-1                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | localhost.localdomain                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                                              |
| OS-EXT-SRV-ATTR:kernel_id            | 2d00f4e0-7d63-4ef5-b8bc-b53309c57168                           |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                              |
| OS-EXT-SRV-ATTR:ramdisk_id           | 833d02c4-257f-4bcc-93ef-f1253ea9248a                           |
| OS-EXT-SRV-ATTR:reservation_id       | r-if7nrs04                                                     |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                                                       |
| OS-EXT-SRV-ATTR:user_data            | -                                                              |
| OS-EXT-STS:power_state               | 1                                                              |
| OS-EXT-STS:task_state                | -                                                              |
| OS-EXT-STS:vm_state                  | active                                                         |
| OS-SRV-USG:launched_at               | 2016-03-17T15:56:59.000000                                     |
| OS-SRV-USG:terminated_at             | -                                                              |
| accessIPv4                           |                                                                |
| accessIPv6                           |                                                                |
| config_drive                         | True                                                           |
| created                              | 2016-03-17T15:56:26Z                                           |
| description                          | instance-1                                                     |
| flavor                               | m1.nano (42)                                                   |
| hostId                               | 9413437868d0fceb8830a22725926417b3e0a9f8136ecabbd0a463ad       |
| host_status                          | UP                                                             |
| id                                   | 5a7ad0e0-c43c-4973-9d45-db7d734b31ee                           |
| image                                | cirros-0.3.4-x86_64-uec (3e6394eb-05a8-49f8-b97b-98431af17425) |
| key_name                             | -                                                              |
| locked                               | False                                                          |
| metadata                             | {}                                                             |
| name                                 | instance-1                                                     |
| os-extended-volumes:volumes_attached | []                                                             |
| private network                      | 10.0.0.2                                                       |
| progress                             | 0                                                              |
| security_groups                      | default                                                        |
| status                               | ACTIVE                                                         |
| tenant_id                            | 3331f97515194082ab1b17a106a14846                               |
| updated                              | 2016-03-17T15:56:59Z                                           |
| user_id                              | 9fc7088873c24cb3befb20aaf95e7b53                               |
+--------------------------------------+----------------------------------------------------------------+

ssh to the instance and query the metadata:

[diana@localhost devstack]$ ssh [email protected]
[email protected]'s password: 
$ curl http://169.254.169.254/openstack/latest/meta_data.json
{
  "random_seed": "pilKA8+oQBELT8E/bGz0JZgd8TXcirypbSURplefnVPr8wm2mA4uB0VxCike9/XS8GNkUgnxOV6XafBq4Gauqep7tVVkyC+H1Mv52j0PuzML4TJlA5ZAcv7HHvmv/LEE8/9z2miza8g4TN9uFcv1Db47efQefzj2DGrnCA8y+WXSMj3TM0edwPXezP1kgB2wcY6FwLrHZT55+geCKiTjepvy810J5hWy0b9MOZrIu0T31ZtdGHK3/mqmpwVDkbfu8YBJcq0uJhF4KFGjo61PpKY1rIqB0CISXiPC56aApq6WDenM2OPrvVLIgCz0chIxjTIEpIUpLX+yid7MI4I2a021ztoA9QkpMMQOhU3kM5fyomdc4vBTrD1LKoLz9viCRSsdEdqsyv2UqtJwCrGSO60pOv8iVa0qWGvOwY65CD2ifOGHSISsvJkAfZtA6JszGKnlrU+1sWI3TLB3ecg84Qbc/Jc1XGwvLVXZR+wWjUQKxhb3sHTvIeObGWEWOS5M83CrxFK54c4pHGmh5OpMVNgTBozENkHBKoG0SzPRbpC+Z6acqemhyhW0+WlLrFRXjfki9LTE+L/KRZzQLIhCJvYrT0BdtIeKa653ctUShjJr9rPxLyex2SdM5l0Aar8hFE6X1KyaXRn9wTeq+mL7IqMdM/s+2S5Qcnyj8oBO4Mw=", 
  "uuid": "5a7ad0e0-c43c-4973-9d45-db7d734b31ee", 
  "availability_zone": "nova", 
  "hostname": "instance-1.novalocal",   
  "launch_index": 0, 
  "project_id": "3331f97515194082ab1b17a106a14846", 
  "name": "instance-1"
}

3 Queries:

2016-03-17 13:41:38.419 DEBUG nova.metadata.wsgi.server [req-f6772642-278f-4a32-9776-6f74fd5011f7 None None] (4796) accepted ('10.0.0.2', 49210) from (pid=4796) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2016-03-17 13:41:38.450 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT 1
2016-03-17 13:41:38.451 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {}
2016-03-17 13:41:38.452 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] BEGIN (implicit)
2016-03-17 13:41:38.453 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT fixed_ips.created_at AS fixed_ips_created_at, fixed_ips.updated_at AS fixed_ips_updated_at, fixed_ips.deleted_at AS fixed_ips_deleted_at, fixed_ips.deleted AS fixed_ips_deleted, fixed_ips.id AS fixed_ips_id, fixed_ips.address AS fixed_ips_address, fixed_ips.network_id AS fixed_ips_network_id, fixed_ips.virtual_interface_id AS fixed_ips_virtual_interface_id, fixed_ips.instance_uuid AS fixed_ips_instance_uuid, fixed_ips.allocated AS fixed_ips_allocated, fixed_ips.leased AS fixed_ips_leased, fixed_ips.reserved AS fixed_ips_reserved, fixed_ips.host AS fixed_ips_host
FROM fixed_ips
WHERE fixed_ips.deleted = %(deleted_1)s AND fixed_ips.address = %(address_1)s
 LIMIT %(param_1)s
2016-03-17 13:41:38.454 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {u'param_1': 1, u'address_1': '10.0.0.2', u'deleted_1': 0}
2016-03-17 13:41:38.457 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] ROLLBACK
2016-03-17 13:41:38.481 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT 1
2016-03-17 13:41:38.481 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {}
2016-03-17 13:41:38.483 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] BEGIN (implicit)
2016-03-17 13:41:38.489 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %(deleted_1)s AND instances.uuid = %(uuid_1)s
 LIMIT %(param_1)s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %(deleted_2)s LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %(deleted_3)s AND security_groups_1.deleted = %(deleted_4)s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %(deleted_5)s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %(deleted_6)s LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid
2016-03-17 13:41:38.491 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {u'uuid_1': '5a7ad0e0-c43c-4973-9d45-db7d734b31ee', u'deleted_6': 0, u'deleted_5': 0, u'deleted_4': 0, u'param_1': 1, u'deleted_3': 0, u'deleted_1': 0, u'deleted_2': 0}
2016-03-17 13:41:38.509 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] ROLLBACK
2016-03-17 13:41:38.514 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT 1
2016-03-17 13:41:38.514 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {}
2016-03-17 13:41:38.515 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] BEGIN (implicit)
2016-03-17 13:41:38.516 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info
FROM block_device_mapping
WHERE block_device_mapping.deleted = %(deleted_1)s AND block_device_mapping.instance_uuid = %(instance_uuid_1)s
2016-03-17 13:41:38.516 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] {u'deleted_1': 0, u'instance_uuid_1': '5a7ad0e0-c43c-4973-9d45-db7d734b31ee'}
2016-03-17 13:41:38.518 INFO sqlalchemy.engine.base.Engine [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] ROLLBACK
2016-03-17 13:41:38.525 INFO nova.metadata.wsgi.server [req-c57a9037-23f4-418c-ab4b-ef387419b432 None None] 10.0.0.2 "GET /openstack/latest/meta_data.json HTTP/1.1" status: 200 len: 1031 time: 0.0830131

It does use the cache if you query it again before it expires. No additional database queries are executed for cache hits.

    2016-03-17 14:00:27.887 DEBUG nova.api.metadata.handler [-]
Using cached metadata for 10.0.0.2 from (pid=4797) get_metadata_by_remote_address
/opt/stack/nova/nova/api/metadata/handler.py:85

  - https://github.com/openstack/nova/blob/3ad46e3c7888f533745d4ba71fd3d4095d24e242/nova/api/metadata/handler.py#L85

A metadata call takes between 0.04s and 0.08s, for cache hits and misses

Now play with the metadata service on devstack (liberty):
- Create an instance:

[diana@localhost devstack]$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID                                   | Name       | Status | Task State | Power State | Networks         |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| d1542f40-9b9a-41a1-ac5d-3ec277f9d984 | instance-2 | ACTIVE | -          | Running     | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+
[diana@localhost devstack]$ nova show d1542f40-9b9a-41a1-ac5d-3ec277f9d984
+--------------------------------------+----------------------------------------------------------------+
| Property                             | Value                                                          |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                           |
| OS-EXT-AZ:availability_zone          | nova                                                           |
| OS-EXT-SRV-ATTR:host                 | localhost.localdomain                                          |
| OS-EXT-SRV-ATTR:hostname             | instance-2                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | localhost.localdomain                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                                              |
| OS-EXT-SRV-ATTR:kernel_id            | dd252ba8-3242-4728-9c47-3429fb430bd1                           |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                              |
| OS-EXT-SRV-ATTR:ramdisk_id           | 8f4bca54-3fd4-4994-9aea-9286cbc1ec61                           |
| OS-EXT-SRV-ATTR:reservation_id       | r-i419t9nu                                                     |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                                                       |
| OS-EXT-SRV-ATTR:user_data            | -                                                              |
| OS-EXT-STS:power_state               | 1                                                              |
| OS-EXT-STS:task_state                | -                                                              |
| OS-EXT-STS:vm_state                  | active                                                         |
| OS-SRV-USG:launched_at               | 2016-03-17T19:20:35.000000                                     |
| OS-SRV-USG:terminated_at             | -                                                              |
| accessIPv4                           |                                                                |
| accessIPv6                           |                                                                |
| config_drive                         | True                                                           |
| created                              | 2016-03-17T19:20:04Z                                           |
| flavor                               | m1.nano (42)                                                   |
| hostId                               | c9e6f8ee8ae28cc69cf0d1ee4d37a59fcec7d12e79cfad513c955ab6       |
| id                                   | d1542f40-9b9a-41a1-ac5d-3ec277f9d984                           |
| image                                | cirros-0.3.4-x86_64-uec (7dd2249f-52f2-44c6-af02-196d683dd8a6) |
| key_name                             | -                                                              |
| metadata                             | {}                                                             |
| name                                 | instance-2                                                     |
| os-extended-volumes:volumes_attached | []                                                             |
| private network                      | 10.0.0.2                                                       |
| progress                             | 0                                                              |
| security_groups                      | default                                                        |
| status                               | ACTIVE                                                         |
| tenant_id                            | a379783483154bc3b93932b70f04b07f                               |
| updated                              | 2016-03-17T19:20:35Z                                           |
| user_id                              | 7fd9a53ef9034b68ae7745c47f8edc0b                               |
+--------------------------------------+----------------------------------------------------------------+

ssh to the instance and query the metadata:

[diana@localhost devstack]$ ssh [email protected] 
[email protected]'s password: 
$ curl http://169.254.169.254/openstack/latest/meta_data.json
{
  "random_seed": "ZOeXV4wSNym9/udq8l7wk2FAg1MZnX9Yafa43OsbsCdz6UyGY8pVtp0mjnco7ThzI7ACeZ1yD58bmL/KCKCI/GKrzxxHuRnGFQ268xdDTnZ5P/J/+oTOMPEXQQ0WPzy95YHLm7YPQuMIhM2kvKvwy5JyEUxcDzb/Az6+7/W7vLfUond2lVU6+RHKUJAm7PO674DGm2HgGONlqOruKjfIPlXeNH/p8iD6jnzkFGj2BysqHKMgdpsNoAlhi99wwUy7IsGie6YRMrYI0WT1Nf2xSSa4D0huRJbirCtXh3CmUXxBm2Qy8JqiGVlN38pmiw4pbnA3AW/i5/7T4nCwlpb8Fnc08JPUYdj4h4vS+8Bd4Fi26cW+fZPUcuPEhduWGyoyx1wFhLRY9vn5iNpw35nWb+MQShT/PUBmsudQrkiFOhirXJxbf02m2wiNxjcIdjVuegrz/Zjj7I/IhdMcqF+P8bcqHz2uvWVF91PYFsPUw0gXqykDpHsK7qBR3kCawvdnngDIOE44Pipw8L7qsIyvNBlQUllcxTS4v1UcssIRc8ivG2DpfSrVgEyZuU1KUOhyZzx/v0jiw0OEFxIGslW4vEM9Jk0kiRhEfoUNNPu/AYZaDLPOACZQSIE7juusuCsA0ihiiZKdrYUnx1CoyOnIan4kKxa41+I1Ugpbtu2Pt/w=", 
  "uuid": "d1542f40-9b9a-41a1-ac5d-3ec277f9d984", 
  "availability_zone": "nova", 
  "hostname": "instance-2.novalocal",
  "launch_index": 0,
  "project_id": "a379783483154bc3b93932b70f04b07f",
  "name": "instance-2"
}

11 Queries:

2016-03-17 15:25:48.001 DEBUG nova.metadata.wsgi.server [-] (19372) accepted ('10.0.0.2', 58041) from (pid=19372) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826
2016-03-17 15:25:48.015 DEBUG nova.utils [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] Reloading cached file /etc/nova/policy.json from (pid=19372) read_cached_file /opt/stack/nova/nova/utils.py:1525
2016-03-17 15:25:48.019 DEBUG nova.openstack.common.policy [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] Reloaded policy file: /etc/nova/policy.json from (pid=19372) _load_policy_file /opt/stack/nova/nova/openstack/common/policy.py:295
2016-03-17 15:25:48.020 DEBUG oslo_db.api [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] Loading backend 'sqlalchemy' from 'nova.db.sqlalchemy.api' from (pid=19372) _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:233
2016-03-17 15:25:48.035 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SHOW VARIABLES LIKE 'sql_mode'
2016-03-17 15:25:48.036 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.041 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT DATABASE()
2016-03-17 15:25:48.041 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.042 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'
2016-03-17 15:25:48.043 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.046 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1
2016-03-17 15:25:48.047 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.048 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1
2016-03-17 15:25:48.049 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.050 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin AS anon_1
2016-03-17 15:25:48.051 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.052 DEBUG oslo_db.sqlalchemy.engines [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION from (pid=19372) _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:256
2016-03-17 15:25:48.054 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.054 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.175 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.175 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.176 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] BEGIN (implicit)
2016-03-17 15:25:48.178 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT fixed_ips.created_at AS fixed_ips_created_at, fixed_ips.updated_at AS fixed_ips_updated_at, fixed_ips.deleted_at AS fixed_ips_deleted_at, fixed_ips.deleted AS fixed_ips_deleted, fixed_ips.id AS fixed_ips_id, fixed_ips.address AS fixed_ips_address, fixed_ips.network_id AS fixed_ips_network_id, fixed_ips.virtual_interface_id AS fixed_ips_virtual_interface_id, fixed_ips.instance_uuid AS fixed_ips_instance_uuid, fixed_ips.allocated AS fixed_ips_allocated, fixed_ips.leased AS fixed_ips_leased, fixed_ips.reserved AS fixed_ips_reserved, fixed_ips.host AS fixed_ips_host
FROM fixed_ips
WHERE fixed_ips.deleted = %s AND fixed_ips.address = %s
 LIMIT %s
2016-03-17 15:25:48.179 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, '10.0.0.2', 1)
2016-03-17 15:25:48.181 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] COMMIT
2016-03-17 15:25:48.205 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.205 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.215 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:25:48.216 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-17 15:25:48.233 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.234 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.236 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT instance_id_mappings.created_at AS instance_id_mappings_created_at, instance_id_mappings.updated_at AS instance_id_mappings_updated_at, instance_id_mappings.deleted_at AS instance_id_mappings_deleted_at, instance_id_mappings.deleted AS instance_id_mappings_deleted, instance_id_mappings.id AS instance_id_mappings_id, instance_id_mappings.uuid AS instance_id_mappings_uuid
FROM instance_id_mappings
WHERE instance_id_mappings.uuid = %s
 LIMIT %s
2016-03-17 15:25:48.236 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ('d1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1)
2016-03-17 15:25:48.242 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.242 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.245 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT s3_images.created_at AS s3_images_created_at, s3_images.updated_at AS s3_images_updated_at, s3_images.deleted_at AS s3_images_deleted_at, s3_images.deleted AS s3_images_deleted, s3_images.id AS s3_images_id, s3_images.uuid AS s3_images_uuid
FROM s3_images
WHERE s3_images.uuid = %s
 LIMIT %s
2016-03-17 15:25:48.245 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (u'7dd2249f-52f2-44c6-af02-196d683dd8a6', 1)
2016-03-17 15:25:48.248 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.248 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.249 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT s3_images.created_at AS s3_images_created_at, s3_images.updated_at AS s3_images_updated_at, s3_images.deleted_at AS s3_images_deleted_at, s3_images.deleted AS s3_images_deleted, s3_images.id AS s3_images_id, s3_images.uuid AS s3_images_uuid
FROM s3_images
WHERE s3_images.uuid = %s
 LIMIT %s
2016-03-17 15:25:48.251 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (u'dd252ba8-3242-4728-9c47-3429fb430bd1', 1)
2016-03-17 15:25:48.253 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.253 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.255 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT s3_images.created_at AS s3_images_created_at, s3_images.updated_at AS s3_images_updated_at, s3_images.deleted_at AS s3_images_deleted_at, s3_images.deleted AS s3_images_deleted, s3_images.id AS s3_images_id, s3_images.uuid AS s3_images_uuid
FROM s3_images
WHERE s3_images.uuid = %s
 LIMIT %s
2016-03-17 15:25:48.255 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (u'8f4bca54-3fd4-4994-9aea-9286cbc1ec61', 1)
2016-03-17 15:25:48.266 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.266 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.269 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT aggregates.created_at AS aggregates_created_at, aggregates.updated_at AS aggregates_updated_at, aggregates.deleted_at AS aggregates_deleted_at, aggregates.deleted AS aggregates_deleted, aggregates.id AS aggregates_id, aggregates.name AS aggregates_name, aggregate_hosts_1.created_at AS aggregate_hosts_1_created_at, aggregate_hosts_1.updated_at AS aggregate_hosts_1_updated_at, aggregate_hosts_1.deleted_at AS aggregate_hosts_1_deleted_at, aggregate_hosts_1.deleted AS aggregate_hosts_1_deleted, aggregate_hosts_1.id AS aggregate_hosts_1_id, aggregate_hosts_1.host AS aggregate_hosts_1_host, aggregate_hosts_1.aggregate_id AS aggregate_hosts_1_aggregate_id, aggregate_metadata_1.created_at AS aggregate_metadata_1_created_at, aggregate_metadata_1.updated_at AS aggregate_metadata_1_updated_at, aggregate_metadata_1.deleted_at AS aggregate_metadata_1_deleted_at, aggregate_metadata_1.deleted AS aggregate_metadata_1_deleted, aggregate_metadata_1.id AS aggregate_metadata_1_id, aggregate_metadata_1.`key` AS aggregate_metadata_1_key, aggregate_metadata_1.value AS aggregate_metadata_1_value, aggregate_metadata_1.aggregate_id AS aggregate_metadata_1_aggregate_id
FROM aggregates INNER JOIN aggregate_hosts ON aggregates.id = aggregate_hosts.aggregate_id AND aggregate_hosts.deleted = %s AND aggregates.deleted = %s INNER JOIN aggregate_metadata ON aggregates.id = aggregate_metadata.aggregate_id AND aggregate_metadata.deleted = %s AND aggregates.deleted = %s LEFT OUTER JOIN aggregate_hosts AS aggregate_hosts_1 ON aggregates.id = aggregate_hosts_1.aggregate_id AND aggregate_hosts_1.deleted = %s AND aggregates.deleted = %s LEFT OUTER JOIN aggregate_metadata AS aggregate_metadata_1 ON aggregates.id = aggregate_metadata_1.aggregate_id AND aggregate_metadata_1.deleted = %s AND aggregates.deleted = %s
WHERE aggregates.deleted = %s AND aggregate_hosts.host = %s AND aggregate_metadata.`key` = %s
2016-03-17 15:25:48.270 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 0, 0, 0, 0, 0, 0, 0, 0, u'localhost.localdomain', 'availability_zone')
2016-03-17 15:25:48.279 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.280 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.282 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT security_groups.created_at AS security_groups_created_at, security_groups.updated_at AS security_groups_updated_at, security_groups.deleted_at AS security_groups_deleted_at, security_groups.deleted AS security_groups_deleted, security_groups.id AS security_groups_id, security_groups.name AS security_groups_name, security_groups.description AS security_groups_description, security_groups.user_id AS security_groups_user_id, security_groups.project_id AS security_groups_project_id, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM security_groups INNER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_groups.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups.deleted = %s INNER JOIN instances ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups.id AND security_group_rules_1.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_group_rules_1.group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
WHERE security_groups.deleted = %s AND instances.uuid = %s
2016-03-17 15:25:48.283 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 0, 0, 0, 0, 0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-17 15:25:48.288 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.289 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.293 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info
FROM block_device_mapping
WHERE block_device_mapping.deleted = %s AND block_device_mapping.instance_uuid = %s
2016-03-17 15:25:48.293 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-17 15:25:48.297 DEBUG nova.objects.instance [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] Lazy-loading `metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-17 15:25:48.314 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.315 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.320 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:25:48.321 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0, 0)
2016-03-17 15:25:48.333 DEBUG nova.objects.instance [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] Lazy-loading `system_metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-17 15:25:48.350 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT 1
2016-03-17 15:25:48.350 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] ()
2016-03-17 15:25:48.357 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:25:48.358 INFO sqlalchemy.engine.base.Engine [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-17 15:25:48.375 INFO nova.api.ec2 [-] 0.364187s 10.0.0.2 GET /openstack/latest/meta_data.json None:None 200 [curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6] text/plain application/json
2016-03-17 15:25:48.376 INFO nova.metadata.wsgi.server [req-73543503-eb28-4ece-805c-40f4b3c6436f None None] 10.0.0.2 "GET /openstack/latest/meta_data.json HTTP/1.1" status: 200 len: 1031 time: 0.3652151

6 Queries:


2016-03-17 15:52:13.269 DEBUG nova.metadata.wsgi.server [-] (19372) accepted ('10.0.0.2', 58100) from (pid=19372) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826
2016-03-17 15:52:13.278 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.278 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.279 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] BEGIN (implicit)
2016-03-17 15:52:13.280 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT fixed_ips.created_at AS fixed_ips_created_at, fixed_ips.updated_at AS fixed_ips_updated_at, fixed_ips.deleted_at AS fixed_ips_deleted_at, fixed_ips.deleted AS fixed_ips_deleted, fixed_ips.id AS fixed_ips_id, fixed_ips.address AS fixed_ips_address, fixed_ips.network_id AS fixed_ips_network_id, fixed_ips.virtual_interface_id AS fixed_ips_virtual_interface_id, fixed_ips.instance_uuid AS fixed_ips_instance_uuid, fixed_ips.allocated AS fixed_ips_allocated, fixed_ips.leased AS fixed_ips_leased, fixed_ips.reserved AS fixed_ips_reserved, fixed_ips.host AS fixed_ips_host
FROM fixed_ips
WHERE fixed_ips.deleted = %s AND fixed_ips.address = %s
 LIMIT %s
2016-03-17 15:52:13.281 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, '10.0.0.2', 1)
2016-03-17 15:52:13.283 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] COMMIT
2016-03-17 15:52:13.298 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.299 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.305 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:52:13.305 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-17 15:52:13.332 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.332 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.335 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT security_groups.created_at AS security_groups_created_at, security_groups.updated_at AS security_groups_updated_at, security_groups.deleted_at AS security_groups_deleted_at, security_groups.deleted AS security_groups_deleted, security_groups.id AS security_groups_id, security_groups.name AS security_groups_name, security_groups.description AS security_groups_description, security_groups.user_id AS security_groups_user_id, security_groups.project_id AS security_groups_project_id, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM security_groups INNER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_groups.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups.deleted = %s INNER JOIN instances ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups.id AND security_group_rules_1.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_group_rules_1.group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
WHERE security_groups.deleted = %s AND instances.uuid = %s
2016-03-17 15:52:13.336 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, 0, 0, 0, 0, 0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-17 15:52:13.343 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.343 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.344 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info
FROM block_device_mapping
WHERE block_device_mapping.deleted = %s AND block_device_mapping.instance_uuid = %s
2016-03-17 15:52:13.345 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-17 15:52:13.348 DEBUG nova.objects.instance [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] Lazy-loading `metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-17 15:52:13.365 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.365 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.370 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:52:13.371 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0, 0)
2016-03-17 15:52:13.385 DEBUG nova.objects.instance [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] Lazy-loading `system_metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-17 15:52:13.400 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT 1
2016-03-17 15:52:13.401 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] ()
2016-03-17 15:52:13.406 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-17 15:52:13.406 INFO sqlalchemy.engine.base.Engine [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-17 15:52:13.426 INFO nova.api.ec2 [-] 0.150080s 10.0.0.2 GET /openstack/latest/meta_data.json None:None 200 [curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6] text/plain application/json
2016-03-17 15:52:13.428 INFO nova.metadata.wsgi.server [req-6c158571-a55a-4e59-8d78-47a1350d7830 None None] 10.0.0.2 "GET /openstack/latest/meta_data.json HTTP/1.1" status: 200 len: 1031 time: 0.1520700

How about the EC2 metadata endpoint in the BZ issue?

$ curl http://169.254.169.254/latest/meta-data/local-hostname
instance-2.novalocal
   
2016-03-18 13:53:54.489 DEBUG nova.metadata.wsgi.server [-] (19372) accepted ('10.0.0.2', 58104) from (pid=19372) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826
2016-03-18 13:53:54.505 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.505 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.506 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] BEGIN (implicit)
2016-03-18 13:53:54.507 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT fixed_ips.created_at AS fixed_ips_created_at, fixed_ips.updated_at AS fixed_ips_updated_at, fixed_ips.deleted_at AS fixed_ips_deleted_at, fixed_ips.deleted AS fixed_ips_deleted, fixed_ips.id AS fixed_ips_id, fixed_ips.address AS fixed_ips_address, fixed_ips.network_id AS fixed_ips_network_id, fixed_ips.virtual_interface_id AS fixed_ips_virtual_interface_id, fixed_ips.instance_uuid AS fixed_ips_instance_uuid, fixed_ips.allocated AS fixed_ips_allocated, fixed_ips.leased AS fixed_ips_leased, fixed_ips.reserved AS fixed_ips_reserved, fixed_ips.host AS fixed_ips_host
FROM fixed_ips
WHERE fixed_ips.deleted = %s AND fixed_ips.address = %s
 LIMIT %s
2016-03-18 13:53:54.507 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, '10.0.0.2', 1)
2016-03-18 13:53:54.509 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] COMMIT
2016-03-18 13:53:54.525 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.525 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.530 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-18 13:53:54.531 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-18 13:53:54.555 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.556 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.558 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT security_groups.created_at AS security_groups_created_at, security_groups.updated_at AS security_groups_updated_at, security_groups.deleted_at AS security_groups_deleted_at, security_groups.deleted AS security_groups_deleted, security_groups.id AS security_groups_id, security_groups.name AS security_groups_name, security_groups.description AS security_groups_description, security_groups.user_id AS security_groups_user_id, security_groups.project_id AS security_groups_project_id, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM security_groups INNER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_groups.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups.deleted = %s INNER JOIN instances ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups.id AND security_group_rules_1.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_group_rules_1.group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
WHERE security_groups.deleted = %s AND instances.uuid = %s
2016-03-18 13:53:54.558 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, 0, 0, 0, 0, 0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-18 13:53:54.563 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.563 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.564 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info
FROM block_device_mapping
WHERE block_device_mapping.deleted = %s AND block_device_mapping.instance_uuid = %s
2016-03-18 13:53:54.565 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984')
2016-03-18 13:53:54.567 DEBUG nova.objects.instance [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] Lazy-loading `metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-18 13:53:54.584 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.584 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.590 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-18 13:53:54.591 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0, 0)
2016-03-18 13:53:54.607 DEBUG nova.objects.instance [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] Lazy-loading `system_metadata' on Instance uuid d1542f40-9b9a-41a1-ac5d-3ec277f9d984 from (pid=19372) obj_load_attr /opt/stack/nova/nova/objects/instance.py:858
2016-03-18 13:53:54.625 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT 1
2016-03-18 13:53:54.625 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] ()
2016-03-18 13:53:54.630 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id
FROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned
FROM instances
WHERE instances.deleted = %s AND instances.uuid = %s
 LIMIT %s) AS anon_1 LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s
2016-03-18 13:53:54.633 INFO sqlalchemy.engine.base.Engine [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] (0, 'd1542f40-9b9a-41a1-ac5d-3ec277f9d984', 1, 0, 0, 0, 0)
2016-03-18 13:53:54.654 INFO nova.api.ec2 [-] 0.150574s 10.0.0.2 GET /latest/meta-data/local-hostname None:None 200 [curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6] text/plain text/plain
2016-03-18 13:53:54.654 INFO nova.metadata.wsgi.server [req-cb9151bd-d9e7-4cc4-a1bd-a25f175f8ee1 None None] 10.0.0.2 "GET /latest/meta-data/local-hostname HTTP/1.1" status: 200 len: 137 time: 0.1514549

ApacheBench (master, fedora):

$ sudo yum install httpd-tools

$ curl http://169.254.169.254/latest/meta-data/local-hostname
instance-abc.novalocal

$ ab -c 10 -n 1000 http://169.254.169.254/latest/meta-data/local-hostname
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.254.169.254 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        169.254.169.254
Server Port:            80

Document Path:          /latest/meta-data/local-hostname
Document Length:        22 bytes

Concurrency Level:      10
Time taken for tests:   11.167 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      158000 bytes
HTML transferred:       22000 bytes
Requests per second:    89.55 [#/sec] (mean)
Time per request:       111.674 [ms] (mean)
Time per request:       11.167 [ms] (mean, across all concurrent requests)
Transfer rate:          13.82 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   15  13.2     14      73
Processing:     2   94 708.9     24    7685
Waiting:        0   87 708.7     16    7672
Total:          3  109 710.4     44    7724

Percentage of the requests served within a certain time (ms)
  50%     44
  66%     53
  75%     59
  80%     62
  90%     77
  95%     94
  98%    113
  99%   2653
 100%   7724 (longest request)

ApacheBench (liberty, ubuntu):

$ sudo apt-get install apache2-utils

$ curl http://169.254.169.254/latest/meta-data/local-hostname
instance-foo.novalocal

$ ab -c 10 -n 1000 http://169.254.169.254/latest/meta-data/local-hostname
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.254.169.254 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        169.254.169.254
Server Port:            80

Document Path:          /latest/meta-data/local-hostname
Document Length:        22 bytes

Concurrency Level:      10
Time taken for tests:   8.502 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      158000 bytes
HTML transferred:       22000 bytes
Requests per second:    117.62 [#/sec] (mean)
Time per request:       85.017 [ms] (mean)
Time per request:       8.502 [ms] (mean, across all concurrent requests)
Transfer rate:          18.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    9  10.4      7     129
Processing:     4   74 481.2     20    5215
Waiting:        1   69 480.8     16    5212
Total:          6   83 487.5     30    5317

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     37
  75%     42
  80%     46
  90%     58
  95%     69
  98%     92
  99%   4394
 100%   5317 (longest request)

How do you change the metadata caching approach using devstack?
- By default, devstack appears to use this one (oslo_cache.dict): https://github.com/openstack/nova/blob/e276284791be5ee203a11283887d5cdfd2bd064e/nova/cache_utils.py#L100
- With memcached_servers set, you use this one (dogpile.cache.memcached): https://github.com/openstack/nova/blob/e276284791be5ee203a11283887d5cdfd2bd064e/nova/cache_utils.py#L89
- Pre-stack, you can set memcached_servers in local.conf:

[local](/dianaclarke/openstack-notes/wiki/localrc)
DEST=/opt/stack/
LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log
RABBIT_PASSWORD=123456

...

[post-config](/dianaclarke/openstack-notes/wiki/$NOVA_CONF)
[DEFAULT]
memcached_servers=127.0.0.1:11211

Post-stack, you can edit memcached_servers in /etc/nova/nova.conf (and restart n-api):

[DEFAULT]
memcached_servers=127.0.0.1:11211
flat_interface = ens3
flat_network_bridge = br100
vlan_interface = ens3
public_interface = br100

...

Hmmm... anything useful in here?
- http://www.cybera.ca/news-and-events/tech-radar/using-memcached-openstack-nova/
- https://bugs.launchpad.net/nova/+bug/1293794
ApacheBench liberty vs master with and without memcached:
- Master (Newton): http://paste.openstack.org/show/494825/
- Liberty: http://paste.openstack.org/show/494872/
- "Requests per second (mean)", "Time per request (mean)"
  - Master (+ memcached): 118.65 #/sec, 84.282 ms
  - Master (- memcached): 147.21 #/sec, 67.930 ms
  - Liberty (+ memcached): 147.27 #/sec, 67.904 ms
  - Liberty (- memcached): 117.62 #/sec, 85.017 ms
- Those numbers look suspiciously swapped. Repeat just in case.
Second pass (double check the above numbers):
- Master: http://paste.openstack.org/show/494881/
- Liberty: http://paste.openstack.org/show/494880/
- "Requests per second (mean)", "Time per request (mean)"
  - Master (+ memcached): 129.52 #/sec, 77.208 ms
  - Master (- memcached): 168.95 #/sec, 59.190 ms
  - Liberty (+ memcached): 153.29 #/sec, 65.236 ms
  - Liberty (- memcached): 185.31 #/sec, 53.965 ms
And how do those numbers compare to the bug report?
- Red Hat Bug #1244852 - metadata service is surprisingly slow with memcached
  - Version? (+ memcached): 41.99 #/sec, 238.137 ms
  - Version? (- memcached): 1636.23 #/sec, 6.112 ms