tripleo quickstart part 3 - dianaclarke/openstack-notes GitHub Wiki

[virthost] ifconfig
- http://paste.openstack.org/show/606167/
[host] virsh dumpxml (before <cpu mode='host-passthrough'/> change)
- http://paste.openstack.org/show/606168/
SSH keys

[diana@localhost ~]$ VIRTHOST=192.168.122.220
[diana@localhost ~]$ ssh-copy-id root@$VIRTHOST
[diana@localhost ~]$ ssh root@$VIRTHOST uname -a
Linux tripleo 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Grab fresh copy of quickstart.sh

[diana@localhost ~]$ mv quickstart.sh quickstart.sh.bak
[diana@localhost ~]$ wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
[diana@localhost ~]$     bash quickstart.sh --install-deps
Nothing to do.

Run the quickstart again

[diana@localhost ~]$ mv .quickstart .quickstart-bak
[diana@localhost ~]$ time bash quickstart.sh --tags all $VIRTHOST

Failed after an hour

TASK [undercloud-deploy : Install the undercloud] ******************************
task path: /home/diana/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/install-undercloud.yml:15
Tuesday 11 April 2017  10:45:49 -0400 (0:00:08.416)       0:22:01.327 ********* 
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/stack/undercloud-install.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/undercloud_install.log", "delta": "0:36:22.629249", "end": "2017-04-11 15:22:16.806623", "failed": true, "rc": 1, "start": "2017-04-11 14:45:54.177374", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}

PLAY RECAP *********************************************************************
192.168.122.220            : ok=124  changed=54   unreachable=0    failed=0   
localhost                  : ok=18   changed=8    unreachable=0    failed=0   
undercloud                 : ok=5    changed=4    unreachable=0    failed=1   

Tuesday 11 April 2017  11:22:17 -0400 (0:36:28.189)       0:58:29.516 *********

SSH to the undercloud and look at the logs

[diana@localhost ~]$ ssh -F $HOME/.quickstart/ssh.config.ansible undercloud
[stack@undercloud ~]$ ls
instackenv.json                overcloud-full.initrd   quickstart-hieradata-overrides.yaml  undercloud-install.sh
ironic-python-agent.initramfs  overcloud-full.qcow2    undercloud.conf                      undercloud-passwords.conf
ironic-python-agent.kernel     overcloud-full.vmlinuz  undercloud_install.log

[stack@undercloud ~]$ tail -n 100 undercloud_install.log
...
2017-04-11 15:22:16 | + rc=6
2017-04-11 15:22:16 | + set -e
2017-04-11 15:22:16 | + echo 'puppet apply exited with exit code 6'
2017-04-11 15:22:16 | puppet apply exited with exit code 6
2017-04-11 15:22:16 | + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-04-11 15:22:16 | + exit 6
2017-04-11 15:22:16 | [2017-04-11 15:22:16,642] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-04-11 15:22:16 | 
2017-04-11 15:22:16 | [2017-04-11 15:22:16,643] (os-refresh-config) [ERROR] Aborting...
2017-04-11 15:22:16 | Traceback (most recent call last):
2017-04-11 15:22:16 |   File "<string>", line 1, in <module>
2017-04-11 15:22:16 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1214, in install
2017-04-11 15:22:16 |     _run_orc(instack_env)
2017-04-11 15:22:16 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1003, in _run_orc
2017-04-11 15:22:16 |     _run_live_command(args, instack_env, 'os-refresh-config')
2017-04-11 15:22:16 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 519, in _run_live_command
2017-04-11 15:22:16 |     raise RuntimeError('%s failed. See log for details.' % name)
2017-04-11 15:22:16 | RuntimeError: os-refresh-config failed. See log for details.
2017-04-11 15:22:16 | Command 'instack-install-undercloud' returned non-zero exit status 1

What other errors are there?

[stack@undercloud ~]$ grep -i error undercloud_install.log
2017-04-11 14:50:47 | 2017-04-11 14:50:47 - Error: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-11 14:50:47 | 2017-04-11 14:50:47 - Error: /Stage[main]/Tripleo::Profile::Base::Ui/Package[openstack-tripleo-ui]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Error: Cannot create /var/www/openstack-tripleo-ui/dist; parent directory /var/www/openstack-tripleo-ui does not exist
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Error: /Stage[main]/Tripleo::Ui/Apache::Vhost[tripleo-ui]/File[/var/www/openstack-tripleo-ui/dist]/ensure: change from absent to directory failed: Cannot create /var/www/openstack-tripleo-ui/dist; parent directory /var/www/openstack-tripleo-ui does not exist
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage  db_sync returned 1 instead of one of [0]
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: keystone-manage  db_sync returned 1 instead of one of [0]
2017-04-11 15:07:08 | 2017-04-11 15:07:08 - Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/services: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3898690>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:09:59 | 2017-04-11 15:09:59 - Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/roles: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/roles (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x37e5e10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 24, for a total of 170 seconds)
2017-04-11 15:12:57 | 2017-04-11 15:12:57 - Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/endpoints: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/endpoints (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3993e10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     packet.check_error()
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     raise errorclass(errno, errval)
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'192.168.24.1' (using password: YES)")
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2017-04-11 15:17:34 | 2017-04-11 15:17:34 - Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/projects: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/projects (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x328da10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 24, for a total of 170 seconds)
2017-04-11 15:20:47 | 2017-04-11 15:20:47 - Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/domains: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x413a610>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'192.168.24.1' (using password: YES)")
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
2017-04-11 15:22:16 | [2017-04-11 15:22:16,642] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-04-11 15:22:16 | [2017-04-11 15:22:16,643] (os-refresh-config) [ERROR] Aborting...
2017-04-11 15:22:16 |     raise RuntimeError('%s failed. See log for details.' % name)
2017-04-11 15:22:16 | RuntimeError: os-refresh-config failed. See log for details.

With a bit more context

[stack@undercloud ~]$ grep -i error -C 2 undercloud_install.log
2017-04-11 14:50:43 | 2017-04-11 14:50:43 - Notice: /Stage[main]/Aodh::Auth/Aodh_config[service_credentials/user_domain_id]/ensure: created
2017-04-11 14:50:43 | 2017-04-11 14:50:43 - Notice: /Stage[main]/Aodh::Auth/Aodh_config[service_credentials/password]/ensure: created
2017-04-11 14:50:47 | 2017-04-11 14:50:47 - Error: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-11 14:50:47 | 2017-04-11 14:50:47 -   openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch: [Errno 256] No more mirrors to try.
2017-04-11 14:50:47 | 2017-04-11 14:50:47 - Error: /Stage[main]/Tripleo::Profile::Base::Ui/Package[openstack-tripleo-ui]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-11 14:50:47 | 2017-04-11 14:50:47 -   openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch: [Errno 256] No more mirrors to try.
2017-04-11 14:50:47 | 2017-04-11 14:50:47 - Notice: /Stage[main]/Glance::Backend::Swift/Glance_api_config[glance_store/swift_store_config_file]/ensure: created
--
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[service_credentials/region_name]/ensure: created
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer/Ceilometer_config[database/event_time_to_live]/ensure: created
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[service_credentials/password]/ensure: created
2017-04-11 14:50:50 | 2017-04-11 14:50:50 - Notice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/http_timeout]/ensure: created
--
2017-04-11 15:02:44 | 2017-04-11 15:02:44 - Notice: /Stage[main]/Keystone/Oslo::Messaging::Rabbit[keystone_config]/Keystone_config[oslo_messaging_rabbit/rabbit_password]/ensure: created
2017-04-11 15:02:44 | 2017-04-11 15:02:44 - Notice: /Stage[main]/Keystone/Oslo::Middleware[keystone_config]/Keystone_config[oslo_middleware/enable_proxy_headers_parsing]/ensure: created
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Error: Cannot create /var/www/openstack-tripleo-ui/dist; parent directory /var/www/openstack-tripleo-ui does not exist
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Error: /Stage[main]/Tripleo::Ui/Apache::Vhost[tripleo-ui]/File[/var/www/openstack-tripleo-ui/dist]/ensure: change from absent to directory failed: Cannot create /var/www/openstack-tripleo-ui/dist; parent directory /var/www/openstack-tripleo-ui does not exist
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Notice: /Stage[main]/Keystone/Keystone_config[DEFAULT/public_port]/ensure: created
2017-04-11 15:02:45 | 2017-04-11 15:02:45 - Notice: /Stage[main]/Keystone/Oslo::Messaging::Rabbit[keystone_config]/Keystone_config[oslo_messaging_rabbit/rabbit_host]/ensure: created
--
2017-04-11 15:02:57 | 2017-04-11 15:02:57 - Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:02:57 | 2017-04-11 15:02:57 - Warning: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Skipping because of failed dependencies
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage  db_sync returned 1 instead of one of [0]
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: keystone-manage  db_sync returned 1 instead of one of [0]
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:04:09 | 2017-04-11 15:04:09 - Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Skipping because of failed dependencies
--
2017-04-11 15:04:12 | 2017-04-11 15:04:12 - Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:04:12 | 2017-04-11 15:04:12 - Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Skipping because of failed dependencies
2017-04-11 15:07:08 | 2017-04-11 15:07:08 - Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/services: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3898690>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:07:08 | 2017-04-11 15:07:08 - Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:07:08 | 2017-04-11 15:07:08 - Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:07:08 | 2017-04-11 15:07:08 - Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]: Skipping because of failed dependencies
2017-04-11 15:09:59 | 2017-04-11 15:09:59 - Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/roles: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/roles (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x37e5e10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 24, for a total of 170 seconds)
2017-04-11 15:09:59 | 2017-04-11 15:09:59 - Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:09:59 | 2017-04-11 15:09:59 - Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:10:01 | 2017-04-11 15:10:01 - Notice: /Stage[main]/Main/Keystone_service[nova::computev3]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:10:01 | 2017-04-11 15:10:01 - Warning: /Stage[main]/Main/Keystone_service[nova::computev3]: Skipping because of failed dependencies
2017-04-11 15:12:57 | 2017-04-11 15:12:57 - Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/endpoints: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/endpoints (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3993e10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:12:57 | 2017-04-11 15:12:57 - Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_endpoint[regionOne/aodh::alarming]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:12:57 | 2017-04-11 15:12:57 - Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_endpoint[regionOne/aodh::alarming]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     auth_packet = self._read_packet()
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     packet.check_error()
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     err.raise_mysql_exception(self._data)
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     raise errorclass(errno, errval)
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'192.168.24.1' (using password: YES)")
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::end]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::end]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:14:40 | 2017-04-11 15:14:40 - Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Skipping because of failed dependencies
2017-04-11 15:17:34 | 2017-04-11 15:17:34 - Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/projects: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/projects (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x328da10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 24, for a total of 170 seconds)
2017-04-11 15:17:34 | 2017-04-11 15:17:34 - Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:17:34 | 2017-04-11 15:17:34 - Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:17:53 | 2017-04-11 15:17:53 - Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:17:53 | 2017-04-11 15:17:53 - Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Skipping because of failed dependencies
2017-04-11 15:20:47 | 2017-04-11 15:20:47 - Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/domains: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x413a610>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 25, for a total of 170 seconds)
2017-04-11 15:20:47 | 2017-04-11 15:20:47 - Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:20:47 | 2017-04-11 15:20:47 - Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:20:50 | 2017-04-11 15:20:50 - Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Dependency Package[openstack-tripleo-ui] has failures: true
2017-04-11 15:20:50 | 2017-04-11 15:20:50 - Warning: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Skipping because of failed dependencies
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'192.168.24.1' (using password: YES)")
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency File[/var/www/openstack-tripleo-ui/dist] has failures: true
2017-04-11 15:22:01 | 2017-04-11 15:22:01 - Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency Package[openstack-tripleo-ui] has failures: true
--
2017-04-11 15:22:16 | + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-04-11 15:22:16 | + exit 6
2017-04-11 15:22:16 | [2017-04-11 15:22:16,642] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-04-11 15:22:16 | 
2017-04-11 15:22:16 | [2017-04-11 15:22:16,643] (os-refresh-config) [ERROR] Aborting...
2017-04-11 15:22:16 | Traceback (most recent call last):
2017-04-11 15:22:16 |   File "<string>", line 1, in <module>
--
2017-04-11 15:22:16 |     _run_live_command(args, instack_env, 'os-refresh-config')
2017-04-11 15:22:16 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 519, in _run_live_command
2017-04-11 15:22:16 |     raise RuntimeError('%s failed. See log for details.' % name)
2017-04-11 15:22:16 | RuntimeError: os-refresh-config failed. See log for details.
2017-04-11 15:22:16 | Command 'instack-install-undercloud' returned non-zero exit status 1

What's in the undercloud config?

[stack@undercloud ~]$ cat undercloud.conf | grep '=' | grep -v '#'
network_cidr = 192.168.24.0/24
local_ip = 192.168.24.1/24
network_gateway = 192.168.24.1
undercloud_public_vip = 192.168.24.2
undercloud_admin_vip = 192.168.24.3
generate_service_certificate = True
masquerade_network = 192.168.24.0/24
dhcp_start = 192.168.24.5
dhcp_end = 192.168.24.30
hieradata_override = quickstart-hieradata-overrides.yaml
inspection_iprange = 192.168.24.100,192.168.24.120

I see weshay talking about similar errors here:
- http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2016-11-18.log.html
These errors look the same:
- https://www.redhat.com/archives/rdo-list/2016-June/msg00229.html
Just the openstack-tripleo-ui error, wrapped

Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1:
Error downloading packages:
openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch:
[Errno 256] No more mirrors to try.

Manually install this one, maybe it will used this failed undercloud when I re-run it.
- https://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch.rpm

[stack@undercloud ~]$ sudo yum -d 0 -e 0 -y install openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch.rpm

Failed again

TASK [undercloud-deploy : Install the undercloud] ******************************
task path: /home/diana/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/install-undercloud.yml:15
Wednesday 12 April 2017  11:30:22 -0400 (0:00:08.683)       0:17:43.707 ******* 
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/stack/undercloud-install.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/undercloud_install.log", "delta": "0:36:13.469993", "end": "2017-04-12 16:06:40.850613", "failed": true, "rc": 1, "start": "2017-04-12 15:30:27.380620", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}

PLAY RECAP *********************************************************************
192.168.122.220            : ok=121  changed=48   unreachable=0    failed=0   
localhost                  : ok=19   changed=8    unreachable=0    failed=0   
undercloud                 : ok=5    changed=4    unreachable=0    failed=1

Hmmm... same error. So it's not using my failed undercloud from the last failed pass.

017-04-12 15:35:34 | 2017-04-12 15:35:34 - Error: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-12 15:35:34 | 2017-04-12 15:35:34 -   openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch: [Errno 256] No more mirrors to try.
2017-04-12 15:35:34 | 2017-04-12 15:35:34 - Error: /Stage[main]/Tripleo::Profile::Base::Ui/Package[openstack-tripleo-ui]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-tripleo-ui' returned 1: Error downloading packages:
2017-04-12 15:35:34 | 2017-04-12 15:35:34 -   openstack-tripleo-ui-1.1.0-0.20170201184830.36b0b58.el7.centos.noarch: [Errno 256] No more mirrors to try.

Maybe I can just skip this openstack-tripleo-ui stuff, comment it out somewhere.

diana@localhost .quickstart]$ grep -r 'tripleo-ui' *
config/release/tripleo-ci/master.yml:      - openstack-tripleo-ui
config/release/master-tripleo-ci.yml:      - openstack-tripleo-ui
tripleo-quickstart/ci-scripts/full-deploy.sh:    export ZUUL_CHANGES=openstack/tripleo-ui:master:refs/changes/25/422025/2
tripleo-quickstart/config/release/tripleo-ci/master.yml:      - openstack-tripleo-ui
tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/inject_repos.yml:    line: "includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,openstack-tripleo-docs,puppet-*"
usr/local/share/ansible/roles/validate-undercloud/templates/undercloud-sanity-check.sh.j2:    if [ -f "/etc/httpd/conf.d/25-tripleo-ui.conf" ]; then
usr/local/share/ansible/roles/repo-setup/README.md:              - openstack-tripleo-ui
usr/local/share/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/inject_repos.yml:    line: "includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,openstack-tripleo-docs,puppet-*"

See how people respond to this bug (similar root cause): https://bugzilla.redhat.com/show_bug.cgi?id=1441750
Explore here & read some of these docs, see if they help... https://trunk.rdoproject.org/
What mirrors is it currently pointing? (vs /rdo-trunk-newton-tested/)

[stack@undercloud ~]$ grep -r baseurl /etc/yum.repos.d/ | grep delorean
/etc/yum.repos.d/delorean.repo:baseurl=https://trunk.rdoproject.org/centos7-newton/94/1e/941ec3f4d1860cbc43c007ebe1f8e9bf9d5c519a_4297a92f
/etc/yum.repos.d/delorean-deps.repo:baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-newton/
/etc/yum.repos.d/delorean-deps.repo:baseurl=http://cbs.centos.org/repos/cloud7-openstack-common-pending/$basearch/os/
/etc/yum.repos.d/delorean-deps.repo:baseurl=http://mirror.centos.org/centos/7/virt/$basearch/kvm-common/

What was wrong with this one? How does it decide?
- https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-newton/openstack-tripleo-ui-1.1.0-1.el7.noarch.rpm
It's looking here:
- https://trunk.rdoproject.org/centos7-newton/94/1e/941ec3f4d1860cbc43c007ebe1f8e9bf9d5c519a_4297a92f/
What determines that baseurl? Why is neutron in the name? Is that fishy? Seems fishy...

[stack@undercloud ~]$ cat /etc/yum.repos.d/delorean.repo
[delorean]
name=delorean-openstack-neutron-941ec3f4d1860cbc43c007ebe1f8e9bf9d5c519a
baseurl=https://trunk.rdoproject.org/centos7-newton/94/1e/941ec3f4d1860cbc43c007ebe1f8e9bf9d5c519a_4297a92f
enabled=1
gpgcheck=0

I dunno... try grabbing a fresh quickstart. Maybe someone else fixed it.

[diana@localhost ~]$ rm -rf .quickstart
[diana@localhost ~]$ mv quickstart.sh quickstart.sh.bak2
[diana@localhost ~]$ wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
[diana@localhost ~]$ time bash quickstart.sh --tags all $VIRTHOST

Nope, failed again (same error, same delorean.repo contents)
Review these patches, see if someone is already on this:
- https://review.openstack.org/#/q/project:openstack/tripleo-quickstart
Answer from IRC:

<diana_clarke> Is anyone else seeing openstack-tripleo-ui dependency errors with the tripleo quickstart? http://paste.openstack.org/show/606483/
* fragatina (~fragatina@2001:420:283:1004:59cb:3dd9:d744:a5ef) has joined #tripleo
* lblanchard has quit (Quit: Leaving.)
* trown|brb is now known as trown
<weshay> diana_clarke, don't see it here http://trunk.rdoproject.org/centos7-newton/94/1e/941ec3f4d1860cbc43c007ebe1f8e9bf9d5c519a_4297a92f/
<trown> weshay: diana_clarke that is the CDN issue
<trown> we need https://review.openstack.org/#/c/456225/
<trown> diana_clarke: if you pull in that change ^, and then run quickstart with --no-clone, then you should get the much slower but actually up to date image location
<diana_clarke> trown: Thanks! I should have asked sooner. I keep running into this, but I couldn't find similar errors in logstash, google, etc, so I figured it was just me doing something wrong.
<trown> diana_clarke: ya CI has never pulled from the CDN location, so it doesnt show up in CI
<trown> diana_clarke: sorry you were blocked on this... it is a pretty frustrating issue
* cylopez has quit (Quit: Leaving.)
<diana_clarke> trown: I learned things trying to figure it out at least ;) I'll give your suggestion a shot now. Have a great weekend!

Give it a shot (cdn.diff is from that review mentioned above):

[diana@localhost .quickstart]$ git apply cdn.diff
[diana@localhost .quickstart]$ cd
[diana@localhost ~]$ time bash quickstart.sh --tags all $VIRTHOST

Shit! I forgot --no-clone

[diana@localhost ~]$ time bash quickstart.sh --no-clone --tags all $VIRTHOST
Installing OpenStack newton on host 192.168.122.220
Using directory /home/diana/.quickstart for a local working directory
+ export ANSIBLE_CONFIG=/home/diana/ansible.cfg
+ ANSIBLE_CONFIG=/home/diana/ansible.cfg
+ export ANSIBLE_INVENTORY=/home/diana/.quickstart/hosts
+ ANSIBLE_INVENTORY=/home/diana/.quickstart/hosts
+ export ARA_DATABASE=sqlite:////home/diana/.quickstart/ara.sqlite
+ ARA_DATABASE=sqlite:////home/diana/.quickstart/ara.sqlite
+ source /home/diana/ansible_ssh_env.sh
quickstart.sh: line 454: /home/diana/ansible_ssh_env.sh: No such file or directory

real	0m0.073s
user	0m0.012s
sys	0m0.006s

Hmmm... is that broken too? Or am I just doing something wrong?
- http://codesearch.openstack.org/?q=--no-clone&i=nope&files=&repos=
Is this the culprit?
- https://github.com/openstack/tripleo-quickstart/blob/41183e5edb792183b17149808090234e85b56f60/quickstart.sh#L454
- https://github.com/openstack/tripleo-quickstart/blob/41183e5edb792183b17149808090234e85b56f60/quickstart.sh#L347
- Try setting OOOQ_DIR=$OPT_WORKDIR/tripleo-quickstart in the $OPT_NO_CLONE case too.
- Didn't finish this thought, circle back to this.
Workaround: quickly ssh to the undercloud and install the missing tripleo-ui dependency before the quickstart tries to and fails.
- Example: https://gist.github.com/lvlds/bc5e2254712028b6666d72a3c2bc9775
- Thanks Luis Valdés for the workaround!
Oh, there's a --gerrit <change-id> option. Try it with I698f0d161a3cd8fb26cd7eb4d9f43ccc15c8b9a1 from https://review.openstack.org/#/c/456225/

[diana@localhost ~]$ rm -rf .quickstart
[diana@localhost ~]$ wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
[diana@localhost ~]$ time bash quickstart.sh --gerrit I698f0d161a3cd8fb26cd7eb4d9f43ccc15c8b9a1 --tags all $VIRTHOST

So... even if that fixes it, I'm still unclear why switching from buildlogs.centos.org/centos/7 to https://trunk.rdoproject.org/centos7 fixes this since my original error was with https://trunk.rdoproject.org/centos7. /me shrugs
- Ah, I see. The buildlogs delorean.repo just points back to trunk.rdoproject.org/centos7
- https://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-ocata-tested/delorean.repo
Man... now what.

TASK [setup/overcloud : Define overcloud vms] **********************************
task path: /home/diana/.quickstart/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:84
Wednesday 19 April 2017  13:15:35 -0400 (0:00:05.799)       0:05:04.269 ******* 
failed: [192.168.122.151] (item={u'flavor': u'control', u'virtualbmc_port': 6230, u'name': u'control_0'}) => {"failed": true, "item": {"flavor": "control", "name": "control_0", "virtualbmc_port": 6230}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}
failed: [192.168.122.151] (item={u'flavor': u'compute', u'virtualbmc_port': 6231, u'name': u'compute_0'}) => {"failed": true, "item": {"flavor": "compute", "name": "compute_0", "virtualbmc_port": 6231}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}

PLAY RECAP *********************************************************************
192.168.122.151            : ok=88   changed=32   unreachable=0    failed=1   
localhost                  : ok=12   changed=5    unreachable=0    failed=0

Looks like I forgot to shut down the virthost so that the domain xml is updated with host-passthough
Finally!

PLAY RECAP *********************************************************************
192.168.122.151            : ok=128  changed=52   unreachable=0    failed=0   
localhost                  : ok=22   changed=7    unreachable=0    failed=0   
undercloud                 : ok=60   changed=40   unreachable=0    failed=0   

...

##################################
Virtual Environment Setup Complete
##################################

Access the undercloud by:

    ssh -F /home/diana/.quickstart/ssh.config.ansible undercloud

Follow the documentation in the link below to complete your deployment.

    http://ow.ly/c44w304begR

##################################
Virtual Environment Setup Complete
##################################

real	161m29.908s
user	8m32.037s
sys	1m55.951s

tripleo quickstart part 3 - dianaclarke/openstack-notes GitHub Wiki

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️