tripleo quickstart part 2 - dianaclarke/openstack-notes GitHub Wiki

Old notes: https://github.com/dianaclarke/openstack-notes/wiki/tripleo-quick-start
This time, use centos (not fedora): CentOS-7-x86_64-Everything-1611.iso
[host] Domain XML:

$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 18    tripleo                        running

$ sudo virsh dumpxml tripleo
<domain type='kvm' id='18'>
  <name>tripleo</name>
  <uuid>51f9542d-b6d0-4a0e-99a4-9beec4569054</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell-noTSX</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/diana/images/centos7.0.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/diana/Desktop/iso/CentOS-7-x86_64-Everything-1611.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:0a:39:17'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-18-tripleo/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='connected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c397,c509</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c397,c509</imagelabel>
  </seclabel>
</domain>

[host] Change the cpu mode in the domain XML:

$ sudo virsh edit tripleo
Domain tripleo XML configuration edited.

<cpu mode='host-passthrough'>
</cpu>

[host] Restart tripleo VM, virtual machine manager
[guest] Note the ip address: 192.168.122.29
[host] Set up ssh keys for $VIRTHOST etc

$ VIRTHOST=192.168.122.29
$ ssh-copy-id root@$VIRTHOST
$ ssh root@$VIRTHOST uname -a
Linux tripleo 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[host] Run the quickstart (with -tags all so that you get undercloud + overcloud)

$ time bash quickstart.sh --tags all $VIRTHOST

Failed after 143m13.532s minutes

Full logs: https://gist.github.com/dianaclarke/54ba9d943d8847c41ebdbd0e428ad745
Play recap

TASK [did the deployment pass or fail?] ****************************************
task path: /home/diana/.quickstart/playbooks/quickstart-extras.yml:80
Monday 27 February 2017  17:09:55 -0500 (0:00:00.645)       2:23:08.451 ******* 
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "failed_when_result": true, 
    "overcloud_deploy_result": "failed"    <---------------- !!!
}

PLAY RECAP *********************************************************************
192.168.122.29             : ok=119  changed=49   unreachable=0    failed=0   
localhost                  : ok=21   changed=8    unreachable=0    failed=1   
undercloud                 : ok=53   changed=33   unreachable=0    failed=0

The did the deployment pass or fail? task is here:
- https://github.com/openstack/tripleo-quickstart/blob/d65aff5e6ba1bd15b4d21965ce6be9b0518c9f20/playbooks/quickstart-extras.yml#L82
Or maybe here, not sure which yet.
- https://github.com/openstack/tripleo-quickstart/blob/a4afe28c0577e5f3af1dcfa0beccb68a2b94eadb/playbooks/tripleo-roles.yml#L89
What writes to overcloud_deployment_result.json?
- check the overcloud_deployment_result variable to determine playbook exit - https://github.com/openstack/tripleo-quickstart/commit/c73de324a4e424ca82fbc963953918efe7210d42
- Maybe this? https://review.gerrithub.io/#/c/281750/10/tasks/overcloud-status.yml
- Hmmm... this repo (redhat-openstack/ansible-role-tripleo-overcloud) appears to be dead
- Looks like it became: tripleo-quickstart-extras - https://github.com/openstack/tripleo-quickstart-extras
- Here we go (I think): https://github.com/openstack/tripleo-quickstart-extras/blob/97dcd6f0ee87774d8f267df4769df454d49761c3/roles/overcloud-deploy/tasks/overcloud-status.yml
- Hmmm... I wonder why that repo isn't being indexed by http://codesearch.openstack.org/
Ah, perhaps this explains why I have no logs on overcloud deploy failure: "At the moment the playbook will exit if the deployment fails, which leaves us with out any logs from the overcloud." -- https://review.gerrithub.io/#/c/281750/
But then again, maybe that's old. Try to find this {{ deploy_log }} I'm seeing in overcloud-status.yml
- defined here? https://github.com/openstack/tripleo-quickstart-extras/blob/736f92ef980b71162d14ed58057b62c75cacdb14/roles/overcloud-deploy/defaults/main.yml#L5
[host] What's in .quickstart again?

$ ls -1 .quickstart/
ara.sqlite
bin
config
hosts
id_rsa_overcloud
id_rsa_undercloud
id_rsa_undercloud.pub
id_rsa_virt_host
id_rsa_virt_host.pub
id_rsa_virt_power
id_rsa_virt_power.pub
include
lib
lib64
overcloud_deployment_result.json
pip-selfcheck.json
playbooks
ssh.config.ansible
ssh.config.local.ansible
stackrc
tripleo-quickstart
tripleo_quickstart.egg-info
usr

What about on the undercloud?

[diana@localhost ~]$ ssh -F $HOME/.quickstart/ssh.config.ansible undercloud
[stack@undercloud ~]$ . stackrc

[stack@undercloud ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+--------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| 567cf6e8-fc34-4fba-ad90-17cadc482ac0 | overcloud  | CREATE_FAILED | 2017-02-27T21:53:30Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+

[stack@undercloud ~]$ openstack stack list
+--------------------------------------+------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| 567cf6e8-fc34-4fba-ad90-17cadc482ac0 | overcloud  | CREATE_FAILED | 2017-02-27T21:53:30Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+

[stack@undercloud ~]$ ls
cloud-names.yaml                    overcloud_custom_tht_script.log  overcloud-prep-images.sh
containers-default-parameters.yaml  overcloud-custom-tht-script.sh   overcloud_prep_network.log
enable-tls.yaml                     overcloud_deploy.log             overcloud-prep-network.sh
get-overcloud-nodes.py              overcloud-deploy-post.sh         quickstart-hieradata-overrides.yaml
inject-trust-anchor.yaml            overcloud-deploy.sh              server-cert.pem
instackenv.json                     overcloud-env.json               server-key.pem
ironic-python-agent.initramfs       overcloud-full.initrd            server-req.pem
ironic-python-agent.kernel          overcloud-full.qcow2             stackrc
network-environment.yaml            overcloud-full.vmlinuz           undercloud.conf
overcloud-cacert.pem                overcloud-prep-containers.sh     undercloud_install.log
overcloud-ca-privkey.pem            overcloud_prep_flavors.log       undercloud-install.sh
overcloud_create_ssl_cert.log       overcloud-prep-flavors.sh        undercloud-passwords.conf
overcloud-create-ssl-cert.sh        overcloud_prep_images.log        upload_images_to_local_registry.py

Here's the overcloud_deploy.log
- https://gist.github.com/dianaclarke/722dcb0367765237e89b56dd30d9cafb
Hmm.... I don't see any of those three messages in the log. Wrong log? Wrong line of thinking?

[stack@undercloud ~]$ grep 'Stack overcloud CREATE_FAILED' overcloud_deploy.log 
[stack@undercloud ~]$ grep 'Stack overcloud CREATE_COMPLETE' overcloud_deploy.log 
[stack@undercloud ~]$ grep 'Stack overcloud UPDATE_COMPLETE' overcloud_deploy.log

And nova list?

[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 8b5115bd-c8dd-4e42-a657-d8a9d1626ea0 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.12 |
| 43c8c8af-e492-4c87-a6d7-b8f4cc94c8eb | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+

Monkey balls! I got further with this last time.
Last time:

[stack@undercloud ~]$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+-----------------+----------------------+--------------+
| 0c4f7222-97f1-49a7-b818-69dd8c76a571 | overcloud  | CREATE_COMPLETE | 2017-02-16T21:33:50Z | None         |
+--------------------------------------+------------+-----------------+----------------------+--------------+

This time:

[stack@undercloud ~]$ openstack stack list
+--------------------------------------+------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| 567cf6e8-fc34-4fba-ad90-17cadc482ac0 | overcloud  | CREATE_FAILED | 2017-02-27T21:53:30Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+

Try again (grab a fresh copy of quickstart.sh in case something was fixed, and yum update the host, etc)

[diana@localhost ~]$ rm quickstart.sh 
[diana@localhost ~]$ wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
[diana@localhost ~]$ bash quickstart.sh --install-deps

Do the virsh edit, ssh keys, etc dance again. Then try again, this time with --release master

[diana@localhost ~]$ time bash quickstart.sh --tags all --release master $VIRTHOST

Failed after 207m23.840s minutes
- https://gist.github.com/dianaclarke/973476af4c145aa2d63b0388a78160a1
Play recap

TASK [tripleo-inventory : fetch overcloud node names and IPs] ******************
task path: /home/diana/.quickstart/tripleo-quickstart/roles/tripleo-inventory/tasks/main.yml:19
Wednesday 01 March 2017  16:39:30 -0500 (0:00:23.704)       3:27:00.998 ******* 
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "source /home/stack/stackrc; python /home/stack/get-overcloud-nodes.py", "delta": "0:00:08.045122", "end": "2017-03-01 21:39:48.366255", "failed": true, "rc": 1, "start": "2017-03-01 21:39:40.321133", "stderr": "Traceback (most recent call last):\n  File \"/home/stack/get-overcloud-nodes.py\", line 14, in <module>\n    for server in nova.servers.list()\n  File \"/usr/lib/python2.7/site-packages/novaclient/v2/servers.py\", line 861, in list\n    \"servers\")\n  File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 254, in _list\n    resp, body = self.api.client.get(url)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 217, in get\n    return self.request(url, 'GET', **kwargs)\n  File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 74, in request\n    **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 374, in request\n    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 142, in request\n    return self.session.request(url, method, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/positional/__init__.py\", line 101, in inner\n    return wrapped(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 491, in request\n    auth_headers = self.get_auth_headers(auth)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 818, in get_auth_headers\n    return auth.get_headers(self, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 90, in get_headers\n    token = self.get_token(session)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 90, in get_token\n    return self.get_access(session).auth_token\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 136, in get_access\n    self.auth_ref = self.get_auth_ref(session)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 198, in get_auth_ref\n    return self._plugin.get_auth_ref(session, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py\", line 65, in get_auth_ref\n    authenticated=False, log=False)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 766, in post\n    return self.request(url, 'POST', **kwargs)\n  File \"/usr/lib/python2.7/site-packages/positional/__init__.py\", line 101, in inner\n    return wrapped(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 655, in request\n    raise exceptions.from_response(resp, method, url)\nkeystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-3ed5d9d0-e3cd-4f35-909f-1c935981201a)", "stdout": "", "stdout_lines": [], "warnings": []}

PLAY RECAP *********************************************************************
192.168.122.45             : ok=122  changed=51   unreachable=0    failed=0   
localhost                  : ok=19   changed=8    unreachable=0    failed=0   
undercloud                 : ok=48   changed=30   unreachable=0    failed=1

Okay, try one last time... (go back to newton)

$ time bash quickstart.sh --tags all $VIRTHOST

OMG! It worked! (I think)
- https://gist.github.com/dianaclarke/a2916a284003ad55ae6c6a8e7be64859
- What did I do differently this time? Nothing...

PLAY RECAP *********************************************************************
192.168.122.45             : ok=119  changed=49   unreachable=0    failed=0   
localhost                  : ok=22   changed=8    unreachable=0    failed=0   
undercloud                 : ok=58   changed=39   unreachable=0    failed=0

Disk space

[diana@tripleo ~]$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/cl_tripleo-root   50G  7.7G   43G  16% /
devtmpfs                      16G     0   16G   0% /dev
tmpfs                         16G   92K   16G   1% /dev/shm
tmpfs                         16G   25M   16G   1% /run
tmpfs                         16G     0   16G   0% /sys/fs/cgroup
/dev/vda1                   1014M  174M  841M  18% /boot
/dev/mapper/cl_tripleo-home  334G   33G  301G  10% /home
tmpfs                        3.2G  4.0K  3.2G   1% /run/user/42
tmpfs                        3.2G  8.0K  3.2G   1% /run/user/1000
tmpfs                        3.2G   52K  3.2G   1% /run/user/1001

[diana@tripleo ~]$ sudo ls -lah /home/stack/pool
total 29G
drwxr-xr-x.  2 stack stack   76 Mar  1 21:54 .
drwx------. 10 stack stack  280 Mar  1 21:54 ..
-rw-------.  1 stack stack 5.5G Mar  2 05:44 compute_0.qcow2
-rw-------.  1 stack stack 6.9G Mar  2 05:45 control_0.qcow2
-rw-------.  1 stack stack  16G Mar  2 05:45 undercloud.qcow2

Undercloud

[diana@localhost ~]$ ssh -F $HOME/.quickstart/ssh.config.ansible undercloud
[stack@undercloud ~]$ source stackrc

[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| e65c3c9c-378d-482a-9d14-3400d6e2b916 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.17 |
| 71fba34f-e58f-4031-932b-0b7c69699039 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.13 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+

[stack@undercloud ~]$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+-----------------+----------------------+--------------+
| 774afb90-c7b5-4e75-ab17-dca00b113323 | overcloud  | CREATE_COMPLETE | 2017-03-02T03:45:36Z | None         |
+--------------------------------------+------------+-----------------+----------------------+--------------+

And what do successful overcloud deploy logs look like now that I finally have them? Behold:
- https://gist.github.com/dianaclarke/f14e8add2b05548282b616eec4233938
- And it does have that magical line grep is looking for: CREATE_COMPLETE Stack CREATE completed successfully
Overcloud
- Step through these docs now: https://github.com/openstack/tripleo-quickstart/blob/master/doc/source/accessing-overcloud.rst

[diana@localhost ~]$ ssh -F $HOME/.quickstart/ssh.config.ansible overcloud-controller-0

tripleo quickstart part 2 - dianaclarke/openstack-notes GitHub Wiki

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️