image retry bug - dianaclarke/openstack-notes GitHub Wiki

• https://bugzilla.redhat.com/show_bug.cgi?id=1313254
• https://github.com/openstack/nova/blob/master/nova/image/glance.py
• Easier to read traces than the ones in the Red Hat bug report: http://paste.openstack.org/show/490396/
• This is returning a generator, so it will never raise an exception causing it to retry:
  • https://github.com/openstack/nova/blob/83261f3106a8bdde38d258a74da777add4956290/nova/image/glance.py#L249
  • https://github.com/openstack/python-glanceclient/blob/d59e341a4cd99a8488d5cf41052d9b218379ac87/glanceclient/v1/images.py#L268
• Related? https://bugs.launchpad.net/openstack-ansible/+bug/1468393
• The faulty logic is essentially the following (which just prints a generator & never retries):

def call(exception):
    for x in [1, 2, 3]:
        if exception:
            raise Exception("BOOM!")
        yield x

try:
   images = call(exception=True)
except:
   print 'retrying...'
   images = call(exception=False)

print images

<generator object call at 0x10a04e050>

• The goal is something more like this:

def call(exception):
    for x in [1, 2, 3]:
        if exception:
            raise Exception("BOOM!")
        yield x

try:
   images = list(call(exception=True))
except:
   print 'retrying...'
   images = list(call(exception=False))

print images

retrying...
[1, 2, 3]

• The DevStack config I used to manually test this:

$ cat local.conf 
[local](/dianaclarke/openstack-notes/wiki/localrc)
DEST=/opt/stack/
LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log
RABBIT_PASSWORD=123456
MYSQL_PASSWORD=123456
SERVICE_TOKEN=123456
SERVICE_PASSWORD=123456
ADMIN_PASSWORD=123456
DATABASE_QUERY_LOGGING=True

[post-config](/dianaclarke/openstack-notes/wiki/$NOVA_CONF)
[database]
connection_debug=50
[glance]
api_servers=http://192.168.122.30:9292/v1,http://192.168.122.31:9292/v1
num_retries = 5

• Resulting links:
  • Issue: https://bugs.launchpad.net/nova/+bug/1557584
  • Patch: https://review.openstack.org/#/c/293127/