taikun.cloud

Taikun Logo
Request Demo
Sign Up
Sign In

Taikun OCP Guide

Table of Contents

Rolling Upgrades

Note

The Rolling Upgrades feature is EXPERIMENTAL and its use in
production systems is currently not supported.

This statement remains true for the Queens release of Glance. What is
the holdup, you ask? Before asserting that the feature is fully
supported, the Glance team needs to have automated tests that perform
rolling upgrades in the OpenStack Continuous Integration gates. The
Glance project team has not had sufficient testing and development
resources in recent cycles to prioritize this work.

The Glance project team is committed to the stability of Glance. As
part of OpenStack, we are committed to The Four Opens. If
the ability to perform rolling upgrades in production systems is
important to you, feel free to participate in the Glance community to
help coordinate and drive such an effort. (We gently remind you that
“participation” includes providing testing and development
resources.)

Scope of this document

This page describes one way to perform a rolling upgrade from Newton
to Ocata for a particular configuration of Glance services. There may be
other ways to perform a rolling upgrade from Newton to Ocata for other
configurations of Glance services, but those are beyond the scope of
this document. For the experimental rollout of rolling upgrades, we
describe only the following simple case.

Prerequisites

  • MySQL/MariaDB 5.5 or later
  • Glance running Images API v2 only
  • Glance not using the Glance Registry
  • Multiple Glance nodes
  • A load balancer or some other type of redirection device is being
    used in front of the Glance nodes in such a way that a node can be
    dropped out of rotation, that is, that Glance node continues running the
    Glance service but is no longer having requests routed to it

Procedure

Following is the process to upgrade Glance with zero downtime:

  1. Backup the Glance database.
  2. Choose an arbitrary Glance node or provision a new node to install
    the new release. If an existing Glance node is chosen, gracefully stop
    the Glance services. In what follows, this node will be referred to as
    the NEW NODE.

Note

Gracefully stopping services

Before stopping the Glance processes on a node, one may choose to
wait until all the existing connections drain out. This could be
achieved by taking the node out of rotation, that is, by ensuring that
requests are no longer routed to that node. This way all the requests
that are currently being processed will get a chance to finish
processing. However, some Glance requests like uploading and downloading
images may last a long time. This increases the wait time to drain out
all connections and consequently the time to upgrade Glance completely.
On the other hand, stopping the Glance services before the connections
drain out will present the user with errors. While arguably this is not
downtime given that Images API requests are continually being serviced
by other nodes, this is nonetheless an unpleasant user experience for
the user whose in-flight request has terminated in an error. Hence, an
operator must be judicious when stopping the services.

  1. Upgrade the NEW NODE with new release and update the
    configuration accordingly. DO NOT start the Glance
    services on the NEW NODE at this time.

  2. Using the NEW NODE, expand the database using the command:

    glance-manage db expand

.. warning::

 For MySQL, using the ``glance-manage db_expand`` command requires that
 you either grant your glance user ``SUPER`` privileges, or run
 ``set global log_bin_trust_function_creators=1;`` in mysql beforehand.

  3. Then, also on the NEW NODE, perform the data migrations using the
    command:

    glance-manage db migrate

    The data migrations must be completed before you proceed to the
    next step.

  4. Start the Glance processes on the NEW NODE. It is now ready to
    receive traffic from the load balancer.

  5. Taking one node at a time from the remaining nodes, for each
    node:

    1. Stop the Glance
      processes gracefully       as described in Step 2, above. Do not
      proceed until the “old” Glance services on the node have been completely
      shut down.
    2. Upgrade the node to the new release (and corresponding
      configuration).
    3. Start the updated Glance processes on the upgraded node.

  6. After ALL of the nodes have been upgraded to run
    the new Glance services, and there are NO nodes running
    any old Glance services, contract the database by running the command
    from any one of the upgraded nodes:

    glance-manage db contract