Zero-Downtime Database Upgrades
Warning
This feature is EXPERIMENTAL in the Ocata, Pike and Queens releases.
We encourage operators to try it out, but its use in production
environments is currently NOT SUPPORTED.
A zero-downtime database upgrade enables true rolling
upgrades of the Glance nodes in your cloud’s control plane. At the
appropriate point in the upgrade, you can have a mixed deployment of
release n (for example, Ocata) and release n-1 (for
example, Newton) Glance nodes, take the n-1 release nodes out
of rotation, allow them to drain, and then take them out of service
permanently, leaving all Glance nodes in your cloud at release
n.
That’s a rough sketch of how a rolling upgrade would work. For full
details, see rolling-upgrades.
Note
When we speak of a “database upgrade”, we are simply talking about
changing the database schema and its data from the version used in
OpenStack release n (say, Pike) to the version used in
OpenStack release n+1 (say, Queens). We are
not talking about upgrading the database management
software.
Note
Downgrading a database is not supported. See downgrades for more
information.
The Expand-Migrate-Contract
Cycle
It’s possible to characterize three phases of a database upgrade:
- Expand: in this phase, new columns, tables,
indexes, are added to the database. - Migrate: in this phase, data is migrated to the new
columns or tables. - Contract: in this phase, the “old” tables or
columns (which are no longer in use) are removed from the database.
The “legacy” Glance database migrations performed these phases as
part of a single monolithic upgrade script. Currently, the Glance
project creates a separate script for each the three parts of the cycle.
We call such an upgrade an E-M-C database
migration.
Zero-Downtime Database
Upgrade
The E-M-C strategy can be performed offline when Glance is not using
the database. With some adjustments, however, the E-M-C strategy can be
applied online when the database is in use, making true rolling upgrades
possible.
Note
Don’t forget that zero-downtime database upgrades are currently
considered experimental and their use in production environments is NOT
SUPPORTED.
A zero-downtime database upgrade takes place as part of a rolling upgrade for upgrading your entire
strategy <rolling-upgrades>
Glance installation. In such a situation, you want to upgrade to release
n of Glance (say, Queens) while your release n-1 API
nodes are still running Pike. To make this possible, in the
Expand phase, database triggers can be added to the
database to keep the data in “old” and “new” columns synchronized.
Likewise, after all data has been migrated and all Glance nodes have
been updated to release n code, these triggers are deleted in
the Contract phase.
Note
Unlike the E-M-C scripts, database triggers are particular to each
database technology. That’s why the Glance project currently provides
experimental support only for MySQL.
New Database Version
Identifiers
In order to perform zero-downtime upgrades, the version identifier of
a database becomes more complicated since it must reflect knowledge of
what point in the E-M-C cycle the upgrade has reached. To make this
evident, the identifier explicitly contains ‘expand’ or ‘contract’ as
part of its name.
Thus the Ocata cycle migration has two identifiers associated with
it: ocata_expand01 and ocata_contract01.
During the upgrade process, the database is initially marked with
ocata_expand01. Eventually, after completing the full
upgrade process, the database will be marked with
ocata_contract01. So, instead of one database version, an
operator will see a composite database version that will have both
expand and contract versions. A database will be considered at Ocata
version only when both expand and contract revisions are at the latest
revisions. For a successful Ocata zero-downtime upgrade, for example,
the database will be marked with both ocata_expand01,
ocata_contract01.
In the case in which there are multiple changes in a cycle, the
database version record would go through the following progression:
| stage | database identifier |
|
|---|---|---|
|
bexar_expand01 |
upgrade begins |
|
bexar_expand02 |
|
|
bexar_expand03 |
|
|
bexar_expand03 |
bexar_migrate01 occurs |
|
bexar_expand03 |
bexar_migrate02 occurs |
|
bexar_expand03 |
bexar_migrate03 occurs |
|
bexar_expand03, bexar_contract01 |
|
|
bexar_expand03, bexar_contract02 |
|
|
bexar_expand03, bexar_contract03 |
upgrade completed |
Database Upgrade
For offline database upgrades, the glance-manage tool
still has the glance-manage db sync command. This command
will execute the expand, migrate, and contract scripts for you, just as
if they were contained in a single script.
In order to enable zero-downtime database upgrades, the
glance-manage tool has been augmented to include the
following operations so that you can explicitly manage the upgrade.
Warning
For MySQL, using the glance-manage db expand or
glance-manage db contract command requires that you either
grant your glance user SUPER privileges, or run
set global log_bin_trust_function_creators=1; in mysql
beforehand.
Expanding the Database
glance-manage db expandThis will run the expansion phase of a rolling upgrade process.
Database expansion should be run as the first step in the rolling
upgrade process before any new services are started.
Migrating the Data
glance-manage db migrateThis will run the data migrate phase of a rolling upgrade process.
Database migration should be run after database expansion but before any
new services are started.
Contracting the Database
glance-manage db contractThis will run the contraction phase of a rolling upgrade process.
Database contraction should be run as the last step of the rolling
upgrade process after all old services are upgraded to new ones.
