taikun.cloud

Taikun OCP Guide

Table of Contents

Upgrading the Shared File System service

This document outlines steps and notes for operators for reference
when upgrading their Shared File System service (manila) from previous
versions of OpenStack. The service aims to provide a minimal downtime
upgrade experience. Since the service does not operate in the data
plane, the accessibility of any provisioned resources such as shares,
share snapshots, share groups, share replicas, share servers, security
services and share networks will not be affected during an upgrade.
Clients can continue to actively use these resources while the service
control plane is being upgraded.

Plan the upgrade

It is highly recommended that you:

  • update the Shared File System service to the latest code from the
    release you are currently using.
  • read the Shared File
    System service release notes
    for the release that you intended to
    upgrade to. Pay special attention to the deprecations and upgrade
    notes.
  • consider the impact of the service control plane upgrade to your
    cloud’s users. The upgrade process interrupts provisioning of new shared
    file systems and associated resources. It also prevents management
    operations on existing shared file systems and associated resources.
    Data path access to shared file systems will remain uninterrupted.
  • take a backup of the shared file system service database so you can
    rollback any failed upgrades to a previous version of the software.
    Although the manila-manage command offers a database
    downgrade command, it is not supported for production use. The only way
    to recover from a failed update is to restore the database from a
    backup.
  • identify your Shared File System service back end storage
    systems/solutions and their drivers. Ensure that the version of each
    storage system is supported by the respective driver in the target
    release. If you’re using a storage solution from a third party vendor,
    consult their product pages to determine if the solution is supported by
    the release of OpenStack that you are upgrading to. Many vendors publish
    a support matrix either within this service administration guide, or on
    their own websites. If you find an incompatibility, stop, and determine
    if you have to upgrade the storage solution first.
  • develop an upgrade procedure and assess it thoroughly by using a
    test environment similar to your production environment.

Graceful service shutdown

Shared File System service components (scheduler, share-manager,
data-manager) are python processes listening for messages on a AMQP
queue. When the operator sends SIGTERM signal to the process, they stop
getting new work from the queue, complete any outstanding work and then
terminate.

Database Migration

The Shared File System service only supports cold upgrades, meaning
that the service plane is expected to be down during the database
upgrade. Database upgrades include schema changes as well as data
migrations to accommodate newer versions of the schema. Once upgraded,
downgrading the database is not supported. When the database has been
upgraded, older services may misbehave when accessing database objects,
so ensure all manila-* services are down before you upgrade
the database.

Prune deleted database rows

Shared File System service resources are soft deleted in the
database, so users are able to track instances in the DB that are
created and destroyed in production. Soft-deletion also helps cloud
operators adhere to data retention policies. Not purging soft-deleted
entries affects DB performance as indices grow very large and data
migrations take longer as there is more data to migrate. It is
recommended that you prune the service database before upgrading to
prevent unnecessary data migrations. Pruning permanently deletes soft
deleted database records.

manila-manage db purge <age_in_days>

Upgrade procedure

  1. Ensure you’re running the latest Shared File System service packages
    for the OpenStack release that you currently use.
  2. Run the manila-status upgrade check command to validate
    that the service is ready for upgrade.
  3. Backup the manila database
  4. Gracefully stop all Shared File System service processes. We
    recommend in this order: manila-api, manila-scheduler, manila-share and
    manila-data.

Note

The manila-data service may be processing time consuming data
migrations. Shutting it down will interrupt any ongoing migrations, and
these will not be automatically started when the service comes back up.
You can check the status on ongoing migrations with
manila migration-get-progress command; issue
manila migration-complete for any ongoing migrations that
have completed their data copy phase.

  1. Upgrade all the service packages. If upgrading from distribution
    packages, your system package manager is expected to handle this
    automatically.
  2. Fix any deprecated configuration options used.
  3. Fix any deprecated api policies used.
  4. Run manila-manage db sync from any node with the latest
    manila packages.
  5. Start all the Shared File System service processes.
  6. Inspect the services by running
    manila service-list. If there are any orphaned records, run
    manila-manage service cleanup to delete them.

Upgrade testing

The Shared File System service code is continually tested for upgrade
from a previous release to the current release using Grenade
<https://docs .openstack.org/grenade/latest/>
. Grenade is an
OpenStack test harness project that validates upgrade scenarios between
releases. It uses DevStack to initially perform a base OpenStack install
and then upgrade to a target version. Tests include the creation of a
variety of Shared File System service resources on the prior release,
and verification for their existence and functionality after the
upgrade.