This document outlines steps and notes for operators for reference
when upgrading their Shared File System service (manila) from previous
versions of OpenStack. The service aims to provide a minimal downtime
upgrade experience. Since the service does not operate in the data
plane, the accessibility of any provisioned resources such as shares,
share snapshots, share groups, share replicas, share servers, security
services and share networks will not be affected during an upgrade.
Clients can continue to actively use these resources while the service
control plane is being upgraded.

Plan the upgrade

It is highly recommended that you:

• update the Shared File System service to the latest code from the
  release you are currently using.
• read the Shared File
  System service release notes for the release that you intended to
  upgrade to. Pay special attention to the deprecations and upgrade
  notes.
• consider the impact of the service control plane upgrade to your
  cloud’s users. The upgrade process interrupts provisioning of new shared
  file systems and associated resources. It also prevents management
  operations on existing shared file systems and associated resources.
  Data path access to shared file systems will remain uninterrupted.
• take a backup of the shared file system service database so you can
  rollback any failed upgrades to a previous version of the software.
  Although the manila-manage command offers a database
  downgrade command, it is not supported for production use. The only way
  to recover from a failed update is to restore the database from a
  backup.
• identify your Shared File System service back end storage
  systems/solutions and their drivers. Ensure that the version of each
  storage system is supported by the respective driver in the target
  release. If you’re using a storage solution from a third party vendor,
  consult their product pages to determine if the solution is supported by
  the release of OpenStack that you are upgrading to. Many vendors publish
  a support matrix either within this service administration guide, or on
  their own websites. If you find an incompatibility, stop, and determine
  if you have to upgrade the storage solution first.
• develop an upgrade procedure and assess it thoroughly by using a
  test environment similar to your production environment.

Graceful service shutdown

Shared File System service components (scheduler, share-manager,
data-manager) are python processes listening for messages on a AMQP
queue. When the operator sends SIGTERM signal to the process, they stop
getting new work from the queue, complete any outstanding work and then
terminate.

Database Migration

The Shared File System service only supports cold upgrades, meaning
that the service plane is expected to be down during the database
upgrade. Database upgrades include schema changes as well as data
migrations to accommodate newer versions of the schema. Once upgraded,
downgrading the database is not supported. When the database has been
upgraded, older services may misbehave when accessing database objects,
so ensure all manila-* services are down before you upgrade
the database.

Prune deleted database rows

Shared File System service resources are soft deleted in the
database, so users are able to track instances in the DB that are
created and destroyed in production. Soft-deletion also helps cloud
operators adhere to data retention policies. Not purging soft-deleted
entries affects DB performance as indices grow very large and data
migrations take longer as there is more data to migrate. It is
recommended that you prune the service database before upgrading to
prevent unnecessary data migrations. Pruning permanently deletes soft
deleted database records.

manila-manage db purge <age_in_days>

Upgrade procedure

1. Ensure you’re running the latest Shared File System service packages
   for the OpenStack release that you currently use.
2. Run the manila-status upgrade check command to validate
   that the service is ready for upgrade.
3. Backup the manila database
4. Gracefully stop all Shared File System service processes. We
   recommend in this order: manila-api, manila-scheduler, manila-share and
   manila-data.

1. Upgrade all the service packages. If upgrading from distribution
   packages, your system package manager is expected to handle this
   automatically.
2. Fix any deprecated configuration options used.
3. Fix any deprecated api policies used.
4. Run manila-manage db sync from any node with the latest
   manila packages.
5. Start all the Shared File System service processes.
6. Inspect the services by running
   manila service-list. If there are any orphaned records, run
   manila-manage service cleanup to delete them.

Upgrade testing

The Shared File System service code is continually tested for upgrade
from a previous release to the current release using Grenade
<https://docs .openstack.org/grenade/latest/>. Grenade is an
OpenStack test harness project that validates upgrade scenarios between
releases. It uses DevStack to initially perform a base OpenStack install
and then upgrade to a target version. Tests include the creation of a
variety of Shared File System service resources on the prior release,
and verification for their existence and functionality after the
upgrade.