Share replication
Replication of data has a number of use cases in the cloud. One use
case is High Availability of the data in a shared file system, used for
example, to support a production database. Another use case is ensuring
Data Protection; i.e being prepared for a disaster by having a
replication location that will be ready to back up your primary data
source.
The Shared File System service supports user facing APIs that allow
users to create shares that support replication, add and remove share
replicas and manage their snapshots and access rules. Three replication
types are currently supported and they vary in the semantics associated
with the primary share and the secondary copies.
Important
Share replication is an
experimental Shared File Systems API in the Mitaka
release. Contributors can change or remove the experimental part of the
Shared File Systems API in further releases without maintaining backward
compatibility. Experimental APIs have an
X-OpenStack-Manila-API-Experimental: true
header in their
HTTP requests.
Replication types supported
Before using share replication, make sure the Shared File System
driver that you are running supports this feature. You can check it in
the manila-scheduler
service reports. The
replication_type
capability reported can have one of the
following values:
- writable
-
The driver supports creating
writable
share replicas.
All share replicas can be accorded read/write access and would be
synchronously mirrored. - readable
-
The driver supports creating
read-only
share replicas.
All secondary share replicas can be accorded read access. Only the
primary (oractive
share replica) can be written into. - dr
-
The driver supports creating
dr
(abbreviated from
Disaster Recovery) share replicas. A secondary share replica is
inaccessible until after apromotion
. - None
-
The driver does not support Share Replication.
Note
The term active
share replica refers to the
primary
share. In writable
style of
replication, all share replicas are active
, and there could
be no distinction of a primary
share. In
readable
and dr
styles of replication, a
secondary
share replica may be referred to as
passive
, non-active
or simply,
replica
.
Configuration
Two new configuration options have been introduced to support Share
Replication.
- replica_state_update_interval
-
Specify this option in the
DEFAULT
section of your
manila.conf
. The Shared File Systems service requests
periodic update of the replica_state of
allnon-active
share replicas. The update occurs with
respect to an interval corresponding to this option. If it is not
specified, it defaults to 300 seconds. - replication_domain
-
Specify this option in the backend stanza when using a multi-backend
style configuration. The value can be any ASCII string. Two backends
that can replicate between each other would have the same
replication_domain
. This comes from the premise that the
Shared File Systems service expects Share Replication to be performed
between symmetric backends. This option is required for using
the Share Replication feature.
Health of a share replica
Apart from the status
attribute, share replicas have the
replica_state
attribute to denote the state of data
replication on the storage backend. The primary
share
replica will have it’s replica_state
attribute set to active. The
secondary
share replicas may have one of the following as
their replica_state
:
- in_sync
-
The share replica is up to date with the
active
share
replica (possibly within a backend-specific
recovery point objective
). - out_of_sync
-
The share replica is out of date (all new share replicas start out in
thisreplica_state
). - error
-
When the scheduler fails to schedule this share replica or some
potentially irrecoverable error occurred with regard to updating data
for this replica.
Promotion or failover
For readable
and dr
types of replication,
we refer to the task of switching a non-active share replica with the
active
replica as promotion.
For the writable
style of replication, promotion does not
make sense since all share replicas are active
(or
writable) at all times.
The status attribute of the non-active
replica being promoted will be set to replication_change
during its promotion. This has been classified as a busy
state and thus API interactions with the share are restricted while one
of its share replicas is in this state.
Share replication workflows
The following examples have been implemented with the ZFSonLinux
driver that is a reference implementation in the Shared File Systems
service. It operates in driver_handles_share_servers=False
mode and supports the readable
type of replication. In the
example, we assume a configuration of two Availability Zones1, called availability_zone_1 and availability_zone_2.
Since the Train release, some drivers operating in
driver_handles_share_server=True
mode support share
replication.
Multiple availability zones are not necessary to use the replication
feature. However, the use of an availability zone as a
failure domain
is encouraged.
Pay attention to the network configuration for the ZFS driver. Here,
we assume a configuration of zfs_service_ip
and
zfs_share_export_ip
from two separate networks. The service
network is reachable from the host where the manila-share
service is running. The share export IP is from a network that allows
user access.
See Configuring the ZFSonLinux driver
<https://docs.openstack.org/manila/latest
/configuration/shared-file-systems/drivers/zfs-on-linux-driver.html>
for information on how to set up the ZFSonLinux driver.
Creating a share
that supports replication
Create a new share type and specify the replication_type as an extra-spec within the
share-type being used.
Use the manila type-create
command to create a new share
type. Specify the name and the value for the extra-spec
driver_handles_share_servers
.
$ manila type-create readable_type_replication False
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| required_extra_specs | driver_handles_share_servers : False |
| Name | readable_type_replication |
| Visibility | public |
| is_default | - |
| ID | 3b3ee3f7-6e43-4aa1-859d-0b0511c43074 |
| optional_extra_specs | snapshot_support : True |
+----------------------+--------------------------------------+
Use the manila type-key
command to set an extra-spec to
the share type.
$ manila type-key readable_type_replication set replication_type=readable
Note
This command has no output. To verify the extra-spec, use the manila extra-specs-list
command and specify the share type’s name or ID as a parameter.
Create a share with the share type
Use the manila create
command to create a share. Specify
the share protocol, size and the availability zone.
$ manila create NFS 1 --share_type readable_type_replication --name my_share --description "This share will have replicas" --az availability_zone_1
+-----------------------------+--------------------------------------+
| Property | Value |
+-----------------------------+--------------------------------------+
| status | creating |
| share_type_name | readable_type_replication |
| description | This share will have replicas |
| availability_zone | availability_zone_1 |
| share_network_id | None |
| share_server_id | None |
| share_group_id | None |
| host | |
| access_rules_status | active |
| snapshot_id | None |
| is_public | False |
| task_state | None |
| snapshot_support | True |
| id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| size | 1 |
| name | my_share |
| share_type | 3b3ee3f7-6e43-4aa1-859d-0b0511c43074 |
| has_replicas | False |
| replication_type | readable |
| created_at | 2016-03-29T20:22:18.000000 |
| share_proto | NFS |
| project_id | 48a5ca76ac69405e99dc1c13c5195186 |
| metadata | {} |
+-----------------------------+--------------------------------------+
Note
If you are creating a share with the share type specification
driver_handles_share_servers=True
, the share network
parameter is required for the operation to be performed.
Use the manila show
command to retrieve details of the
share. Specify the share ID or name as a parameter.
$ manila show my_share
+-----------------------------+--------------------------------------------------------------------+
| Property | Value |
+-----------------------------+--------------------------------------------------------------------+
| status | available |
| share_type_name | readable_type_replication |
| description | This share will have replicas |
| availability_zone | availability_zone_1 |
| share_network_id | None |
| export_locations | |
| | path = |
| |10.32.62.26:/alpha/manila_share_38efc042_50c2_4825_a6d8_cba2a8277b28|
| | preferred = False |
| | is_admin_only = False |
| | id = e1d754b5-ec06-42d2-afff-3e98c0013faf |
| | share_instance_id = 38efc042-50c2-4825-a6d8-cba2a8277b28 |
| | path = |
| |172.21.0.23:/alpha/manila_share_38efc042_50c2_4825_a6d8_cba2a8277b28|
| | preferred = False |
| | is_admin_only = True |
| | id = 6f843ecd-a7ea-4939-86de-e1e01d9e8672 |
| | share_instance_id = 38efc042-50c2-4825-a6d8-cba2a8277b28 |
| share_server_id | None |
| share_group_id | None |
| host | openstack4@zfsonlinux_1#alpha |
| access_rules_status | active |
| snapshot_id | None |
| is_public | False |
| task_state | None |
| snapshot_support | True |
| id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| size | 1 |
| name | my_share |
| share_type | 3b3ee3f7-6e43-4aa1-859d-0b0511c43074 |
| has_replicas | False |
| replication_type | readable |
| created_at | 2016-03-29T20:22:18.000000 |
| share_proto | NFS |
| project_id | 48a5ca76ac69405e99dc1c13c5195186 |
| metadata | {} |
+-----------------------------+--------------------------------------------------------------------+
Note
When you create a share that supports replication, an
active
replica is created for you. You can verify this with
the manila share-replica-list
command.
From API version 2.53, when creating a replicated share, the manila
quota system will reserve and consume resources for two additional
quotas: share_replicas
and
replica_gigabytes
.
Creating and promoting
share replicas
Create a share replica
Use the manila share-replica-create
command to create a
share replica. Specify the share ID or name as a parameter. You may
optionally provide the availability_zone.
$ manila share-replica-create my_share --az availability_zone_2
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | creating |
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| availability_zone | availability_zone_2 |
| created_at | 2016-03-29T20:24:53.148992 |
| updated_at | None |
| share_network_id | None |
| share_server_id | None |
| host | |
| replica_state | None |
| id | 78a5ef96-6c36-42e0-b50b-44efe7c1807e |
+-------------------+--------------------------------------+
See details of the newly created share replica
Note
Since API version 2.51 (Train release), a share network is able to
span multiple subnets in different availability zones. So, when using a
share type with specification
driver_handles_share_servers=True
, users must ensure that
the share network has a subnet in the availability zone that they desire
the share replica to be created in.
Use the manila share-replica-show
command to see details
of the newly created share replica. Specify the share replica’s ID as a
parameter.
$ manila share-replica-show 78a5ef96-6c36-42e0-b50b-44efe7c1807e
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | available |
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| availability_zone | availability_zone_2 |
| created_at | 2016-03-29T20:24:53.000000 |
| updated_at | 2016-03-29T20:24:58.000000 |
| share_network_id | None |
| share_server_id | None |
| host | openstack4@zfsonlinux_2#beta |
| replica_state | in_sync |
| id | 78a5ef96-6c36-42e0-b50b-44efe7c1807e |
+-------------------+--------------------------------------+
See all replicas of the share
Use the manila share-replica-list
command to see all the
replicas of the share. Specify the share ID or name as an optional
parameter.
$ manila share-replica-list --share-id my_share
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| ID | Status | Replica State | Share ID | Host | Availability Zone | Updated At |
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| 38efc042-50c2-4825-a6d8-cba2a8277b28 | available | active | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_1#alpha | availability_zone_1 | 2016-03-29T20:22:19.000000 |
| 78a5ef96-6c36-42e0-b50b-44efe7c1807e | available | in_sync | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_2#beta | availability_zone_2 | 2016-03-29T20:24:58.000000 |
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
Promote the secondary share replica to be the new active replica
Use the manila share-replica-promote
command to promote a
non-active share replica to become the active
replica.
Specify the non-active replica’s ID as a parameter.
$ manila share-replica-promote 78a5ef96-6c36-42e0-b50b-44efe7c1807e
Note
This command has no output.
The promotion may take time. During the promotion, the
replica_state
attribute of the share replica being promoted
will be set to replication_change
.
$ manila share-replica-list --share-id my_share
+--------------------------------------+-----------+--------------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| ID | Status | Replica State | Share ID | Host | Availability Zone | Updated At |
+--------------------------------------+-----------+--------------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| 38efc042-50c2-4825-a6d8-cba2a8277b28 | available | active | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_1#alpha | availability_zone_1 | 2016-03-29T20:32:19.000000 |
| 78a5ef96-6c36-42e0-b50b-44efe7c1807e | available | replication_change | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_2#beta | availability_zone_2 | 2016-03-29T20:32:19.000000 |
+--------------------------------------+-----------+--------------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
Once the promotion is complete, the replica_state
will
be set to active
.
$ manila share-replica-list --share-id my_share
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| ID | Status | Replica State | Share ID | Host | Availability Zone | Updated At |
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
| 38efc042-50c2-4825-a6d8-cba2a8277b28 | available | in_sync | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_1#alpha | availability_zone_1 | 2016-03-29T20:32:19.000000 |
| 78a5ef96-6c36-42e0-b50b-44efe7c1807e | available | active | e496ed61-8f2e-436b-b299-32c3e90991cc | openstack4@zfsonlinux_2#beta | availability_zone_2 | 2016-03-29T20:32:19.000000 |
+--------------------------------------+-----------+---------------+--------------------------------------+-------------------------------+---------------------+----------------------------+
Access rules
Create an IP access rule for the share
Use the manila access-allow
command to add an access rule.
Specify the share ID or name, protocol and the target as parameters.
$ manila access-allow my_share ip 0.0.0.0/0 --access-level rw
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| access_type | ip |
| access_to | 0.0.0.0/0 |
| access_level | rw |
| state | new |
| id | 8b339cdc-c1e0-448f-bf6d-f068ee6e8f45 |
+--------------+--------------------------------------+
Note
Access rules are not meant to be different across the replicas of the
share. However, as per the type of replication, drivers may choose to
modify the access level prescribed. In the above example, even though
read/write access was requested for the share, the driver will provide
read-only access to the non-active replica to the same target, because
of the semantics of the replication type: readable
.
However, the target will have read/write access to the (currently)
non-active replica when it is promoted to become the active
replica.
The manila access-deny
command can be used to remove a
previously applied access rule.
List the export locations of the share
Use the manila share-export-locations-list
command to list
the export locations of a share.
$ manila share-export-location-list my_share
+--------------------------------------+---------------------------------------------------------------------------+-----------+
| ID | Path | Preferred |
+--------------------------------------+---------------------------------------------------------------------------+-----------+
| 3ed3fbf5-2fa1-4dc0-8440-a0af72398cb6 | 10.32.62.21:/beta/subdir/manila_share_78a5ef96_6c36_42e0_b50b_44efe7c1807e| False |
| 6f843ecd-a7ea-4939-86de-e1e01d9e8672 | 172.21.0.23:/alpha/manila_share_38efc042_50c2_4825_a6d8_cba2a8277b28 | False |
| e1d754b5-ec06-42d2-afff-3e98c0013faf | 10.32.62.26:/alpha/manila_share_38efc042_50c2_4825_a6d8_cba2a8277b28 | False |
| f3c5585f-c2f7-4264-91a7-a4a1e754e686 | 172.21.0.29:/beta/subdir/manila_share_78a5ef96_6c36_42e0_b50b_44efe7c1807e| False |
+--------------------------------------+---------------------------------------------------------------------------+-----------+
Identify the export location corresponding to the share replica on
the user accessible network and you may mount it on the target node.
Note
As an administrator, you can list the export locations for a
particular share replica by using the manila share-instance-export-location-list
command
and specifying the share replica’s ID as a parameter.
Snapshots
Create a snapshot of the share
Use the manila snapshot-create
command to create a
snapshot of the share. Specify the share ID or name as a parameter.
$ manila snapshot-create my_share --name "my_snapshot"
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | creating |
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| user_id | 5c7bdb6eb0504d54a619acf8375c08ce |
| description | None |
| created_at | 2016-03-29T21:14:03.000000 |
| share_proto | NFS |
| provider_location | None |
| id | 06cdccaf-93a0-4e57-9a39-79fb1929c649 |
| project_id | cadd7139bc3148b8973df097c0911016 |
| size | 1 |
| share_size | 1 |
| name | my_snapshot |
+-------------------+--------------------------------------+
Show the details of the snapshot
Use the manila snapshot-show
to view details of a
snapshot. Specify the snapshot ID or name as a parameter.
$ manila snapshot-show my_snapshot
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | available |
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| user_id | 5c7bdb6eb0504d54a619acf8375c08ce |
| description | None |
| created_at | 2016-03-29T21:14:03.000000 |
| share_proto | NFS |
| provider_location | None |
| id | 06cdccaf-93a0-4e57-9a39-79fb1929c649 |
| project_id | cadd7139bc3148b8973df097c0911016 |
| size | 1 |
| share_size | 1 |
| name | my_snapshot |
+-------------------+--------------------------------------+
Note
The status
attribute of a snapshot will transition from
creating
to available
only when it is present
on all the share replicas that have their replica_state
attribute set to active
or in_sync
.
Likewise, the replica_state
attribute of a share replica
will transition from out_of_sync
to in_sync
only when all available
snapshots are present on it.
Planned failovers
As an administrator, you can use the manila share-replica-resync
command to attempt to
sync data between active
and non-active
share
replicas of a share before promotion. This will ensure that share
replicas have the most up-to-date data and their relationships can be
safely switched.
$ manila share-replica-resync 38efc042-50c2-4825-a6d8-cba2a8277b28
Note
This command has no output.
Updating attributes
If an error occurs while updating data or replication relationships
(during a promotion
), the Shared File Systems service may
not be able to determine the consistency or health of a share replica.
It may require administrator intervention to make any fixes on the
storage backend as necessary. In such a situation, state correction
within the Shared File Systems service is possible.
As an administrator, you can:
Reset the status
attribute of a share replica
Use the manila share-replica-reset-state
command to reset
the status
attribute. Specify the share replica’s ID as a
parameter and use the --state
option to specify the state
intended.
$ manila share-replica-reset-state 38efc042-50c2-4825-a6d8-cba2a8277b28 --state=available
Note
This command has no output.
Reset the replica_state
attribute
Use the manila share-replica-reset-replica-state
command
to reset the replica_state
attribute. Specify the share
replica’s ID and use the --state
option to specify the
state intended.
$ manila share-replica-reset-replica-state 38efc042-50c2-4825-a6d8-cba2a8277b28 --state=out_of_sync
Note
This command has no output.
Force delete a specified share replica in any state
Use the manila share-replica-delete
command with the
‘–force’ key to remove the share replica, regardless of the state it is
in.
$ manila share-replica-show 9513de5d-0384-4528-89fb-957dd9b57680
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | error |
| share_id | e496ed61-8f2e-436b-b299-32c3e90991cc |
| availability_zone | availability_zone_1 |
| created_at | 2016-03-30T01:32:47.000000 |
| updated_at | 2016-03-30T01:34:25.000000 |
| share_network_id | None |
| share_server_id | None |
| host | openstack4@zfsonlinux_1#alpha |
| replica_state | out_of_sync |
| id | 38efc042-50c2-4825-a6d8-cba2a8277b28 |
+-------------------+--------------------------------------+
$ manila share-replica-delete --force 38efc042-50c2-4825-a6d8-cba2a8277b28
Note
This command has no output.
Use the policy.yaml
file to grant permissions for these
actions to other roles.
Deleting share replicas
Use the manila share-replica-delete
command with the share
replica’s ID to delete a share replica.
$ manila share-replica-delete 38efc042-50c2-4825-a6d8-cba2a8277b28
Note
This command has no output.
Note
You cannot delete the last active
replica with this
command. You should use the manila delete
command to remove the share.
-
When running in a multi-backend configuration, until the
Stein release, deployers could only configure one Availability Zone per
manila configuration file. This is achieved with the option
storage_availability_zone
defined under the
[DEFAULT]
section.Beyond the Stein release, the option
backend_availability_zone
can be specified in each back end
stanza. The value of this configuration option will override any
configuration of thestorage_availability_zone
from the
[DEFAULT]
section.↩︎