OneFS 9.3 introduces writable snapshots to the PowerScale data services portfolio, enabling the creation and management of space and time efficient, modifiable copies of regular OneFS snapshots, at a directory path within the /ifs namespace, and which can be accessed and edited via any of the cluster’s file and object protocols, including NFS, SMB, and S3.
The primary focus of writable snaps in 9.3 is disaster recovery testing, where they can be used to quickly clone production datasets, allowing DR procedures to be routinely tested on identical, thin copies of production data. This is of considerable benefit to the growing number of enterprises that are using Isolate & Test, or Bubble networks, where DR testing is conducted inside a replicated environment that closely mimics production.
Other writable snapshot use cases can include parallel processing workloads that span a server fleet, which can be configured to use multiple writable snaps of a single production data set, to accelerate time to outcomes and results. And writable snaps can also be used to build and deploy templates for near-identical environments, enabling highly predictable and scalable dev and test pipelines.
The OneFS writable snapshot architecture provides an overlay to a read-only source snapshot, allowing a cluster administrator to create a lightweight copy of a production dataset using a simple CLI command, and present and use it as a separate writable namespace.
In this scenario, a SnapshotIQ snapshot (snap_prod_1) is taken of the /ifs/prod directory. The read-only ‘snap_prod_1’ snapshot is then used as the backing for a writable snapshot created at /ifs/wsnap. This writable snapshot contains the same subdirectory and file structure as the original ‘prod’ directory, just without the added data capacity footprint.
Internally, OneFS 9.3 introduces a new protection group data structure, ‘PG_WSNAP’, which provides an overlay that allows unmodified file data to be read directly from the source snapshot, while storing only the changes in the writable snapshot tree.
In this example, a file (Head) comprises four data blocks, A through D. A read-only snapshot is taken of the directory containing the Head file. This file is then modified through a copy-on-write operation. As a result, the new Head data, B1, is written to block 102, and the original data block ‘B’ is copied to a new physical block (110). The snapshot pointer now references block 110 and the new location for the original data ‘B’, so the snapshot has its own copy of that block.
Next, a writable snapshot is created using the read-only snapshot as its source. This writable snapshot is then modified, so its updated version of block C is stored in its own protection group (PG_WSNAP). A client then issues a read request for the writable snapshot version of the file. This read request is directed, through the read-only snapshot, to the Head versions of blocks A and D, the read-only snapshot version for block B and the writable snapshot file’s own version of block C (C1 in block 120).
OneFS directory quotas provide the writable snapshots accounting and reporting infrastructure, allowing users to easily view the space utilization of a writable snapshot. Additionally, IFS domains are also used to bound and manage writable snapshot membership. In OneFS, a domain defines a set of behaviors for a collection of files under a specified directory tree. If a directory has a protection domain applied to it, that domain will also affect all of the files and subdirectories under that top-level directory.
When files within a newly created writable snapshot are first accessed, data is read from the source snapshot, populating the files’ metadata, in a process known as copy-on-read (CoR). Unmodified data is read from the source snapshot and any changes are stored in the writable snapshot’s namespace data structure (PG_WSNAP).
Since a new writable snapshot is not copy-on-read up front, its creation is extremely rapid. As files are subsequently accessed, they are enumerated and begin to consume metadata space.
On accessing a writable snapshot file for the first time, a read is triggered from the source snapshot and the file’s data is accessed directly from the read-only snapshot. At this point, the MD5 checksums for both the source file and writable snapshot file are identical. If, for example, the first block of file is overwritten, just that single block is written to the writable snapshot, and the remaining unmodified blocks are still read from the source snapshot. At this point, the source and writable snapshot files are now different, so their MD5 checksums will also differ.
Before writable snapshots can be created and managed on a cluster, the following prerequisites must be met:
- The cluster is running OneFS 9.3 or later with the upgrade committed.
- SnapshotIQ is licensed across the cluster.
Note that for replication environments using writable snapshots and SyncIQ, all target clusters must be running OneFS 9.3 or later, have SnapshotIQ licensed, and provide sufficient capacity for the full replicated dataset.
By default, up to thirty active writable snapshots can be created and managed on a cluster from either the OneFS command-line interface (CLI) or RESTful platform API.
On creation of a new writable snap, all ﬁles contained in the snapshot source, or HEAD, directory tree are instantly available for both reading and writing in the target namespace.
Once no longer required, a writable snapshot can be easily deleted by CLI. Be aware that WebUI configuration of writable snapshots is not available as of OneFS 9.3. That said, a writable snapshot can easily be created from the CLI as follows:
The source snapshot (src-snap) is an existing read-only snapshot (prod1), and the destination path (dst-path) is a new directory within the /ifs namespace (/ifs/test wsnap1). A read-only source snapshot can be generated as follows:
# isi snapshot snapshots create prod1 /ifs/test/prod # isi snapshot snapshots list ID Name Path ------------------------------------------------------- 7142 prod1 /ifs/test/prod
Next, the following command creates a writable snapshot in an ‘active’ state.
# isi snapshot writable create prod1 /ifs/test/wsnap1 # isi snapshot snapshots delete -f prod1 Snapshot "prod1" can't be deleted because it is locked
While the OneFS CLI is not explicitly prevented from unlocking a writable snapshot’s lock on the backing snapshot, it does provide a clear warning.
# isi snap lock view prod1 1 ID: 1 Comment: Locked/Unlocked by Writable Snapshot(s), do not force delete lock. Expires: 2106-02-07T06:28:15 Count: 1 # isi snap lock delete prod1 1 Are you sure you want to delete snapshot lock 1 from s13590? (yes/[no]):
Be aware that a writable snapshot cannot be created on an existing directory. A new directory path must be specified in the CLI syntax, otherwise the command will fail with the following error:
# isi snapshot writable create prod1 /ifs/test/wsnap1 mkdir /ifs/test/wsnap1 failed: File exists
Similarly, if an unsupported path is specified, the following error will be returned:
# isi snapshot writable create prod1 /isf/test/wsnap2 Error in field(s): dst_path Field: dst_path has error: The value: /isf/test/wsnap2 does not match the regular expression: ^/ifs$|^/ifs/ Input validation failed.
A writable snapshot also cannot be created from a source snapshot of the /ifs root directory, and will fail with the following error:
# isi snapshot writable create s1476 /ifs/test/ifs-wsnap Cannot create writable snapshot from a /ifs snapshot: Operation not supported
Be aware that OneFS 9.3 does not currently provide support for scheduled or automated writable snapshot creation.
When it comes to deleting a writable snapshot, OneFS uses the job engine’s TreeDelete job under the hood to unlink all the contents. As such, running the ‘isi snapshots writable delete’ CLI command automatically queues a TreeDelete instance, which the job engine executes asynchronously in order to remove and clean up a writable snapshot’s namespace and contents. However, be aware that the TreeDelete job execution, and hence the data deletion, is not instantaneous. Instead, the writable snapshot’s directories and files are moved under a temporary ‘*.deleted’ directory. For example:
# isi snapshot writable create prod1 /ifs/test/wsnap2 # isi snap writable delete /ifs/test/wsnap2 Are you sure? (yes/[no]): yes # ls /ifs/test prod wsnap2.51dc245eb.deleted wsnap1
Next, this temporary directory is removed in a non-synchronous operation. If the TreeDelete job fails for some reason, the writable snapshot can be deleted using its renamed path. For example:
# isi snap writable delete /ifs/test/wsnap2.51dc245eb.deleted
Deleting a writable snap removes the lock on the backing read-only snapshot so it can also then be deleted, if required, provided there are no other active writable snapshots based off that read-only snapshot.
The deletion of writable snapshots in OneFS 9.3 is also a strictly manual process. There is currently no provision for automated, policy-driven control such as the ability to set a writable snapshot expiry date, or a bulk snapshot deletion mechanism.
The recommended practice is to quiesce any client sessions to a writable snapshot prior to its deletion. Since the backing snapshot can no longer be trusted once its lock is removed during the deletion process, any ongoing IO may experience errors as the writable snapshot is removed.
In the next article in this series, we’ll take a look at writable snapshots monitoring and performance.