SNAPSHOT METADATA ARRANGEMENT FOR EFFICIENT CLOUD INTEGRATED DATA MANAGEMENT

A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation. 120-. (canceled)21. A method , comprising:identifying, by a computing device, a key-value store for a data collection based on a data collection identifier included in a snapshot restore request, wherein the snapshot restore request further comprises a source snapshot identifier for a snapshot;retrieving, by the computing device, a first plurality of records from the key-value store using a local snapshot identifier as a key prefix for each of the first plurality of records, wherein the first plurality of records comprises at least data map records and the local snapshot identifier is mapped to the source snapshot identifier in metadata for the data collection;assembling, by the computing device, data into a data buffer according to the data map records, wherein the data is included in a snapshot data file identified using the local snapshot identifier; andsending, by the computing device, at least the data buffer and information ...

ENABLING DATA REPLICATION PROCESSES BETWEEN HETEROGENEOUS STORAGE SYSTEMS

A system and method for enabling data replication is described. A set of protocol messages can be associated with a set of corresponding set of requests. The associated set of protocol messages can be provided to each of a source storage system and a destination storage system. The source storage system and the destination storage system can be heterogeneous storage systems that implement different types of file systems or file layouts. The source storage system and the destination storage system are enabled to perform a data replication process, in which at least a set of data is to be replicated from the source storage system to the destination storage system, using at least a protocol message from the associated set of protocol messages. The protocol message can correspond to a request for a list of snapshots stored at the source storage system. 1. A method of enabling data replication between heterogeneous storage systems , the method comprising:associating a set of protocol messages with a corresponding set of requests;providing the associated set of protocol messages to each of a source storage system and a destination storage system, the source storage system and the destination storage system implementing different types of file systems or file layouts; andenabling the source storage system and the destination storage system to perform a data replication process, in which at least a set of data is to be replicated from the source storage system to the destination storage system, using at least a protocol message from the associated set of protocol messages, the protocol message corresponding to a request for a list of snapshots stored at the source storage system.2. The method of claim 1 , wherein enabling the source storage system and the destination storage system to perform the data replication process includes:providing, from the destination storage system to the source storage system, a first protocol message corresponding to a request for a reference ...

CLONED VIRTUAL MACHINE DISK REPLICATION

One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory. 1. A method comprising:receiving a virtual machine disk clone operation targeting first storage;determining that the virtual machine disk clone operation targets a virtual machine hosted by a first node having synchronous replication protection with respect to a second node;splitting the virtual machine disk clone operation to create a replication virtual machine disk clone operation;locally implementing the virtual machine disk clone operation upon the first storage; andsending the replication virtual machine disk clone operation to the second node for implementation upon second storage.2. The method of claim 1 , wherein the locally implementing comprises:implementing the virtual machine disk clone operation as a clone create operation to create a virtual machine disk clone of a virtual machine disk of the virtual machine.3. The method of claim 1 , wherein the locally implementing comprises:implementing the virtual machine disk clone operation as a clone delete operation to delete a target virtual machine disk clone of a virtual machine disk of the virtual machine.4. The method of claim 1 , wherein the locally implementing comprises:implementing the virtual machine disk clone operation as a clone rename operation to rename a target virtual machine ...

SYNCHRONOUS REPLICATION BASED CUTOVER ENGINE

Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object. 1. A method comprising:executing an asynchronous replication process to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met;executing a synchronous replication process to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met; andperforming a cutover to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object where operations are committed to both the storage object and the replicated storage object.2. The method of claim 1 , wherein the cutover is performed at a predicted entry point based upon the cutover criteria being met by the synchronous replication operations.3. The method of claim 1 , comprising:defining the cutover criteria as a timespan for performing asynchronous replication.4. The method of claim 1 , ...

FILE SYSTEM OPERATION HANDLING DURING CUTOVER AND STEADY STATE

Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object. 1. A method comprising:asynchronously replicating data of a storage object hosted by a first node to a replicated storage object hosted by a second node;synchronously replicating operations, targeting the storage object, to the replicated storage object, wherein replicated operations targeting the replicated storage object are terminated upon the second node receiving the replicated operations; andredirecting operations from targeting the storage object to targeting the replicated storage object based upon the storage object and the replicated storage object reaching a steady state of synchronous replication.2. The method of claim 1 , wherein the replicated operations are committed by the second node as a background process.3. The method of claim 1 , wherein a replicated operation is a replica of an operation claim 1 , and the method comprising:transmitting the replicated operation to the second node based upon the operation successfully being executed upon the storage object.4. The method of claim 1 , comprising:executing the operations upon the storage object according to a defined ordering.5. The method of claim 4 , ...

HANDLING METADATA OPERATIONS AND TIMESTAMP CHANGES DURING RESYNCHRONIZATION

Techniques are provided for resynchronizing a synchronous replication relationship. Asynchronous incremental transfers are performed to replicate data of a storage object to a replicated storage object. Incoming write requests, targeting the storage object, are logged into a dirty region log during a last asynchronous incremental transfer. Metadata operations, executed on the storage object, are logged into a metadata log during the last asynchronous incremental transfer. Sequence numbers are assigned to the metadata operations based upon an order of execution. The metadata operations are replicated to the replicated storage object for execution according to the sequence numbers, and the dirty regions are replicated to the replicated storage object in response to the metadata operations having been replicated to the replicated storage object. The storage object and replicated storage object are transitioned to a synchronous replication state where incoming operations are synchronously replicated to the replicated storage object. 1. A method comprising:asynchronously transferring data of a storage object of a first node to a replicated storage object at a second node;logging timestamp changes for the storage object and metadata operations executed upon the storage object according to an order of execution;replicating the timestamp changes and the metadata operations to the replicated storage object according to the order of execution; andreplicating modified data of the storage object to the replicated storage object upon the metadata operations being replicated.2. The method of claim 1 , wherein the metadata operations are logged into a metadata log claim 1 , and the method comprisingassigning sequence numbers to the metadata operations within the metadata log based upon the order of execution.3. The method of claim 2 , comprising:logging the timestamp changes into the metadata log.4. The method of claim 2 , wherein the metadata log is a queue within which the ...

Resynchronization to a filesystem synchronous replication relationship endpoint

Techniques are provided for resynchronizing a synchronous replication relationship. Asynchronous incremental transfers are performed to replicate data of a storage object to a replicated storage object. Incoming write requests, targeting the storage object, are logged into a dirty region log during a last asynchronous incremental transfer. Metadata operations, executed on the storage object, are logged into a metadata log during the last asynchronous incremental transfer. Sequence numbers are assigned to the metadata operations based upon an order of execution. The metadata operations are replicated to the replicated storage object for execution according to the sequence numbers, and the dirty regions are replicated to the replicated storage object in response to the metadata operations having been replicated to the replicated storage object. The storage object and replicated storage object are transitioned to a synchronous replication state where incoming operations are synchronously replicated to the replicated storage object.

Resilient implementation of client file operations and replication

One or more techniques and/or computing devices are provided for resilient replication of storage operations. For example, a first storage controller may host first storage having a replication relationship with second storage hosted by a second storage controller. To improve resiliency against transient network issues of a network between the storage controllers, the first storage controller may implement a queue and retry mechanism to retry replication operations not acknowledge back by the second storage controller within a threshold time. The second storage controller may maintain a cumulative sequence number of a latest replication operation performed in order, an operation response map of replication operations performed out of order, and an operation finder map identifying currently implemented replication operations, which may be used to process incoming replication operations. Single write semantics, write order consistency, and reduction of write amplification may be provided.

GRANSETS FOR MANAGING CONSISTENCY GROUPS OF DISPERSED STORAGE ITEMS

One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items. 1. A method comprising:maintaining a first granset for a set of storage items and a second granset for a set of mirrored storage items maintained as replicas of the set of storage items, wherein the first granset and the second granset specify access types, states, fencing properties, redirection properties, and granset identifiers; andutilizing the first granset to process operations targeting the set of storage items and the second granset to process replicated operation targeting the set of mirrored storage items.2. The method of claim 1 , comprising:atomically creating the first granset with a first access type.3. The method of claim 1 , comprising:labeling incoming operations with a generation number, wherein operations labeled with the generation number are executed and incoming operations without the generation number are rejected.4. The method of claim 1 , comprising:atomically creating the first granset with a first access type, a first state, a first fencing property, a first redirection property, and a first ...

METHOD AND APPARATUS TO NEUTRALIZE REPLICATION ERROR AND RETAIN PRIMARY AND SECONDARY SYNCHRONIZATION DURING SYNCHRONOUS REPLICATION

Techniques are provided for neutralizing replication errors. An operation is executed upon a first storage object and is replicated as a replicated operation for execution upon a second storage object. A first error may be received for the replicated operation. Instead of transitioning to an out of sync state and aborting the operation, a wait is performed until a result of the attempted execution of the operation is received. If the first error is the same as a second error returned for the operation, then the operation and replicated operation are considered successful and a synchronous replication relationship is kept in sync. If the first error and the second error are different errors, then an error response is returned for the operation and the synchronous replication relationship is transitioned to out of sync. 120-. (canceled)21. A method , comprising:receiving a replicated operation from a first node that executed an operation from a client upon a first storage object and generated the replicated operation as a replica of the operation for a second node to execute upon a second storage object;attempting, by the second node, to execute the replicated operation upon the second storage object; andin response to receiving an error associated with execution of the replicated operation, determining that the replicated operation is a failed replicated operation and tracking a sequence number of the failed replicated operation within a failed ops cache.22. The method of claim 21 , wherein the sequence number is assigned to both the operation and the replicated operation.23. The method of claim 21 , comprising:returning an error code for an incoming replicated operation having the sequence number mapped within the failed ops cache to the error code.24. The method of claim 21 , wherein the tracking comprises:mapping the sequence number to an error code within the failed ops cache.25. The method of claim 21 , comprising:returning an error code for an incoming ...

Synchronous replication

One or more techniques and/or computing devices are provided for synchronous replication. For example, synchronous replication relationships are established between a first storage object (e.g., a file, a logical unit number (LUN), a consistency group, etc.), hosted by a first storage controller, and a plurality of replication storage objects hosted by other storage controllers. In this way, a write operation to the first storage object is implemented in parallel upon the first storage object and the replication storage objects in a synchronous manner, such as using a zero-copy operation to reduce overhead otherwise introduced by performing copy operations. Reconciliation is performed in response to a failure so that the first storage object and the replication storage objects comprise consistent data. Failed write operations and replication write operations are retried, while enforcing a single write semantic. Dependent write order consistency is enforced for dependent write operations, such as overlapping write operations.

LOW OVERHEAD RESYNCHRONIZATION SNAPSHOT CREATION AND UTILIZATION

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups. 1. A method comprising:determining that a replication relationship, between a first consistency group hosted by a first device and a second consistency group hosted by a second device, has transitioned into an out-of-sync state where a delta exists between the first consistency group and the second consistency group;creating a local rollback base snapshot; andmodifying an active file system of the second device based upon differences between the local rollback base snapshot and a first pseudo common snapshot corresponding to a first point in time representation of the second consistency group.2. The method of claim 1 , comprising:applying data differences, identified within an intercept tracking log of a second pseudo common snapshot corresponding to a second point in time representation of the first consistency group, to the second consistency group.3. The method of claim 2 , wherein the data difference are applied until the second consistency group mirrors the first consistency group.4. The method ...

SYNCHRONOUS REPLICATION FOR STORAGE

One or more techniques and/or computing devices are provided for implementing synchronous replication. For example, a synchronous replication relationship may be established between a first storage controller hosting local storage and a second storage controller hosting remote storage (e.g., replication may be specified at a file, logical unit number (LUN), or any other level of granularity). Data operations and offloaded operations may be implemented in parallel upon the local storage and the remote storage. Error handling operations may be implemented upon the local storage and implement in parallel as a best effort on the remote storage, and a reconciliation may be performed to identify any data divergence from the best effort parallel implementation. Storage area network (SAN) operations may be implemented upon the local storage, and upon local completion may be remotely implemented upon the remote storage. 1. A method comprising:evaluating operations targeting first storage to identify data operations and metadata operations;executing the metadata operations upon the first storage in parallel with executing the data operations based upon the metadata operations being non-overlapping with respect to pending operations;executing and replicating the data operations upon the first storage and second storage in parallel; andstoring an operation into a queue based upon the operation depending upon prior execution of a pending operation.2. The method of claim 1 , comprising:completing the pending operations while the operation is within the queue.3. The method of claim 2 , comprising:de-queuing and executing the operation based upon execution of the pending operations completing.4. The method of claim 1 , comprising:storing the operation into the queue based upon the operation overlapping a region targeted by the pending operation.5. The method of claim 1 , comprising:replicating the operations to create replicated operations to execute upon the second storage.6. The ...

ENABLING DATA INTEGRITY CHECKING AND FASTER APPLICATION RECOVERY IN SYNCHRONOUS REPLICATED DATASETS

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster. 120-. (canceled)21. A method , comprising:creating a snapshot of first storage based upon a snapshot request comprising a replication label;creating a replication snapshot of second storage that is maintained as a replication destination for the first storage; andcomparing the snapshot and the replication snapshot to determine whether synchronous replication between a first controller hosting the first storage and a second controller hosting the second storage is being performed correctly.22. The method of claim 21 , comprising:performing a data integrity validation upon the snapshot and the replication snapshot to determine that data is being correctly replicated by the synchronous replication from the first storage to the second storage based upon the snapshot and the replication snapshot comprising the same data.23. The method of claim 21 , comprising:performing a data integrity validation upon the snapshot and the replication snapshot to determine that data is not being ...

SNAPSHOT METADATA ARRANGEMENT FOR EFFICIENT CLOUD INTEGRATED DATA MANAGEMENT

A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation. 1. A method comprising: identifying each of the plurality of snapshots with a deterministically generated identifier,', 'for the snapshot metadata of each of the plurality of snapshots, using the deterministically generated identifier as a key prefix for the snapshot metadata of the snapshot in the key-value store; and, 'for snapshot metadata of a plurality of snapshots received for a defined data collection, arranging the snapshot metadata of the plurality of snapshots in a key-value store for the defined data collection based on receipt of the snapshot metadata from one or more data sources, wherein arranging the snapshot metadata of the plurality of snapshots comprises,'} arranging the snapshot data of each of the plurality of snapshots into separate data files, and', 'generating snapshot metadata in the key-value store for each of the plurality of snapshots that maps the snapshot data of the snapshot to one or more locations in ...

Synchronous replication for storage area network protocol storage

One or more techniques and/or computing devices are provided for implementing synchronous replication. For example, a synchronous replication relationship may be established between a first storage controller hosting local storage and a second storage controller hosting remote storage (e.g., replication may be specified at a file, logical unit number (LUN), or any other level of granularity). Data operations and offloaded operations may be implemented in parallel upon the local storage and the remote storage. Error handling operations may be implemented upon the local storage and implement in parallel as a best effort on the remote storage, and a reconciliation may be performed to identify any data divergence from the best effort parallel implementation. Storage area network (SAN) operations may be implemented upon the local storage, and upon local completion may be remotely implemented upon the remote storage.

ENABLING DATA INTEGRITY CHECKING AND FASTER APPLICATION RECOVERY IN SYNCHRONOUS REPLICATED DATASETS

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster. 1. A method comprising:determining that synchronous replication, between a first node hosting first storage and a second node hosting second storage and having a synchronous replication relationship with the first node, is being performed; andverifying a dependent write order consistency of replicating operations to the second storage, wherein the dependent write order consistency corresponds to replication operations being executed upon the second storage by the second node in a same order as corresponding operations being executed upon the first storage by the first node.2. The method of claim 1 , comprising:creating a snapshot of the first storage at the first node based upon a snapshot create request.3. The method of claim 2 , comprising:creating a replication snapshot of the second storage at the second node based upon the snapshot create request.4. The method of claim 1 , comprising:determining that the synchronous replication, between the first node and the second node, is ...

SYNCHRONOUS REPLICATION FOR FILE ACCESS PROTOCOL STORAGE

One or more techniques and/or computing devices are provided for implementing synchronous replication. For example, a synchronous replication relationship may be established between a local storage controller hosting local storage and a remote storage controller hosting remote storage (e.g., replication may be specified at a file, logical unit number (LUN), or any other level of granularity). Data file operations may be implemented in parallel upon the local storage and the remote storage. Independent metadata file operations may be independently implemented from data file operations upon the local storage, and upon local completion may be remotely implemented upon the remote storage. In-flight data file operations may be drained before dependent metadata file operations are locally implemented upon the local storage, and upon local completion may be remotely implemented upon the remote storage. 1. A method comprising:receiving, by a storage server, a file operation for local storage hosted by a local storage controller having a synchronous replication relationship with a remote storage controller hosting remote storage;splitting the file operation to create a replication file operation for the remote storage; 'implementing the data file operation upon the local storage and the replication data file operation upon the remote storage in parallel; and', 'responsive to the file operation corresponding to a data file operation and the replication file operation corresponding to a replication data file operation implementing the independent metadata file operation upon the local storage; and', 'responsive to receiving a completion notification for the independent metadata file operation, implementing the replication independent metadata file operation upon the remote storage., 'responsive to the file operation corresponding to an independent metadata file operation that is independent from data file operations and the replication file operation corresponding to a ...

SYNCHRONOUS REPLICATION FOR STORAGE AREA NETWORK PROTOCOL STORAGE

One or more techniques and/or computing devices are provided for implementing synchronous replication. For example, a synchronous replication relationship may be established between a first storage controller hosting local storage and a second storage controller hosting remote storage (e.g., replication may be specified at a file, logical unit number (LUN), or any other level of granularity). Data operations and offloaded operations may be implemented in parallel upon the local storage and the remote storage. Error handling operations may be implemented upon the local storage and implement in parallel as a best effort on the remote storage, and a reconciliation may be performed to identify any data divergence from the best effort parallel implementation. Storage area network (SAN) operations may be implemented upon the local storage, and upon local completion may be remotely implemented upon the remote storage. 1. A method comprising:receiving, by a storage server, an operation for local storage hosted by a first storage controller having a synchronous replication relationship with a second storage controller hosting remote storage;splitting the operation to create a replication operation for the remote storage;responsive to the operation corresponding to a data operation and the replication operation corresponding to a replication data operation, implementing the data operation upon the local storage and the replication data operation upon the remote storage in parallel; andresponsive to the operation corresponding to an offloaded operation and the replication operation corresponding to a replication offloaded operation, implementing the offloaded operation upon the local storage and the replication offloaded operation upon the remote storage in parallel.2. The method of claim 1 , wherein the implementing the offloaded operation comprises:identifying an inflight write operation that overlaps a region targeted by the offloaded operation; andqueuing the inflight ...

Synchronous replication for synchronous mirror copy guarantee

Techniques are provided for synchronous replication for synchronous mirror copy guarantee. A file system dependent technique for synchronous mirror copy guarantee is provided by overriding default behavior of a persistent fence so that the persistent fence is activated to block operations targeting a storage object having a synchronous replication relationship based upon the synchronous replication relationship being out of sync. The default behavior of the persistent fence is overridden to allow operations to be executed upon the storage object based upon the synchronous replication relationship being in sync. A file system independent technique for synchronous mirror copy guarantee is provided by intercepting operations before the operations are received by a file system. The operations are selectively forwarded to the file system or not based upon a state of a synchronous replication relationship.

Synchronous replication for synchronous mirror copy guarantee

Techniques are provided for synchronous replication for synchronous mirror copy guarantee. A file system dependent technique for synchronous mirror copy guarantee is provided by overriding default behavior of a persistent fence so that the persistent fence is activated to block operations targeting a storage object having a synchronous replication relationship based upon the synchronous replication relationship being out of sync. The default behavior of the persistent fence is overridden to allow operations to be executed upon the storage object based upon the synchronous replication relationship being in sync. A file system independent technique for synchronous mirror copy guarantee is provided by intercepting operations before the operations are received by a file system. The operations are selectively forwarded to the file system or not based upon a state of a synchronous replication relationship.

ENABLING DATA INTEGRITY CHECKING AND FASTER APPLICATION RECOVERY IN SYNCHRONOUS REPLICATED DATASETS

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster. 1. A method comprising:creating a snapshot of first storage at a first node having a synchronous replication relationship with a second node hosting second storage based upon a snapshot create request;creating a replication snapshot of the second storage at the second node based upon the snapshot create request; andutilizing the replication snapshot to recover an application to utilize the second storage in place of the first storage.2. The method of claim 1 , wherein the snapshot is an application consistent snapshot associated with the application that utilizes the first storage when the first node is operational.3. The method of claim 1 , wherein the utilizing comprises:recovering the application based upon the first node failing.4. The method of claim 3 , wherein the application utilizes the first storage when the first node is operational and the application utilizes the second storage after the application is recovered and the first node has failed.5. The method of claim 3 , ...

SYNCHRONOUS REPLICATION

One or more techniques and/or computing devices are provided for synchronous replication. For example, synchronous replication relationships are established between a first storage object (e.g., a file, a logical unit number (LUN), a consistency group, etc.), hosted by a first storage controller, and a plurality of replication storage objects hosted by other storage controllers. In this way, a write operation to the first storage object is implemented in parallel upon the first storage object and the replication storage objects in a synchronous manner, such as using a zero-copy operation to reduce overhead otherwise introduced by performing copy operations. Reconciliation is performed in response to a failure so that the first storage object and the replication storage objects comprise consistent data. Failed write operations and replication write operations are retried, while enforcing a single write semantic. Dependent write order consistency is enforced for dependent write operations, such as overlapping write operations. 1. A method comprising:receiving, by a storage server, a write operation targeting a first storage object hosted by a first storage controller, the first storage object having a synchronous replication relationship with a first replication storage object hosted by a second storage controller and a second replication storage object hosted by a third storage controller;splitting the write operation into a first replication write operation targeting the first replication storage object and a second replication write operation targeting the second replication storage object;implementing the write operation upon the first storage object, the first replication write operation upon the first replication storage object, and the second replication write operation upon the second replication storage object in parallel; andwithholding a client acknowledgement that the write operation is complete until the write operation, the first replication write ...

LOW OVERHEAD RESYNCHRONIZATION SNAPSHOT CREATION AND UTILIZATION

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups. 1. A method comprising:receiving, by a storage server, a request to make pseudo common snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group hosted by a second storage controller, the first consistency group and the second consistency group having a synchronous replication relationship;enabling logging, of incoming client write requests, to an intercept tracking log at a first point in time, the intercept tracking log comprising indicators set to indicate whether portions of the first consistency group have been modified or not modified with respect to the second consistency group;performing a first drain without hold operation to drain inflight client write requests occurring before the first point in time;triggering a capture of a first pseudo common snapshot of the second consistency group at the second storage controller without pausing incoming client write requests;performing a second drain without hold operation to drain inflight client ...

FILE SYSTEM OPERATION HANDLING DURING CUTOVER AND STEADY STATE

Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object. 1. A method comprising:asynchronously replicating incremental changes of data within a volume of a storage virtual machine at a first node to a replicated volume at a second node utilizing asynchronous incremental transfers;in response to completing the asynchronous incremental transfers, synchronously replicating operations, targeting the volume, as replicated operations targeting the replicated volume; andin response to reaching a steady state of synchronous replication, migrating the storage virtual machine at the first node to the second node as a replicated storage virtual machine.2. The method of claim 1 , comprising:redirecting clients from accessing the storage virtual machine at the first node to accessing the replicated storage virtual machine at the second node.3. The method of claim 1 , comprising:determining that the asynchronous incremental transfers are complete based upon a cutover criteria.4. The method of claim 1 , wherein the synchronously replicating the operations comprises:executing the operations upon the volume based upon a defined ordering.5. The method of claim 4 , comprising:assigning ...

RESILIENT IMPLEMENTATION OF CLIENT FILE OPERATIONS AND REPLICATION

One or more techniques and/or computing devices are provided for resilient replication of storage operations. For example, a first storage controller may host first storage having a replication relationship with second storage hosted by a second storage controller. To improve resiliency against transient network issues of a network between the storage controllers, the first storage controller may implement a queue and retry mechanism to retry replication operations not acknowledge back by the second storage controller within a threshold time. The second storage controller may maintain a cumulative sequence number of a latest replication operation performed in order, an operation response map of replication operations performed out of order, and an operation finder map identifying currently implemented replication operations, which may be used to process incoming replication operations. Single write semantics, write order consistency, and reduction of write amplification may be provided. 1. A method comprising:receiving, by a second storage controller, a replication operation from a first storage controller, the replication operation corresponding to a replication of a storage operation received by the first storage controller for implementation upon first storage associated with the first storage controller, the replication operation targeting second storage, associated with the second storage controller, based upon a replication relationship between the first storage and the second storage;responsive to a sequence number, assigned to the replication operation, being less than or equal to a cumulative sequence number, terminating the replication operation and responding to the first storage controller that replication operation implementation for the sequence number succeeded; and responsive to the sequence number being specified within an operation response map, terminating the replication operation and responding to the first storage controller that replication ...

Enabling data replication processes between heterogeneous storage systems

A system and method for enabling data replication is described. A set of protocol messages can be associated with a set of corresponding set of requests. The associated set of protocol messages can be provided to each of a source storage system and a destination storage system. The source storage system and the destination storage system can be heterogeneous storage systems that implement different types of file systems or file layouts. The source storage system and the destination storage system are enabled to perform a data replication process, in which at least a set of data is to be replicated from the source storage system to the destination storage system, using at least a protocol message from the associated set of protocol messages. The protocol message can correspond to a request for a list of snapshots stored at the source storage system.

ENABLING DATA INTEGRITY CHECKING AND FASTER APPLICATION RECOVERY IN SYNCHRONOUS REPLICATED DATASETS

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster. 1. A method comprising:defining, by a storage server, a snapshot replication policy rule for a synchronous replication relationship between a first storage controller, hosting first storage, and a second storage controller hosting second storage, the snapshot replication policy rule specifying that a replication label will be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage;creating a snapshot creation policy to issue snapshot create requests comprising the replication label; creating a snapshot of the first storage at the first storage controller; and', 'creating a replication snapshot of the second storage at the second storage controller; and, 'responsive to receiving a snapshot create request comprising the replication labelcomparing the snapshot and the replication snapshot for data integrity validation.2. The method of claim 1 , comprising:responsive to receiving a second snapshot create request not comprising the ...

METHOD TO ADDRESS MISALIGNED HOLES AND WRITES TO END OF FILES WHILE PERFORMING QUICK RECONCILE OPERATION DURING SYNCHRONOUS FILESYSTEM REPLICATION

Techniques are provided for handling misaligned holes and writes beyond end of files during a quick reconciliation process. During quick reconciliation, a read operation is performed to read data from a first storage object and is replicated to a second storage object. If the data read from the first storage object comprises misaligned holes, then a different range of data is read from the first storage object so that aligned holes are read and replicated to the second storage object. If the read operation targets a region beyond an end of the first storage object, then the second storage object is truncated to a size of the first storage object. 1. A method comprising:executing a read operation to a first storage object at a first range of an offset and length, previously targeted by a write operation, during a quick reconciliation process to reconcile the first storage object and a second storage object;evaluating a read response of the read operation to determine that the read response has a misaligned hole;modifying the first range to a second range that targets an aligned hole within the first storage object;executing a second read operation to the first storage object at the second range; andreplicating a second read response of the second read operation to the second storage object.2. The method of claim 1 , comprising:modifying a start offset of the first range to create the second range by rounding the start offset down to a block size used to store the second storage object.3. The method of claim 1 , comprising:modifying an end offset of the first range to create the second range by rounding the end offset up to a nearest aligned block size value as a new length.4. The method of claim 1 , wherein the quick reconciliation process is performed based upon the write operation failing to successfully execute upon the first storage object and a replication of the write operation successfully executing upon the second storage object.5. The method of claim 1 , ...

METHOD AND APPARATUS TO NEUTRALIZE REPLICATION ERROR AND RETAIN PRIMARY AND SECONDARY SYNCHRONIZATION DURING SYNCHRONOUS REPLICATION

Techniques are provided for neutralizing replication errors. An operation is executed upon a first storage object and is replicated as a replicated operation for execution upon a second storage object. A first error may be received for the replicated operation. Instead of transitioning to an out of sync state and aborting the operation, a wait is performed until a result of the attempted execution of the operation is received. If the first error is the same as a second error returned for the operation, then the operation and replicated operation are considered successful and a synchronous replication relationship is kept in sync. If the first error and the second error are different errors, then an error response is returned for the operation and the synchronous replication relationship is transitioned to out of sync. 1. A method comprising:executing an operation upon a first storage object and replicating the operation as a replicated operation for execution upon a second storage object;refraining from aborting the operation and transitioning to an out of sync state based upon receiving a first error for the replicated operation;comparing the first error with a second error received for the operation; andacknowledging the operation and replicated operation as successfully completing based upon the first error being the same as the second error.2. The method of claim 1 , comprising:maintaining the first storage object and the second storage object in a synchronous replication state based upon the first error being the same as the second error.3. The method of claim 1 , comprising:returning an error response to the operation based upon the first error being different than the second error.4. The method of claim 1 , comprising:transitioning from a synchronous replication state to the out of sync state based upon the first error being different than the second error.5. The method of claim 1 , comprising:determining that the first error and the second error correspond ...

NON-DISRUPTIVE TRANSITION TO SYNCHRONOUS REPLICATION STATE

Techniques are provided for transitioning a first storage object and a second storage object from an asynchronous replication state to a synchronous replication state. Metadata operations are logged into a metadata log. Dirty data, modified by data operations executed upon the first storage object, are tracked using a dirty region log. Metadata operations are replicated from the metadata log to the second storage object during a metadata drain phase. During the metadata drain phase, incoming operations are logged into the metadata log when the metadata log has a non-zero size, otherwise, the metadata operations are replicated to the second storage object. After the metadata drain phase, a data drain phase is performed using the dirty region log. 1. A method comprising:logging metadata operations, executed upon a first storage object, into a metadata log;tracking dirty data, modified by data operations executed upon the first storage object, into a dirty region log;replicating the metadata operations from the metadata log to a second storage object during a metadata drain phase;logging a first incoming metadata operation during the metadata drain phase into the metadata log based upon the metadata log having a non-zero size; andreplicating a second incoming metadata operation to the second storage object based upon the metadata log having a zero size.2. The method of claim 1 , comprising:replicating incoming metadata operations to the second storage object based upon a state indicating a transition from the metadata drain phase to a data drain phase where the dirty data is replicated to the second storage object.3. The method of claim 1 , comprising:replicating the dirty data from the first storage object, as indicated by the dirty region log, to the second storage object during a data drain phase based upon completion of the metadata drain phase.4. The method of claim 3 , comprising:replicating incoming metadata operations to the second storage object during the ...

GRANSETS FOR MANAGING CONSISTENCY GROUPS OF DISPERSED STORAGE ITEMS

One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items. 1. A method comprising:constructing, by a storage server, a first granset for a set of storage items grouped into a consistency group hosted by a first storage controller, the first granset specifying a first access type property for the set of storage items;constructing a second granset for a set of mirrored storage items grouped into the consistency group hosted by a second storage controller, the second granset specifying a second access type property for the set of mirrored storage items, the consistency group specifying that data of the set of storage items is to be replicated to the set of mirrored storage items;responsive to receiving a first operation targeting a storage item within the set of storage items, processing the first operation based upon the first access type property; andresponsive to receiving a second operation targeting a mirrored storage item within the set of mirrored storage items, processing the second operation based upon the second access type property.2. The method of claim 1 , wherein the ...

CLONED VIRTUAL MACHINE DISK REPLICATION

One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory. 1. A method comprising:determining, by a storage server, that a virtual machine, stored within first storage hosted by a first storage controller having a synchronous replication relationship with a second storage controller hosting second storage, has synchronous replication protection; and identifying a virtual machine disk clone operation targeting the first storage;', 'determining that the virtual machine disk clone operation targets the virtual machine having the synchronous replication protection;', 'splitting the virtual machine disk clone operation to create a replication virtual machine disk clone operation;', 'locally implementing the virtual machine disk clone operation upon the first storage; and', 'sending the replication virtual machine disk clone operation to the second storage controller for implementation upon the second storage., 'while the synchronous replication relationship is in-sync2. The method of claim 1 , wherein the virtual machine disk clone operation comprises at least one of a clone create operation to create a virtual machine disk clone of a virtual machine disk of the virtual machine claim 1 , a clone delete operation to delete a target virtual machine disk clone of the virtual machine disk claim 1 , and a clone ...

Dynamic recycling algorithm to handle overlapping writes during synchronous replication of application workloads with large number of files

Techniques are provided for overlapping write handling. Overlapping write managers are used to maintain the order that write operations are executed at a first computing environment and replicated to a second computing environment. Overlapping write managers are pre-allocated as available for managing overlapping write operations. A mapping is used to track what overlapping write managers are currently allocated for particular file handles of files. Thus, if an incoming write operation targets a file handle of an already allocated overlapping write manager, then that overlapping write manager is used to execute and replicate the incoming write operation so that the order of execution of overlapping writes by the second computing environment is the same as at the first computing environment. If there is no allocated overlapping write manager for the file handle, then a new overlapping write manager is allocated and utilized.

LOW OVERHEAD RESYNCHRONIZATION SNAPSHOT CREATION AND UTILIZATION

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups. 1. A method comprising:receiving a request to make pseudo common snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group hosted by a second storage controller, the first consistency group and the second consistency group having a synchronous replication relationship;enabling logging, of incoming client write requests, to an intercept tracking log at a first point in time, the intercept tracking log comprising indicators set to indicate whether portions of the first consistency group have been modified or not modified with respect to the second consistency group;triggering a capture of a first pseudo common snapshot of the second consistency group at the second storage controller without pausing incoming client write requests; andcreating a second pseudo common snapshot of the first consistency group at the first storage controller without pausing incoming client write requests, the second pseudo common snapshot capturing the intercept tracking log ...

CLONED VIRTUAL MACHINE DISK REPLICATION

One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory. 1. A method comprising:receiving and executing operations targeting a virtual machine of a first node;filtering the operations to identify virtual machine backup operations based upon an indication that a synchronous replication relationship specifies that virtual machine backups are to be replicated to a second node; andreplicating the virtual machine backup operations to the second node for execution.2. The method of claim 1 , wherein the replicating comprises:refraining from transmitting filtered operations to the second node.3. The method of claim 1 , wherein an operation specifies that a virtual machine backup is to be automatically deleted.4. The method of claim 1 , wherein the operations comprise a file clone operation claim 1 , and wherein the method comprises:replacing the file clone operation with a partial file clone request.5. The method of claim 1 , wherein the operations comprise a LUN clone operation claim 1 , and wherein the method comprises:replacing the LUN clone operation with a partial LUN clone request.6. The method of claim 1 , wherein the first node stores data of the virtual machine within a storage area network (SAN) dataset.7. The method of claim 1 , comprising:executing an operation to clone a LUN at the first node.8. The ...

Method to address misaligned holes and writes to end of files while performing quick reconcile operation during synchronous filesystem replication

Techniques are provided for handling misaligned holes and writes beyond end of files during a quick reconciliation process. During quick reconciliation, a read operation is performed to read data from a first storage object and is replicated to a second storage object. If the data read from the first storage object comprises misaligned holes, then a different range of data is read from the first storage object so that aligned holes are read and replicated to the second storage object. If the read operation targets a region beyond an end of the first storage object, then the second storage object is truncated to a size of the first storage object.

Usage of op logs to synchronize across primary and secondary storage clusters of a cross-site distributed storage system and lightweight op logging

In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.

Enabling data integrity checking and faster application recovery in synchronous replicated datasets

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster.

Synchronous replication based cutover engine

Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object.

Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover—even if various failures occur. The planned failover feature also includes a persistent fence to avoid serving I/O operations during a timing window when both primary data storage and secondary data storage are attempting to have a master role to serve I/O operations and this avoids a split-brain situation.

Low overhead resynchronization snapshot creation and utilization

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups.

Resynchronization of individual volumes of a consistency group (CG) within a cross-site storage solution while maintaining synchronization of other volumes of the CG

Systems and methods are provided for bringing a volume of a consistency group (CG) into an in-synchronization (InSync) state while other volumes of the CG remain in the InSync state. According to an example, in order to support recovery from disruptive events in a manner that ensures a zero recovery point objective (RPO) guarantee and insulates an application making use of the CG from adverse impacts, responsive to a triggering event, a Fast Resync process may first be attempted to promptly bring an affected volume back into an in-synchronization (InSync) state from an out of synchronization (OOS) state while allowing other members of the CG to remain in the InSync state. Should the Fast resync process be unsuccessful in bringing the volume back into the InSync state within a predetermined or configurable time threshold, then a second type of resynchronization process may be employed at the CG level.

Method to address misaligned holes and writes to end of files while performing quick reconcile operation during synchronous filesystem replication

Techniques are provided for handling misaligned holes and writes beyond end of files during a quick reconciliation process. During quick reconciliation, a read operation is performed to read data from a first storage object and is replicated to a second storage object. If the data read from the first storage object comprises misaligned holes, then a different range of data is read from the first storage object so that aligned holes are read and replicated to the second storage object. If the read operation targets a region beyond an end of the first storage object, then the second storage object is truncated to a size of the first storage object.

Low overhead resynchronization snapshot creation and utilization

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups.

Searchable backups

Facilitating a search of backup data is disclosed. Data associated with at least a portion of the backup data is received. A searchable index of the backup data is generated based at least in part on the received data. The searchable index includes an index data indicating a location within the backup data of an object comprising the backup data.

Reseeding a mediator of a cross-site storage solution

Systems and methods for making a cross-site storage solution resilient towards mediator unavailability are provided. According to one embodiment, a stretched storage system is operable to bring a mediator associated with a primary and secondary distributed storage system back into the role of an arbitrator for peered consistency groups (CGs). A mediator reseed status indicator is maintained for multiple CGs to identify when the mediator's status information for a CG is stale. When the mediator becomes available and a local CG is identified as the subject of a mediator reseed process, the master node of the primary that hosts a master copy of a dataset for the local CG performs the reseed process, including: (i) causing relationship status information for the local CG to be updated on the mediator to the current state maintained by the primary; and (ii) resetting the mediator reseed status indicator.

Backup information management

In one embodiment, a method for information management comprises monitoring output from an application, wherein the output is monitored substantially continuously; determining if the output is associated with a predetermined type of version; and saving data associated with the output if the output is associated with the predetermined type of version.

Enabling data integrity checking and faster application recovery in synchronous replicated datasets

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster.

Cloned virtual machine disk replication

One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory.

Data object search and retrieval

Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity

Systems and methods are described for efficiently processing events related to a relationship between a primary copy of data at a primary storage system and a mirror copy of the data at a cross-site secondary storage system of a multi-site distributed storage system. According to an example, a mediator agent that is configured on both primary and secondary storage systems provides coordination and serialization for various events generated in the relationship. The multi-site distributed storage system performs actions based on the event processing such as performing a failover operation from the primary storage system to the secondary storage system or resynchronizing the relationship to ensure application protection and availability.

Resilient implementation of client file operations and replication

One or more techniques and/or computing devices are provided for resilient replication of storage operations. For example, a first storage controller may host first storage having a replication relationship with second storage hosted by a second storage controller. To improve resiliency against transient network issues of a network between the storage controllers, the first storage controller may implement a queue and retry mechanism to retry replication operations not acknowledge back by the second storage controller within a threshold time. The second storage controller may maintain a cumulative sequence number of a latest replication operation performed in order, an operation response map of replication operations performed out of order, and an operation finder map identifying currently implemented replication operations, which may be used to process incoming replication operations. Single write semantics, write order consistency, and reduction of write amplification may be provided.

Reseeding a mediator of a cross-site storage solution

Systems and methods for making a cross-site storage solution resilient towards mediator unavailability are provided. According to one embodiment, a stretched storage system is operable to bring a mediator associated with a primary and secondary distributed storage system back into the role of an arbitrator for peered consistency groups (CGs). A mediator reseed status indicator is maintained for multiple CGs to identify when the mediator's status information for a CG is stale. When the mediator becomes available and a local CG is identified as the subject of a mediator reseed process, the master node of the primary that hosts a master copy of a dataset for the local CG performs the reseed process, including: (i) causing relationship status information for the local CG to be updated on the mediator to the current state maintained by the primary; and (ii) resetting the mediator reseed status indicator.

Performing various operations at the granularity of a consistency group within a cross-site storage solution

Systems and methods are described for efficiently performing various operations at the granularity of a consistency group (CG) within a cross-site storage solution. An example of one of the various operations includes an independent and parallel resynchronization approach that independently brings individual volumes of a CG to a steady state of in-synchronization (InSync), thereby contributing to scalability of CGs by supporting CGs having a large number of member volumes without requiring a change to the resynchronization process. Another example includes preserving dependent write-order consistency when a remote mirror copy goes out-of-synchronization (OOS) for any reason by driving all member volumes OOS responsive to any member volume becoming OOS. Yet another example includes independent creation of snapshots by member volumes to support efficient and on-demand creation by an application of a common snapshots of all or a subset of peered member volumes of a CG with which the application is associated.

Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover—even if various failures occur. The planned failover feature also includes a persistent fence to avoid serving I/O operations during a timing window when both primary data storage and secondary data storage are attempting to have a master role to serve I/O operations and this avoids a split-brain situation.

Methods and storage nodes to decrease delay in resuming input output (I/O) operations after a non-disruptive event for a storage object of a distributed storage system by utilizing asynchronous inflight replay of the I/O operations

In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.

Serializing execution of replication operations

Techniques are provided for handling misaligned holes and writes beyond end of files during a quick reconciliation process. During quick reconciliation, a read operation is performed to read data from a first storage object and is replicated to a second storage object. If the data read from the first storage object comprises misaligned holes, then a different range of data is read from the first storage object so that aligned holes are read and replicated to the second storage object. If the read operation targets a region beyond an end of the first storage object, then the second storage object is truncated to a size of the first storage object.

Snapshot metadata arrangement for efficient cloud integrated data management

A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation.

Cloned virtual machine disk replication

One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory.

Methods and systems to improve input/output (i/o) resumption time during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes monitoring, with a second cluster, heartbeat information received at ultra-short time intervals from a first connection of one or more storage objects of the first cluster, determining, with the second cluster, whether the heartbeat information from the first connection is received during an ultra-short time interval, and intelligently routing heartbeat information from the one or more storage objects of the first cluster from the first connection to a second connection when the heartbeat information from the first connection is not received during the ultra-short time interval.

Usage of OP logs to synchronize across primary and secondary storage clusters of a cross-site distributed storage system and lightweight OP logging

In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.

Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity

Systems and methods are described for efficiently processing events related to a relationship between a primary copy of data at a primary storage system and a mirror copy of the data at a cross-site secondary storage system of a multi-site distributed storage system. According to an example, a mediator agent that is configured on both primary and secondary storage systems provides coordination and serialization for various events generated in the relationship. The multi-site distributed storage system performs actions based on the event processing such as performing a failover operation from the primary storage system to the secondary storage system or resynchronizing the relationship to ensure application protection and availability.

Facilitating immediate performance of volume resynchronization with the use of passive cache entries

Systems and methods for reducing delays between the time at which a need for a resynchronization of data replication between a volume of a local CG and its peer volume of a remote CG is detected and the time at which the resynchronization is triggered (Reseed Time Period) are provided. According to an example, information indicative of the direction of data replication between the volume and the peer volume is maintained within a cache of a node. Responsive to a disruptive operation (e.g., relocation of the volume from an original node to a new node), the Reseed Time Period is lessened by proactively adding a passive cache entry to a cache within the new node at the time the CG relationship is created when the new node represents an HA partner of the original node and prior to the volume coming online when the new node represents a non-HA partner.

Enabling data integrity checking and faster application recovery in synchronous replicated datasets

One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster.

Resilient implementation of client file operations and replication

One or more techniques and/or computing devices are provided for resilient replication of storage operations. For example, a first storage controller may host first storage having a replication relationship with second storage hosted by a second storage controller. To improve resiliency against transient network issues of a network between the storage controllers, the first storage controller may implement a queue and retry mechanism to retry replication operations not acknowledge back by the second storage controller within a threshold time. The second storage controller may maintain a cumulative sequence number of a latest replication operation performed in order, an operation response map of replication operations performed out of order, and an operation finder map identifying currently implemented replication operations, which may be used to process incoming replication operations. Single write semantics, write order consistency, and reduction of write amplification may be provided.

Method and apparatus to neutralize replication error and retain primary and secondary synchronization during synchronous replication

Techniques are provided for neutralizing replication errors. An operation is executed upon a first storage object and is replicated as a replicated operation for execution upon a second storage object. A first error may be received for the replicated operation. Instead of transitioning to an out of sync state and aborting the operation, a wait is performed until a result of the attempted execution of the operation is received. If the first error is the same as a second error returned for the operation, then the operation and replicated operation are considered successful and a synchronous replication relationship is kept in sync. If the first error and the second error are different errors, then an error response is returned for the operation and the synchronous replication relationship is transitioned to out of sync.

File system operation handling during cutover and steady state

Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object.

Facilitating immediate performance of volume resynchronization with the use of passive cache entries

Systems and methods for reducing delays between the time at which a need for a resynchronization of data replication between a volume of a local CG and its peer volume of a remote CG is detected and the time at which the resynchronization is triggered (Reseed Time Period) are provided. According to an example, information indicative of the direction of data replication between the volume and the peer volume is maintained within a cache of a node. Responsive to a disruptive operation (e.g., relocation of the volume from a first node to a second node), the Reseed Time Period is lessened by proactively adding a passive cache entry to a cache within the second node at the time the CG relationship is created when the second node represents an HA partner of the first node and prior to the volume coming online when the second node represents a non-HA partner.

Gransets for managing consistency groups of dispersed storage items

One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

Methods and storage nodes to decrease delay in resuming input output (i/o) operations after a non-disruptive event for a storage object of a distributed storage system by utilizing asynchronous inflight replay of the i/o operations

In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.

Methods and systems to improve input/output (i/o) resumption time by batching multiple non-conflicting operations during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes determining, with a second storage cluster, whether heartbeat information from one or more storage objects of a CG of a first set of CGs is received during a time period, determining an out of sync state for a data replication relationship between the CG of the first set of CGs and a mirrored CG of a second set of CGs when the heartbeat information is not received during the time period and sending a single bulk role change call with a cluster identifier from the second cluster to an external mediator to provide a role change from follower to leader in the second set of CGs.

Methods and systems for a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for providing an automatic unplanned failover (AUFO) feature to guarantee non-disruptive operations (e.g., operations of business enterprise applications, operations of software application) even in the presence of failures including, but not limited to, network disconnection between multiple data centers and failures of a data center or cluster.

Data object search and retrieval

Searching data is disclosed. A searchable index is used to determine which, if any, data objects comprising a set of backup data are responsive to a search query. For the search query, a search result is generated. The search result includes, in the event a first data object that is responsive to the search query comprises a version of one or more other data objects that are responsive to the search query, data indicating that the first data object comprises a version of said one or more other data objects.

Resynchronization to a filesystem synchronous replication relationship endpoint

Techniques are provided for resynchronizing a synchronous replication relationship. Asynchronous incremental transfers are performed to replicate data of a storage object to a replicated storage object. Incoming write requests, targeting the storage object, are logged into a dirty region log during a last asynchronous incremental transfer. Metadata operations, executed on the storage object, are logged into a metadata log during the last asynchronous incremental transfer. Sequence numbers are assigned to the metadata operations based upon an order of execution. The metadata operations are replicated to the replicated storage object for execution according to the sequence numbers, and the dirty regions are replicated to the replicated storage object in response to the metadata operations having been replicated to the replicated storage object. The storage object and replicated storage object are transitioned to a synchronous replication state where incoming operations are synchronously replicated to the replicated storage object.

Gransets for managing consistency groups of dispersed storage items

One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

Low overhead resynchronization snapshot creation and utilization

One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups.

Methods and systems to improve resumption time of input/output (i/o) operations based on prefetching of configuration data and early abort of conflicting workflows during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time for processing of input/output (I/O) operations during an automatic unplanned failover (AUFO). A first storage cluster includes a first set of consistency groups (CGs) and a second storage cluster includes a second mirrored set of CGs. A computer-implemented method includes prefetching, with a user space of the second storage cluster, configuration information from a replicated database prior to starting the AUFO workflow, sending the configuration information to a kernel space of the second storage cluster on a per CG level while queuing the AUFO workflow, and determining if any in progress workflows conflict with the AUFO workflow.

Re-aligning data replication configuration of primary and secondary data serving entities of a cross-site storage solution after a failover event

Systems and methods for re-aligning data replication configuration of a cross-site storage solution after a failover are provided. According to one embodiment, after a failover, the new primary distributed storage system orchestrates flipping of the data replication configuration of a peered consistency group (CG) to reestablish zero RPO and zero RTO protections for the peered CG. The primary causes the secondary distributed storage system to perform an atomic database operation on its remote configuration database to (i) delete an existing source configuration that identifies the secondary as a source of data replication; and (ii) persist a new destination configuration identifying the secondary as a destination of data replication. Then, the primary performs an atomic database operation on its local configuration database to (i) delete an existing destination configuration identifying the primary as the destination; and (ii) persist a new source configuration identifying the distributed storage system as the source.

Synchronous replication

One or more techniques and/or computing devices are provided for synchronous replication. For example, synchronous replication relationships are established between a first storage object (e.g., a file, a logical unit number (LUN), a consistency group, etc.), hosted by a first storage controller, and a plurality of replication storage objects hosted by other storage controllers. In this way, a write operation to the first storage object is implemented in parallel upon the first storage object and the replication storage objects in a synchronous manner, such as using a zero-copy operation to reduce overhead otherwise introduced by performing copy operations. Reconciliation is performed in response to a failure so that the first storage object and the replication storage objects comprise consistent data. Failed write operations and replication write operations are retried, while enforcing a single write semantic. Dependent write order consistency is enforced for dependent write operations, such as overlapping write operations.

Reseeding a mediator of a cross-site storage solution

Systems and methods for making a cross-site storage solution resilient towards mediator unavailability are provided. According to one embodiment, a stretched storage system is operable to bring a mediator associated with a primary and secondary distributed storage system back into the role of an arbitrator for peered consistency groups (CGs). A mediator reseed status indicator is maintained for multiple CGs to identify when the mediator’s status information for a CG is stale. When the mediator becomes available and a local CG is identified as the subject of a mediator reseed process, the master node of the primary that hosts a master copy of a dataset for the local CG performs the reseed process, including: (i) causing relationship status information for the local CG to be updated on the mediator to the current state maintained by the primary; and (ii) resetting the mediator reseed status indicator.

Snapshot metadata arrangement for efficient cloud integrated data management

A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation.

Synchronous replication for file access protocol storage

Methods and systems to reduce latency of input/output (i/o) operations based on file system optimizations during creation of common snapshots for synchronous replicated datasets of a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot process for storage objects. A computer-implemented method comprises performing a baseline transfer from at least one storage object of a first storage node to at least one replicated storage object of a second storage node, starting the common snapshot process including stop processing of I/O operations, performing a snapshot create operation on the primary storage site for the at least one storage object of the first storage node, resuming processing of I/O operations, and assigning a new universal unique identifier (UUID) to the at least one storage object of the second storage node after resuming processing of I/O operations with the new UUID to identify when file system contents are different than the baseline transfer.

Methods and systems to reduce latency of input/output (i/o) operations based on consistency point optimizations during creation of common snapshots for synchronous replicated datasets of a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot procedure for storage objects. A computer-implemented method includes initiating a snap create handler operation for a storage object of a batch of storage objects having a plurality of replicated datasets with each replicated dataset having a synchronous replication relationship between at least one storage object of the first storage node and at least one replicated storage object of the second storage node, determining whether a consistency point is currently in progress or not, and providing a hint to accelerate a currently in progress consistency point when the consistency point is currently in progress.

Methods and systems to reduce latency of input/output (I/O) operations based on file system optimizations during creation of common snapshots for synchronous replicated datasets of a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot process for storage objects. A computer-implemented method comprises performing a baseline transfer from at least one storage object of a first storage node to at least one replicated storage object of a second storage node, starting the common snapshot process including stop processing of I/O operations, performing a snapshot create operation on the primary storage site for the at least one storage object of the first storage node, resuming processing of I/O operations, and assigning a new universal unique identifier (UUID) to the at least one storage object of the second storage node after resuming processing of I/O operations with the new UUID to identify when file system contents are different than the baseline transfer.

Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity

Systems and methods are described for efficiently processing events related to a relationship between a primary copy of data at a primary storage system and a mirror copy of the data at a cross-site secondary storage system of a multi-site distributed storage system. According to an example, a mediator agent that is configured on both primary and secondary storage systems provides coordination and serialization for various events generated in the relationship. The multi-site distributed storage system performs actions based on the event processing such as performing a failover operation from the primary storage system to the secondary storage system or resynchronizing the relationship to ensure application protection and availability.

Methods and storage nodes to decrease delay in resuming input output (i/o) operations after a non-disruptive event for a storage obect of a distributed storage system by utilizing asynchronous inflight replay of the i/o operations

In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.

Methods and Systems for Storage Virtual Machine Migration Between Clusters of a Networked Storage System

Methods and systems for Vserver migration are provided. Update after claims are finalized. One method includes generating a consistency group (CG) having a plurality of source storage volumes of a source storage virtual machine (Vserver) of a source cluster for a migrate operation to migrate the source storage volumes as a group to a plurality of destination storage volumes of a destination cluster; establishing a mirroring relationship between the source and destination cluster for managing asynchronous transfer of the source storage volumes in the CG to the destination storage volumes during a transfer phase of the migrate operation; replicating a logical interface of the source cluster to the destination cluster, the logical interface providing a network address to access the source cluster; and automatically selecting a destination port at the destination cluster, associated with the replicated logical interface.

Methods and systems to reduce latency of input/output (i/o) operations during creation of common snapshots based on batching of synchronous replicated datasets at a primary storage system to a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot procedure for storage objects. A computer-implemented method includes initiating a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets that belong to a first group of storage disks at the primary storage site and corresponding second group of storage disks at the secondary storage site, performing a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to effectively utilize processing resources on the primary storage site, and initiating an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.

Usage of op logs to synchronize across primary and secondary storage clusters of a cross-site distributed storage system and lightweight op logging

In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.

Methods and systems to improve resumption time of input/output (I/O) operations based on prefetching of configuration data and early abort of conflicting workflows during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time for processing of input/output (I/O) operations during an automatic unplanned failover (AUFO). A first storage cluster includes a first set of consistency groups (CGs) and a second storage cluster includes a second mirrored set of CGs. A computer-implemented method includes prefetching, with a user space of the second storage cluster, configuration information from a replicated database prior to starting the AUFO workflow, sending the configuration information to a kernel space of the second storage cluster on a per CG level while queuing the AUFO workflow, and determining if any in progress workflows conflict with the AUFO workflow.

Methods and Systems for Storage Virtual Machine Migration Between Clusters of a Networked Storage System

Methods and systems for Vserver migration are provided. One method includes maintaining a state of a migrate operation for migrating a plurality of source storage volumes managed by a source storage virtual machine (Vserver) of a source cluster to a plurality of destination storage volumes of a destination cluster of a networked storage environment; restarting a process at a healthy node of the source cluster or the destination cluster to continue the migrate operation, in response to detecting an unhealthy node at the source cluster or the destination cluster executing the process; retrying a task associated with the migrate operation experiencing intermittent failure for a certain number of times, and upon successful execution, continuing the migration operation; and checking the state of the migrate operation and in response to the state of the migrate operation, continuing the migrate operation or restarting the migration operation.

Resynchronization to a filesystem synchronous replication relationship endpoint

Techniques are provided for resynchronizing a synchronous replication relationship. Asynchronous incremental transfers are performed to replicate data of a storage object to a replicated storage object. Incoming write requests, targeting the storage object, are logged into a dirty region log during a last asynchronous incremental transfer. Metadata operations, executed on the storage object, are logged into a metadata log during the last asynchronous incremental transfer. Sequence numbers are assigned to the metadata operations based upon an order of execution. The metadata operations are replicated to the replicated storage object for execution according to the sequence numbers, and the dirty regions are replicated to the replicated storage object in response to the metadata operations having been replicated to the replicated storage object. The storage object and replicated storage object are transitioned to a synchronous replication state where incoming operations are synchronously replicated to the replicated storage object.