Data buffer allocation in a non-blocking data services platform using input/output switching fabric

Data buffering allocation in a microprocessor complex for a request of memory allocation is supported through a remote buffer batch allocation protocol. The separation of control and data placement allows simultaneous maximization of microprocessor complex load sharing, and minimization of inter-processor signaling/metadata migration. Separating processing control from data placement allows the location of data buffering to be chosen so as to maximize bus bandwidth utilization and achieve non-blocking switch behavior. This separation reduces the need for inter-processor communication and associated interrupts thus improving computation efficiency and performance.

SYSTEMS, METHODS, AND COMPUTER READABLE MEDIA PROVIDING ARBITRARY SIZING OF DATA EXTENTS

A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block. 1. A method comprising: wherein the first data extent is partially stored within a first physical block of a storage drive, and', 'wherein the first physical block includes a descriptor indicating that the first data extent starts within the first physical block and is offset from a beginning of the first physical block;, 'receiving a read request for a first data extent,'}reading the first physical block as a whole, including a portion of the first data extent stored within the first physical block and the descriptor;parsing the descriptor to determine the offset;accessing the portion of the first data extent using the offset; andreturning a response, including the portion of the first data extent, to a requesting host.2. The method of claim 1 , wherein the first data extent spans the first physical block and a second physical block of the storage drive.3. The method of claim 1 , wherein the descriptor includes a checksum claim 1 , the method further comprising:verifying the first data extent using the checksum.4. The method of claim 1 , wherein the descriptor includes a checksum claim 1 , the method further comprising:causing an input output controller of a storage system to verify the first data extent using the checksum.5. The method of claim 1 , wherein parsing the descriptor to determine the offset ...

Methods for handling storage element failures to reduce storage device failure rates and devices thereof

Methods, non-transitory computer readable media, and computing devices that determine when a storage element of a data storage device has failed. Address(es) mapped to the failed storage element are identified, when the determining indicates that the storage element has failed. Data corresponding to the address(es) is regenerated according to a data loss protection and recovery scheme (e.g., a RAID scheme). The regenerated data is written to other storage element(s) of the data storage device in order to remap the address(es) to the other storage element(s). This technology allows a data storage device (e.g., an SSD) to be repaired in-place following a failure of storage element(s) (e.g., a die) of the data storage device. Advantageously, entire data storage devices do not have to be failed with this technology as a result of a failure of an individual storage element, thereby reducing data storage device failure rates and associated overhead.

Methods for reducing initialization duration and performance impact during configuration of storage drives

A method, non-transitory computer readable medium, and device that assists with reducing initialization duration and performance impact during configuration of storage drives includes identifying a plurality of new storage drives in a storage system. Next, one or more zeroed out storage drives is identified from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives. A volume group comprising the identified one or more zeroed out drives is created and this created volume group is provided for data operation.

Methods for reducing initialization duration and performance impact during configuration of storage drives

A method, non-transitory computer readable medium, and device that assists with reducing initialization duration and performance impact during configuration of storage drives includes identifying a plurality of new storage drives in a storage system. Next, one or more zeroed out storage drives is identified from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives. A volume group comprising the identified one or more zeroed out drives is created and this created volume group is provided for data operation.

Systems, Methods, and Computer Readable Media Providing Arbitrary Sizing of Data Extents

A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block.

Cross-coupled peripheral component interconnect express switch

A plurality of PCIe switch complexes are interposed between a plurality of I/O devices and a plurality of microprocessor complexes. Each PCIe switching complex comprises a plurality of PCIe switches wherein each switch possesses at least one non-transparent port. The non-transparent port is used to cross-couple each PCIe switch creating an active matrix of paths between the HBAs associated with each I/O device and each microprocessor. The paths between each HBA (I/O device) and each microprocessor are mapped using a recursive algorithm providing each I/O device with direct memory access to each microprocessor.

Systems and Methods for Allocating Data Compression Activities in a Storage System

A method, a computing device, and a non-transitory machine-readable medium for allocating data compression activities in a storage system are provided. A method includes tracking, by a storage controller, computing resources corresponding to a storage server. The storage controller processes one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts. The storage controller determines an amount of available computing resources after processing the one or more host read requests. Based on the amount of available computing resources, the storage controller performs inline compression of a first portion of host write requests and background compression of a second portion of the host write requests.

Online Backup to an Object Service Using Bulk Export

A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log. 1. A method , comprising:entering, by a storage controller, tracked input/output (I/O) into a tracking log while a volume maintained by the storage controller is exported as a first set of objects to a bulk export storage device for transport to a cloud storage system, wherein the first set of objects corresponds to a first set of checksums determined during export of the volume;receiving, by the storage controller, a second set of checksums corresponding to a second set of objects from the cloud storage system, the second set of checksums being determined during import of the volume at the cloud storage service from the bulk export storage device;comparing, by the storage controller, the received second set of checksums to the first set of checksums; andadding, by the storage controller, identification of a subset of data from the volume to the tracking log, the subset of data corresponding to an object, from the volume, determined to have a checksum present in the first set of checksums but missing from the second set of checksums based on the comparison.2. The method of claim ...

Asynchronous replication with write concurrency grouping

A method for controlling write order in a remote data storage system used for asynchronous replication. The data backup method includes receiving writes issued concurrently by a host prior to any being completed. An async write manager, such as a software application on a storage controller or on a host, operates to group each of the writes into a concurrency group with a replica write corresponding to each of the host writes being queued for transmittal over a link to a remote data storage system. The replica writes each include a concurrency group identifier, such as a group number or last completed write number, that identifies which concurrency group the replica has been assigned. The method includes transferring the replica writes to the remote data storage system and processing the concurrency group identifiers with a write order manager to determine which subsets of the replica writes to issue concurrently.

Memory efficient lookup structure

A method for mapping a first address space to a second address space is provided. In some embodiments, the method includes creating a first array of lookup entries and one or more second arrays of metadata entries for maintaining an ordering among the lookup entries using a tree structure. Each of the metadata entries includes one or more data index values identifying a corresponding one of the lookup entries by its position in the first array and one or more metadata index values identifying a corresponding one of the metadata entries by its position in one of the one or more second arrays. The method further includes receiving a request including a lookup value, traversing the tree structure to locate a lookup entry corresponding to the lookup value, and when the lookup value is located among the lookup entries, using the located lookup entry to process the request.

System and Method for Recovering a Storage Array

Systems and techniques for recovering a storage array are disclosed. These systems and techniques include determining a size corresponding to a storage stripe of the storage array. Pieces assigned to the storage stripe are identified. A storage configuration corresponding to the pieces assigned to the storage stripe is detected. Ordinal information and parity information are determined corresponding to the pieces assigned to the storage stripe. The size determined corresponding to the storage stripe, identification of the pieces assigned to the storage stripe, the storage configuration, the ordinal information, and the parity information is stored in a data store to reconstruct lost or corrupted metadata corresponding to the storage array. 1. A method comprising:determining a size corresponding to a storage stripe by parsing data from an array of physical storage devices that store the storage stripe;associating pieces with the storage stripe;parsing the pieces associated with the storage stripe to detect a storage configuration corresponding to the storage stripe and determine ordinal information corresponding to the pieces; andin response to loss or corruption of metadata describing the storage stripe, reconstructing the metadata based on the determined size corresponding to the storage stripe, the pieces associated with the storage stripe, the detected storage configuration, and the determined ordinal information.2. The method of claim 1 , wherein determining the size corresponding to the storage stripe includes determining a size of a storage stripe piece claim 1 , the determining of the size of the storage stripe piece comprising:identifying a contiguous portion of sectors that are associated with a same write sequence number; anddetermining a size of the contiguous portion of sectors.3. The method of claim 1 , wherein determining the size corresponding to the storage stripe comprises determining a size of a RAID piece claim 1 , the determining of the size of ...

DATA ACCESS REQUEST MONITORING TO REDUCE SYSTEM RESOURCE USE FOR BACKGROUND OPERATIONS

An I/O processing stack includes a proxy that can provide processing services for access requests to initialized and uninitialized storage regions. For a write request, the proxy stores write information in a write metadata repository. If the write is requested for an address in an initialized storage region of the storage system, the proxy performs a write to the initialized region based on region information in the write I/O access request. If the write is requested for an address in an uninitialized storage region of the storage system, the proxy performs an on-demand initialization of the storage region and then performs a write to the storage region based on region information provided by the proxy. 1. A method for data access in a storage system:receiving an I/O (input/output) access request by a proxy layer of an I/O processing stack in a storage server of the storage system having initialized storage space and uninitialized storage space;storing host write information for a write I/O access request in a host write metadata repository managed by the proxy layer;determining if an address of the write I/O access request corresponds to an initialized storage region of the storage system or an uninitialized storage region of the storage system; andwhen the address corresponds to an uninitialized storage region, performing an on-demand initialization of the storage region, and performing a write of the write I/O access request based on region information provided by the proxy layer.2. The method of claim 1 , wherein the host write metadata repository comprises a delta log.3. The method of claim 1 , wherein receiving the I/O access request further comprises determining if the I/O access request is a read I/O access request or a write I/O access request.4. The method of claim 3 , wherein in response to determining that the I/O access request is a read I/O access request claim 3 , determining if an address of the read I/O request corresponds to an initialized storage ...

Prioritized Data Recovery From An Object Storage Service and Concurrent Data Backup

A system and method are provided for backing up and recovering data that allows the data to be modified and backed up even while recovery is still in progress. In some embodiments, the method includes performing a data recovery procedure on a computing system. The data recovery procedure includes identifying a set of data objects stored on a recovery system; retrieving the set of data objects; and storing data of the retrieved set of data objects to at least one storage device. Data objects may be prioritized so that data that is in demand is retrieved first. Data that is modified during the data recovery procedure is tracked and backed up to an object-storage system during the data recovery procedure. In some embodiments, backing up the modified data is part of an incremental backup procedure that excludes data objects that contains only unmodified data.

SYSTEMS, METHODS, AND COMPUTER PROGRAM PRODUCTS PROVIDING AN ELASTIC SNAPSHOT REPOSITORY

A system, method, and computer program product for the provision of an elastic snapshot repository is disclosed. A snapshot repository with a particular size stores snapshot images. As the used capacity of the snapshot repository exceeds a predetermined threshold, another volume is added from a pool of available volumes. When the used capacity of the snapshot repository is at, or falls below, a lower threshold, a second snapshot repository is created. The schedule associated with the first snapshot repository is transferred to the second snapshot repository. The snapshot images in the first snapshot repository remain available to meet a minimum history requirement. New snapshot images are stored to the second snapshot repository until there are enough snapshot images in the second snapshot repository, alone, to meet the minimum history requirement. The first snapshot repository is deleted in response and the associated volumes released to the pool. 1. A method , comprising:storing a snapshot image of a base volume to a first repository volume in a snapshot repository, the snapshot repository being configured to maintain a minimum number of snapshot images according to a pre-determined history amount;detecting, in response to storage of the snapshot image, that a used capacity of the snapshot repository has exceeded an upper threshold of available space in the snapshot repository; andconcatenating, in response to the detecting, a second repository volume from a pool of available repository volumes to the first repository volume in the snapshot repository.2. The method of claim 1 , further comprising:pre-allocating each repository volume in the pool of available repository volumes prior to assignment to any snapshot repository.3. The method of claim 1 , further comprising:storing a second snapshot image of the base volume to the snapshot repository;detecting, in response to the storing the second snapshot image, that the used capacity of the snapshot repository has ...

METHODS FOR REDUCING INITIALIZATION DURATION AND PERFORMANCE IMPACT DURING CONFIGURATION OF STORAGE DRIVES

A method, non-transitory computer readable medium, and device that assists with reducing initialization duration and performance impact during configuration of storage drives includes identifying a plurality of new storage drives in a storage system. Next, one or more zeroed out storage drives is identified from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives. A volume group comprising the identified one or more zeroed out drives is created and this created volume group is provided for data operation. 1. A method for managing storage drives comprising:identifying, by a computing device, a plurality of new storage drives in a storage system;identifying, by the computing device, one or more zeroed out storage drives from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives; andcreating, by the computing device, a volume group comprising the identified one or more zeroed out drives; andproviding, by the computing device, the created volume group for data operation.2. The method as set forth in wherein the identifying further comprises reading claim 1 , by the computing device claim 1 , the data portion of each of the identified one or more zeroed out storage drives and subsequently performing a full stripe write operation on the identified one or more zeroed out storage drives.3. The method as set forth in further comprising determining claim 2 , by the computing device claim 2 , for a read error when reading the data portion of each of the identified one or more zeroed out storage drives.4. The method as set forth in further comprising determining claim 3 , by the computing device claim 3 , when PI data in each of the identified one or more zeroed out storage drives having the determined read error includes a known constant value.5. The method as set forth in further ...

Prioritized data recovery from an object storage service and concurrent data backup

A system and method are provided for backing up and recovering data that allows the data to be modified and backed up even while recovery is still in progress. In some embodiments, the method includes performing a data recovery procedure on a computing system. The data recovery procedure includes identifying a set of data objects stored on a recovery system; retrieving the set of data objects; and storing data of the retrieved set of data objects to at least one storage device. Data objects may be prioritized so that data that is in demand is retrieved first. Data that is modified during the data recovery procedure is tracked and backed up to an object-storage system during the data recovery procedure. In some embodiments, backing up the modified data is part of an incremental backup procedure that excludes data objects that contains only unmodified data.

Systems, methods, and computer readable media providing arbitrary sizing of data extents

A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block.

Method and architecture to associate enclosure service data with physical devices on a fibre channel loop with soft addresses

An apparatus comprising one or more drive portions and a controller. The one or more drive portions may each comprise one or more drives. The controller may be configured to map correctly correlating addresses to the one or more drives.

Data sharing and recovery within a network of untrusted storage devices using data object fingerprinting

A data sharing method using fingerprinted data objects for sharing data among untrusted network devices. Each peer device is adapted for storing a plurality of data objects, and a fingerprint generator is used to generate a fingerprint for each stored data object available for sharing or for recovery. The fingerprints are stored in a local data store, and a data manager running on one of the computer devices retrieves from another of the computer devices a copy of one of its data objects through the use of the associated fingerprints. The fingerprints include a hash value output from a strong hashing algorithm. The retrieving includes transmitting query messages with the fingerprints of the needed data objects to the networked, peer devices and then verifying the integrity of received data objects by generating a fingerprint of the received data objects that can be compared with the ones provided in the queries.

CROSS-COUPLED PERIPHERAL COMPONENT INTERCONNECT EXPRESS SWITCH

A plurality of PCIe switch complexes are interposed between a plurality of I/O devices and a plurality of microprocessor complexes. Each PCIe switching complex comprises a plurality of PCIe switches wherein each switch possesses at least one non-transparent port. The non-transparent port is used to cross-couple each PCIe switch creating an active matrix of paths between the HBAs associated with each I/O device and each microprocessor. The paths between each HBA (I/O device) and each microprocessor are mapped using a recursive algorithm providing each I/O device with direct memory access to each microprocessor.

Systems and Methods For Allocating Data Compression Activities In A Storage System

A method, a computing device, and a non-transitory machine-readable medium for allocating data compression activities in a storage system are provided. A method includes tracking, by a storage controller, computing resources corresponding to a storage server. The storage controller processes one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts. The storage controller determines an amount of available computing resources after processing the one or more host read requests. Based on the amount of available computing resources, the storage controller performs inline compression of a first portion of host write requests and background compression of a second portion of the host write requests. 1. A method comprising:tracking, by a storage controller, computing resources corresponding to a storage server;processing, by the storage controller, one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts;determining an amount of available computing resources after processing the one or more host read requests;based on the amount of available computing resources, performing inline compression of a first portion of host write requests, wherein an amount of the host write requests included in the first portion is increased based on the amount of available computer resources exceeding a threshold;performing background compression of a second portion of the host write requests, wherein an amount of the host write requests included in the second portion is increased based on the amount of available computer resources being below the ...

Memory Efficient Lookup Structure

A method for mapping a first address space to a second address space is provided. In some embodiments, the method includes creating a first array of lookup entries and one or more second arrays of metadata entries for maintaining an ordering among the lookup entries using a tree structure. Each of the metadata entries includes one or more data index values identifying a corresponding one of the lookup entries by its position in the first array and one or more metadata index values identifying a corresponding one of the metadata entries by its position in one of the one or more second arrays. The method further includes receiving a request including a lookup value, traversing the tree structure to locate a lookup entry corresponding to the lookup value, and when the lookup value is located among the lookup entries, using the located lookup entry to process the request. 1. A method comprising:creating, by a controller, a first array of data lookup entries, a size of the first array being determined based on a maximum number of data lookup entries supported by the controller;creating, by the controller, one or more second arrays of first metadata entries for maintaining an ordering among the data lookup entries in the first array using a tree structure, each of the first metadata entries comprising one or more first data index values identifying a corresponding one of the data lookup entries by its position in the first array and one or more first metadata index values identifying a corresponding one of the first metadata entries by its position in one of the one or more second arrays;receiving, by the controller, a request comprising a lookup value;traversing, by the controller, the tree structure to locate a data lookup entry corresponding to the lookup value from among the data lookup entries in the first array; andin response to locating the lookup value among the data lookup entries in the first array, using the located data lookup entry to process the request.2. ...

Systems and methods for allocating data compression activities in a storage system

A method, a computing device, and a non-transitory machine-readable medium for allocating data compression activities in a storage system are provided. A method includes tracking, by a storage controller, computing resources corresponding to a storage server. The storage controller processes one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts. The storage controller determines an amount of available computing resources after processing the one or more host read requests. Based on the amount of available computing resources, the storage controller performs inline compression of a first portion of host write requests and background compression of a second portion of the host write requests.

INPUT/OUTPUT ROUTERS WITH DUAL INTERNAL PORTS

Dual ported Input/Output (“I/O”) routers couple I/O devices to a cross-coupled switching fabric providing multiple levels of data path redundancy. Each I/O router possesses two or more internal ports allowing each I/O router to access multiple switches in a cross-coupled switching fabric. The additional redundant paths between each I/O device and each microprocessor complex provide additional means to balance data traffic and thereby maximize bandwidth utilization. I/O routers can be interleaved with single HBAs establishing access a switching fabric that uses cross-coupled nontransparent ports thus providing each I/O device with multiple paths upon which to pass data. Data paths are identified by a recursive address scheme that uniquely identifies each data path option available to each I/O device.

DATA BUFFER ALLOCATION IN A NON-BLOCKING DATA SERVICES PLATFORM USING INPUT/OUTPUT SWITCHING FABRIC

Data buffering allocation in a microprocessor complex for a request of memory allocation is supported through a remote buffer batch allocation protocol. The separation of control and data placement allows simultaneous maximization of microprocessor complex load sharing, and minimization of inter-processor signaling/metadata migration. Separating processing control from data placement allows the location of data buffering to be chosen so as to maximize bus bandwidth utilization and achieve non-blocking switch behavior. This separation reduces the need for inter-processor communication and associated interrupts thus improving computation efficiency and performance.

Online backup to an object service using bulk export

A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

ASYNCHRONOUS REPLICATION WITH WRITE CONCURRENCY GROUPING

A method for controlling write order in a remote data storage system used for asynchronous replication. The data backup method includes receiving writes issued concurrently by a host prior to any being completed. An async write manager, such as a software application on a storage controller or on a host, operates to group each of the writes into a concurrency group with a replica write corresponding to each of the host writes being queued for transmittal over a link to a remote data storage system. The replica writes each include a concurrency group identifier, such as a group number or last completed write number, that identifies which concurrency group the replica has been assigned. The method includes transferring the replica writes to the remote data storage system and processing the concurrency group identifiers with a write order manager to determine which subsets of the replica writes to issue concurrently.

Systems and methods for allocating data compression activities in a storage system

A method, a computing device, and a non-transitory machine-readable medium for allocating data compression activities in a storage system are provided. A method includes tracking, by a storage controller, computing resources corresponding to a storage server. The storage controller processes one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts. The storage controller determines an amount of available computing resources after processing the one or more host read requests. Based on the amount of available computing resources, the storage controller performs inline compression of a first portion of host write requests and background compression of a second portion of the host write requests.

Dynamically Sizing a Hierarchical Tree Based on Activity

A method, a computing device, and a non-transitory machine-readable medium for allocating memory to data structures that map a first address space to a second is provided. In some embodiments, the method includes identifying, by a storage system, a pool of memory resources to allocate among a plurality of address maps. Each of the plurality of address maps includes at least one entry that maps an address in a first address space to an address in a second address space. An activity metric is determined for each of the plurality of address maps, and a portion of the pool of memory is allocated to each of the plurality of address maps based on the respective activity metric. The allocating of the portion of the memory pool to a first map may be performed in response to a merge operation being performed on the first map. 1. A method comprising:identifying, by a storage system, a pool of memory resources to allocate among a plurality of address maps, wherein each of the plurality of address maps includes at least one entry that maps an address in a first address space to an address in a second address space;determining an activity metric for each of the plurality of address maps; andallocating a portion of the pool of memory to each of the plurality of address maps based on the respective activity metric.2. The method of claim 1 , wherein each of the plurality of address maps is structured as a hierarchical tree having a first hierarchical level and wherein the pool of memory is shared between the first hierarchical levels of the plurality of address maps.3. The method of claim 2 , wherein the allocating of the portion of the memory pool to a first map of the plurality of address maps is performed in response to a merge operation being performed on the first hierarchical level of the first map.4. The method of claim 3 , wherein the merge operation includes copying a data range descriptor from the first hierarchical level of the first map to a second hierarchical level of ...

Methods for Reducing Initialization Duration and Performance Impact During Configuration of Storage Drives

A method, non-transitory computer readable medium, and device that assists with reducing initialization duration and performance impact during configuration of storage drives includes identifying a plurality of new storage drives in a storage system. Next, one or more zeroed out storage drives is identified from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives. A volume group comprising the identified one or more zeroed out drives is created and this created volume group is provided for data operation.

Systems, methods, and computer readable media providing arbitrary sizing of data extents

A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block.

Read and write load sharing in a storage array via partitioned ownership of data blocks

A system shares I/O load between controllers in a high availability system. For writes, a controller determines based on one or more factors which controller will flush batches of data from write-back cache to better distribute the I/O burden. The determination occurs after the local storage controller caches the data, mirrors it, and confirms write complete to the host. Once it is determined which storage controller will flush the cache, the flush occurs and the corresponding metadata at a second layer of indirection is updated by that determined storage controller (whether or not it is identified as the owner of the corresponding volume to the host, while the volume owner updates metadata at a first layer of indirection). For a host read, the controller that owns the volume accesses the metadata from whichever controller(s) flushed the data previously and reads the data, regardless of which controller had performed the flush.

METHODS FOR HANDLING STORAGE ELEMENT FAILURES TO REDUCE STORAGE DEVICE FAILURE RATES AND DEVICES THEREOF

Methods, non-transitory computer readable media, and computing devices that determine when a storage element of a data storage device has failed. Address(es) mapped to the failed storage element are identified, when the determining indicates that the storage element has failed. Data corresponding to the address(es) is regenerated according to a data loss protection and recovery scheme (e.g., a RAID scheme). The regenerated data is written to other storage element(s) of the data storage device in order to remap the address(es) to the other storage element(s). This technology allows a data storage device (e.g., an SSD) to be repaired in-place following a failure of storage element(s) (e.g., a die) of the data storage device. Advantageously, entire data storage devices do not have to be failed with this technology as a result of a failure of an individual storage element, thereby reducing data storage device failure rates and associated overhead. 1. A method , comprising:regenerating, by a computing device, data corresponding to one or more addresses mapped to a failed storage element of a data storage device according to a data loss protection and recovery scheme; andwriting, by the computing device, the regenerated data to one or more operational storage elements of the data storage device in order to remap the addresses to the operational storage elements.2. The method of claim 1 , wherein the data storage device utilizes block-based storage and the method further comprises receiving claim 1 , by the computing device claim 1 , one or more results from the data storage device in response to one or more requests to scan one or more ranges of logical block addresses claim 1 , in order to identify the addresses mapped to the failed storage element.3. The method of claim 1 , further comprising determining claim 1 , by the computing device claim 1 , that the storage element has failed based on a failure prediction message received from the data storage device.4. The ...

Read and write load sharing in a storage array via partitioned ownership of data blocks

A system shares I/O load between controllers in a high availability system. For writes, a controller determines based on one or more factors which controller will flush batches of data from write-back cache to better distribute the I/O burden. The determination occurs after the local storage controller caches the data, mirrors it, and confirms write complete to the host. Once it is determined which storage controller will flush the cache, the flush occurs and the corresponding metadata at a second layer of indirection is updated by that determined storage controller (whether or not it is identified as the owner of the corresponding volume to the host, while the volume owner updates metadata at a first layer of indirection). For a host read, the controller that owns the volume accesses the metadata from whichever controller(s) flushed the data previously and reads the data, regardless of which controller had performed the flush.

Selective Data Roll-Back and Roll-Forward

A system and method for recovering a dataset is provided that analyzes the dataset as it currently exists in order to determine those portions that do not need to be recovered. In some embodiments, the method includes identifying a dataset stored on a set of storage devices and corresponding to a first point in time. A request to restore the dataset to a second point in time is received, and a subset of the dataset is identified that is different between the first point in time and the second point in time. Data associated with the subset is selectively retrieved that corresponds to the second point in time, and the retrieved data is merged with the dataset stored on the set of storage devices. The two points in time may have any relationship, and in various examples, the method performs a roll-back or a roll-forward of the dataset. 1. A method comprising:identifying a dataset stored on a set of storage devices and corresponding to a first point in time;receiving a request to restore the dataset to a second point in time;identifying a subset of the dataset for which the subset is different between the first point in time and the second point in time;selectively retrieving data associated with the subset and corresponding to the second point in time; andmerging the selectively retrieved data with the dataset stored on the set of storage devices.2. The method of claim 1 , wherein the retrieved data is structured as at least one data object claim 1 , and wherein the identifying of the subset includes:comparing a first manifest recording a first set of data objects associated with a first recovery point to a second manifest recording a second set of data objects associated with a second recovery point to identify a data object that is different between the first set and the second set.3. The method of claim 1 , wherein the identifying of the subset includes tracing a chain of recovery points between the first point in time and the second point in time.4. The method of ...

Online backup to an object service using bulk export

A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Read and Write Load Sharing in a Storage Array Via Partitioned Ownership of Data Blocks

A system shares I/O load between controllers in a high availability system. For writes, a controller determines based on one or more factors which controller will flush batches of data from write-back cache to better distribute the I/O burden. The determination occurs after the local storage controller caches the data, mirrors it, and confirms write complete to the host. Once it is determined which storage controller will flush the cache, the flush occurs and the corresponding metadata at a second layer of indirection is updated by that determined storage controller (whether or not it is identified as the owner of the corresponding volume to the host, while the volume owner updates metadata at a first layer of indirection). For a host read, the controller that owns the volume accesses the metadata from whichever controller(s) flushed the data previously and reads the data, regardless of which controller had performed the flush.

Read and Write Load Sharing in a Storage Array Via Partitioned Ownership of Data Blocks

A system shares I/O load between controllers in a high availability system. For writes, a controller determines based on one or more factors which controller will flush batches of data from write-back cache to better distribute the I/O burden. The determination occurs after the local storage controller caches the data, mirrors it, and confirms write complete to the host. Once it is determined which storage controller will flush the cache, the flush occurs and the corresponding metadata at a second layer of indirection is updated by that determined storage controller (whether or not it is identified as the owner of the corresponding volume to the host, while the volume owner updates metadata at a first layer of indirection). For a host read, the controller that owns the volume accesses the metadata from whichever controller(s) flushed the data previously and reads the data, regardless of which controller had performed the flush.

Dynamically Growing and Shrinking Snapshot Repositories Without Impacting Performance or Latency

A system, method, and computer program product for the provision of a snapshot repository that grows and shrinks over time. The snapshot repository stores snapshot images. When the used capacity of the repository reaches an upper bound of the repository, the image may be wrapped to the start of the repository if the image stored there previously has been deleted. Otherwise another stripe volume is added from a pool of available volumes and the image continues storage there. As older images are deleted according to a retention policy, stripe volumes that become empty may be dis-associated from their associated LBA range and released to the pool or re-mapped to a lower LBA range of the repository. When the empty stripe volume is at the end of the repository, the ending LBA may be adjusted to the end of the last allocated stripe volume as the volume is released.

WRITE-AHEAD LOG MAINTENANCE AND RECOVERY

A system for tracking metadata changes and recovering from system interruptions. With host I/O, corresponding metadata incremental changes are aggregated and stored in a write-ahead log before being performed to their in-memory buffers. As those buffers are flushed, checkpoints are created and stored in the log. As the log wraps to the start, older entries are overwritten after they are freed from any remaining dependencies by newer checkpoints. If metadata entities have not created new checkpoints, they are instructed to in order to free up space for new aggregated batches and checkpoints. After an interruption, the wrap point is located in the log. From the wrap point, the log is scanned backwards to provide checkpoints to metadata entities. The log is then scanned forwards to perform changes specified by aggregated batches. The metadata entities' volatile memory states are recovered to what they were before the interruption. 1. A method , comprising:writing, by a storage controller, an incremental change set in a write ahead log comprising a plurality of incremental change sets in a non-volatile memory identified by a wrap point, the incremental change set corresponding to a condition of metadata in volatile memory for a metadata client, the incremental change set being written to the non-volatile memory prior to an update to the metadata;tracking a checkpoint in the write ahead log to at least one of the plurality of incremental change sets, an oldest incremental change set corresponding to a region beyond the wrap point; anddetermining, from the tracking, that the checkpoint in the write ahead log records a dependency of the metadata client to the oldest incremental change set, a new checkpoint being persisted in the write ahead log for the metadata client.2. The method of claim 1 , wherein the write ahead log is stored in a solid state drive storing the metadata client.3. The method of claim 1 , wherein determining that the checkpoint in the write ahead log ...

Write-ahead log maintenance and recovery

A system for tracking metadata changes and recovering from system interruptions. With host I/O, corresponding metadata incremental changes are aggregated and stored in a write-ahead log before being performed to their in-memory buffers. As those buffers are flushed, checkpoints are created and stored in the log. As the log wraps to the start, older entries are overwritten after they are freed from any remaining dependencies by newer checkpoints. If metadata entities have not created new checkpoints, they are instructed to in order to free up space for new aggregated batches and checkpoints. After an interruption, the wrap point is located in the log. From the wrap point, the log is scanned backwards to provide checkpoints to metadata entities. The log is then scanned forwards to perform changes specified by aggregated batches. The metadata entities' volatile memory states are recovered to what they were before the interruption.

Data access request monitoring to reduce system resource use for background operations

An I/O processing stack includes a proxy that can provide processing services for access requests to initialized and uninitialized storage regions. For a write request, the proxy stores write information in a write metadata repository. If the write is requested for an address in an initialized storage region of the storage system, the proxy performs a write to the initialized region based on region information in the write I/O access request. If the write is requested for an address in an uninitialized storage region of the storage system, the proxy performs an on-demand initialization of the storage region and then performs a write to the storage region based on region information provided by the proxy.

Write-Ahead Log Maintenance and Recovery

A system for tracking metadata changes and recovering from system interruptions is provided. With host I/O, corresponding metadata incremental changes are aggregated and stored in a write-ahead log before being performed to their in-memory buffers. As those buffers are flushed, checkpoints are created and stored in the log. As the log wraps to the start, older entries are overwritten after they are freed from any remaining dependencies by newer checkpoints. If metadata entities have not created new checkpoints, they are instructed to in order to free up space for new aggregated batches and checkpoints. After an interruption, the wrap point is located in the log. From the wrap point, the log is scanned backwards to provide checkpoints to metadata entities. The log is then scanned forwards to perform changes specified by aggregated batches. The metadata entities' volatile memory states are recovered to what they were before the interruption. 1. A method , comprising:writing, by a storage controller, a metadata tracking structure to a first region identified by a wrap point in a write ahead log, wherein the metadata tracking structure comprises an incremental change set or a checkpoint having a variable data length of data regarding a condition of metadata in volatile memory associated with the storage controller, the write ahead log comprising a plurality of incremental change sets and a plurality of checkpoints corresponding to a plurality of metadata clients;tracking, by the storage controller, dependencies of each of the metadata clients recorded in corresponding checkpoints to one or more of the incremental change sets, an oldest incremental change set dependency corresponding to an incremental change set stored closest to the wrap point at a second region beyond the wrap point;determining, by the storage controller from the tracking, whether any checkpoint records a dependency of a corresponding metadata client to the oldest incremental change set stored at the ...

Online Backup to an Object Service Using Bulk Export

A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log. 1. A method , comprising:receiving, by a storage controller, an import complete status from a cloud storage system in response to a completion status query, the import complete status identifying completion of an import of a first set of objects corresponding to a volume maintained by the storage controller at the cloud storage system;requesting, by the storage controller, an object comprising a first set of checksums from the cloud storage system in response to receiving the import complete status, the first set of checksums corresponding to the first set of objects as determined during export of the volume;receiving, by the storage controller, the object comprising the first set of checksums in response to the request;receiving, by a storage controller, a second set of checksums corresponding to a second set of objects from the cloud storage system, the second set of checksums being determined during import of the volume at the cloud storage system; andadding, by the storage controller in response to a comparison between the second and first sets of checksums, identification of ...

Systems, Methods, and Computer Readable Media Providing Arbitrary Sizing of Data Extents

A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block. 1. A method comprising:storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored at least partially to a first block of the plurality of storage blocks;generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of storage blocks and indicates an offset from the beginning of the first block at which the first data extent starts; andstoring the descriptor within the first block.2. The method of claim 1 , wherein the descriptor is stored at a beginning of the first block or is stored at an end of the first block.3. The method of claim 1 , wherein the descriptor further includes a checksum value for the first block.4. The method of claim 1 , wherein the first data extent is one of a plurality of data extents that are included in a stripe that spans across a plurality of physical media including the physical medium claim 1 , wherein the descriptor further includes a Write Sequence Number associated with a sequentially-written segment that includes the stripe.5. The method of claim 1 , wherein the descriptor ...

System and method for recovering a storage array

Systems and techniques for recovering a storage array are disclosed. These systems and techniques include determining a size corresponding to a storage stripe of the storage array. Pieces assigned to the storage stripe are identified. A storage configuration corresponding to the pieces assigned to the storage stripe is detected. Ordinal information and parity information are determined corresponding to the pieces assigned to the storage stripe. The size determined corresponding to the storage stripe, identification of the pieces assigned to the storage stripe, the storage configuration, the ordinal information, and the parity information is stored in a data store to reconstruct lost or corrupted metadata corresponding to the storage array.

Write-ahead log maintenance and recovery

A system for tracking metadata changes and recovering from system interruptions. With host I/O, corresponding metadata incremental changes are aggregated and stored in a write-ahead log before being performed to their in-memory buffers. As those buffers are flushed, checkpoints are created and stored in the log. As the log wraps to the start, older entries are overwritten after they are freed from any remaining dependencies by newer checkpoints. If metadata entities have not created new checkpoints, they are instructed to in order to free up space for new aggregated batches and checkpoints. After an interruption, the wrap point is located in the log. From the wrap point, the log is scanned backwards to provide checkpoints to metadata entities. The log is then scanned forwards to perform changes specified by aggregated batches. The metadata entities' volatile memory states are recovered to what they were before the interruption.

Determining I/O Performance Headroom

A system and method for determining I/O performance headroom that accounts for a real-world workload is provided. In some embodiments, a computing device is provided that is operable to identify a data transaction received by a storage system and directed to a storage device. The computing system identifies an attribute of the data transaction relating to a performance cost of the data transaction and queries a performance profile to determine a benchmark performance level for the storage device. The computing system determines a benchmark performance level for the storage system based on the benchmark performance level for the storage device and compares a metric of the performance of the data transaction with the storage system benchmark performance level to determine remaining headroom of the storage system. 1. A computing device comprising:a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of performance analysis; and identify a data transaction received by a storage system and directed to a storage device;', 'identify an attribute of the data transaction relating to a performance cost of the data transaction;', 'query a performance profile based on the identified attribute to determine a benchmark performance level for the storage device;', 'determine a benchmark performance level for the storage system in performing the data transaction based on the benchmark performance level for the storage device; and', 'compare a metric based on the storage system performing the data transaction to determine a remaining headroom of the storage system., 'a processor coupled to the memory, the processor configured to execute the machine executable code to2. The computing device of claim 1 , wherein the computing device is a separate computing system from the storage system.3. The computing device of claim 1 , wherein the attribute of the data transaction includes at least one of: an I/O ...

Dynamic Transitioning of Protection Information in Array Systems

A system, method, and computer program product is described for providing dynamic enabling and/or disabling of protection information (PI) in array systems during operation. A storage system receives a request to transition a volume from PI disabled to PI enabled during regular operation. The storage system synchronizes and purges the cache associated with the target volume. The storage system initiates an immediate availability format (IAF-PI) process to initialize PI for the associated data blocks of the volume's storage devices. The storage system continues receiving I/O requests as the IAF-PI process sweeps through the storage devices. The storage system inserts and checks PI for the write data as it is written to the storage devices. The storage system inserts PI for requested data above the IAF-PI boundary and checks PI for requested data below the IAF-PI boundary. The transition remains an online process that avoids downtime. 1. A method , comprising:receiving, at a storage controller, a protection information (PI) enable request for a designated volume that has PI disabled, wherein the volume comprises one or more underlying storage devices that are PI capable;purging, by the storage controller, a portion of a cache associated with the designated volume in response to the PI enable request; andinitializing, by the storage controller, PI for data blocks of the one or more underlying storage devices by working through the data blocks and generating and storing the PI associated with the data blocks.2. The method of claim 1 , wherein the initializing comprises an immediate availability function (IAF-PI) procedure claim 1 , the method further comprising:receiving, during the IAF-PI procedure, an input/output request from a host; anddetermining, in response to the input/output request, to process the input/output request based on a target location of the input/output request with respect to an IAF boundary of the IAF-PI procedure, the IAF boundary dividing a ...

Method and apparatus for transferring data between two devices with reduced microprocessor overhead

A method for transferring data from a first device to a second device where the second device has a main data processor and a secondary processor associated therewith. The method includes the steps of (1) transferring a data stream having a control portion and a data portion from the first device to the second device, and (2) processing the data portion with the secondary processor in accordance with the control portion without interrupting the main data processor. A multi-controller apparatus which is useful for practicing the method is also disclosed.

Method for maintaining parity-data consistency in a disk array

A method for assuring consistency between data and parity in a disk array system following a reset or a power failure condition which interrupts the execution of write I/O operations. The method includes the steps of: examining drive activities to identify unfinished write I/O operations due to an interrupt condition; logging information necessary to identify the unfinished operations and the array redundancy groups associated with the unfinished operations into a non-volatile memory; and checking for log entries in the non-volatile memory during a disk array subsystem initialization or the restoration of power. For each unfinished operation identified in the log, the method further includes the steps of: performing a bit-wise exclusive-OR of corresponding portions of the data stored within the associated redundancy group to calculate parity consistent therewith; and writing the calculated parity to the parity storage areas within the associated redundancy group. For an array operating with a failed disk drive member (i.e., in degraded mode), the method also logs information necessary to identify the storage areas to which the unfinished write I/O operations map new data and parity, and some form of the old data and parity information saved to that storage areas to which the unfinished write I/O operations map new data and parity. Following the system reset, or restoration of power, consistent parity is determined for each associated redundancy group.

Employing request queues and completion queues between main processors and I/O processors wherein a main processor is interrupted when a certain number of completion messages are present in its completion queue

Apparatus and associated methods for improving I/O performance in a computing system which includes one or more MPUs and one or more IOPs. I/O requests are queued by a requesting MPU in a memory shared with one or more IOPs. Each IOP is associated with a queue. Each IOP may continue processing queued I/O requests after completing processing on an earlier request. In addition, each MPU is associated with a queue shared with the IOPs. When an IOP completes processing of an I/O request, a completion message is added to the requesting MPU's queue and an interrupt is generated for that MPU. The MPU services all completion messages in its queue when the interrupt is processed. A threshold value is associated with each MPU queue. The threshold value indicates the minimum number of completed I/O requests required before an interrupt request is generated to the MPU. The threshold value for each MPU may be tuned to permit the computing system to balance the need for rapid I/O response time with the need for reduced interrupt overhead processing in the MPU. Various physical embodiments of the invention are disclosed wherein the queue is located in either a shared memory or in a register file. Additionally, various embodiments separate the queue used to store new I/O requests from the queue used to store completion messages while other embodiments merge the two queuing functions into a single queue where each entry contains an I/O request with a completion status field.

Computer bus data transfer reduction with parity circuit

Redundant controller diagnosis using a private lun

Storage subsystem load balancing

Redundant controller diagnosis using a private lun

Methods and associated apparatus within a RAID subsystem (100) having redundant controllers (118.1, 118.2) define a private LUN as a data storage area known and accessible to all controllers in the system and used by them for diagnostic purposes. The methods involve sending a diagnostic write commande (210) to a second controller with instructions for it to write test data to the private LUN. This second controller writes (280) this test data to the private LUN. A first controller, in response (220) to another diagnostic command, then reads (230) this test data from the private LUN and compares it (240) to expected values provided in the diagnostic command. Using the results, it can then be determined which controller, if any, failed. If the second controller fails, then the first controller takes over ownership of portions of the data storage area assigned to the second controller. The private LUN is preferably striped across all channels used by the controllers to communicate to commonly attached disk drives. This allows the diagnostic process to test disk channel data paths in determining whether a controller has failed. The test data preferably include a data portion and a redundancy portion to enable testing of redundancy computations within the controllers. In an alternate embodiment, a host computer (120.1, 120.2) attached via an interface (154.3) in common with the redundant controllers (118.1, 118.2) initiates and controls the diagnostic process to enable testing of the host/controller communication paths. Timed event messages (e.g., watchdog timer features) may be used in conjunction with other methods of the invention to further enhance failure detection.

Process for producing compact coherent yarns and products therefrom

Process for producing compact coherent yarns and products therefrom

Determining I/O performance headroom

A system and method for determining I/O performance headroom that accounts for a real-world workload is provided. In some embodiments, a computing device is provided that is operable to identify a data transaction received by a storage system and directed to a storage device. The computing system identifies an attribute of the data transaction relating to a performance cost of the data transaction and queries a performance profile to determine a benchmark performance level for the storage device. The computing system determines a benchmark performance level for the storage system based on the benchmark performance level for the storage device and compares a metric of the performance of the data transaction with the storage system benchmark performance level to determine remaining headroom of the storage system.

Dynamic transitioning of protection information in array systems

A system, method, and computer program product is described for providing dynamic enabling and/or disabling of protection information (PI) in array systems during operation. A storage system receives a request to transition a volume from PI disabled to PI enabled during regular operation. The storage system synchronizes and purges the cache associated with the target volume. The storage system initiates an immediate availability format (IAF-PI) process to initialize PI for the associated data blocks of the volume's storage devices. The storage system continues receiving I/O requests as the IAF-PI process sweeps through the storage devices. The storage system inserts and checks PI for the write data as it is written to the storage devices. The storage system inserts PI for requested data above the IAF-PI boundary and checks PI for requested data below the IAF-PI boundary. The transition remains an online process that avoids downtime.