Ausgleich nachlassender Funktionsfähigkeit von Halbleiterdatenträgern auf der Grundlage der von einer RAID-Steuereinheit empfangenen Daten- und Paritätsnutzungsinformationen

Verfahren, das folgende Schritte umfasst:Konfigurieren einer Vielzahl von Halbleiterdatenträgern (solid state disks) als redundante Anordnung unabhängiger Datenträger/Laufwerke (RAID), wobei in der Vielzahl von Halbleiterdatenträgern eine Vielzahl von Blöcken gespeichert sind und wobei die geschätzte Lebenserwartung von Speicherbereichen der Vielzahl von Halbleiterdatenträgern, die mindestens einigen der Vielzahl von Blöcken entsprechen, unterschiedlich hoch ist;Einbeziehen einer Anzeige durch eine Steuereinheit in Datenstrukturen, die einem in den Speicherbereichen der Vielzahl von Halbleiterdatenträgern zu speichernden Block zugehörig sind, dass der Block der RAID entsprechende Paritätsinformationen enthält, wobei die Paritätsinformationen Informationen umfassen, die einem Fehlerkorrekturmechanismus zum Schutz vor Datenträgerausfall entsprechen; undSenden der Datenstrukturen durch die Steuereinheit an die Vielzahl von Halbleiterdatenträgern,wobei die Vielzahl von Halbleiterdatenträgern ...

Prioritizing data reconstruction in distributed storage systems

Generating molecular encoding information for data storage

A method begins by a processing module of one or more processing modules of one or more computing devices generating a number for each encoded data slice of a set of encoded data slices based on the encoded data slice, identifying a gene based on the number to produce an identified gene, and creating a linking identifier that links the encoded data slice to the identified gene, where, for the set of encoded data slices, a set of identified genes and a set of linking identifiers are created. The method continues with the processing module generating molecular encoding information from the set of identified genes and the set of linking identifiers, where the molecular encoding information is used to create a molecular storage structure for each identified gene of the set of identified genes yielding a set of molecular storage structures.

Optimized reconstruction and copyback methodology for a disconnected drive in the presence of a global hot spare disk

Autonomic parity exchange in data storage systems

A computer-implemented method is provided for increasing the failure tolerance of an array of storage elements in a storage system. The computer-implemented method includes configuring an array to include a plurality of storage elements in n > 1 sets of storage elements. The computer-implemented method also includes configuring an erasure-correcting code such that at least one column of the storage elements of the array stores row parity information, and at least one row of the storage elements of the array stores column parity information. Still yet, the computer-implemented method includes, subsequent to a failure of one of the storage elements storing data, selecting a recipient storage element from the array, and rebuilding at least a portion of the data onto the recipient storage element by performing a parity exchange operation that retains a failure tolerance of the set of storage elements containing the failed storage element.

SECURE DATA PARSER METHOD AND SYSTEM

... ² A secure data parser is provided that may be integrated into any ²suitable system for securely storing and communicating data. The secure data ²parser parses data and then splits the data into multiple portions that are ²stored or ²communicated distinctly. Encryption of the original data, the portions of ²data, or both ²may be employed for additional security. The secure data parser may be used to ²²protect data in motion by splitting original data into portions of data, that ²may be ²communicated using multiple communications paths.² ...

Testing device

The invention provides a testing device which comprises an RAID (Redundant Array of Independent Disk) module, a replacement module, a state recognition module, a switching module, a control module and a display module, wherein the RAID module comprises a plurality of RAID members; the replacement module comprises at least one hard disk and is used for replacing the RAID members in the RAID module; the state recognition module is used for determining the current state of the RAID module and the replacement module and sending state information to the control module; the switching module is used for realizing the switching between the RAID members in the RAID module and the replacement module; the control module is used for controlling other modules and judging the test result; and the display module is used for displaying the overall test result and the testing time.

Method and device for selection of data arrays

Provided are a method, system, and article of manufacture, wherein a plurality of data arrays coupled to a storage controller is maintained. Data arrays are selected from the plurality of data arrays based on predetermined selection rules. Data is stored redundantly in the selected data arrays by writing the data to the selected data arrays.

Using file system information in raid data reconstruction and migration

MEMORY SYSTEM WITH MULTIPLE STRIPING OF RAID GROUPS AND METHOD FOR PERFORMING THE SAME

OPTIMIZED RECONSTRUCTION AND COPYBACK METHODOLOGY FOR A FAILED DRIVE IN THE PRESENCE OF A GLOBAL HOT SPARE DISK

The present invention is a system for optimizing the reconstruction and copyback of data contained on a failed disk in a multi-disk mass storage system. A system in accordance with the present invention may comprise the following: a processing unit requiring mass-storage; one or more disks configured as a RAID system; an associated global hot spare disk; and interconnections linking the processing unit, the RAID and the global hot spare disk. In a further aspect of the present invention, a method for the reconstruction and copyback of a failed disk volume utilizing a global hot spare disk is disclosed. The method includes: detecting the failure of a RAID component disk; reconstructing a portion of the data contained on the failed RAID component disk to a global hot spare disk; replacing the failed RAI component disk; reconstructing any data on the failed RAID disk not already reconstructed to the global hot spare disk to the replacement disk; and copying any reconstructed data from the ...

METHOD AND APPARATUS FOR DISTRIBUTED STORAGE INTEGRITY PROCESSING

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

DATA RECONSTRUCTION METHOD AND DATA STORAGE SYSTEM

An object of this invention is to restrict the degradation of performance in data reconstruction and to eliminate the necessity for additional redundant data storages. If one of a plurality of data storages in a system is out of order, the data in the defective storage are reconstructed on the basis of data in the other storages. The data reconstruction method comprising a step of reading out data from data storages, a step of reading out parity data from a data storage storing the parity data, a step of reconstructing data stored in the defective storage by executing a logical operation between the data and the parity data, and a step of replacing the parity data stored in the storage by reconstructed data.

Flexible redundant array of independent disks (RAID) computation device

A method for calculating a plurality (M) of redundancy blocks for multiple (N) data blocks of a plurality (D) of words each, the method comprises: receiving the number (M) of redundancy blocks by a calculator that comprises multiple (R) calculation units; configuring the calculator according to M and R; concurrently calculating, if M equals R, by the multiple (R) calculation units of the calculator, R sets of parity vectors, each set includes a plurality (D) of parity vectors; and calculating the plurality (M) of the redundancy blocks based on the R sets of parity vectors.

Dispersed data storage in a VPN group of devices

A method begins when at least a consensus threshold number of devices are active in a virtual private network (VPN) group of devices. The method continues by encoding group specific data based on a consensus threshold number of devices in the VPN group of devices and a number of devices in the VPN group of devices to produce one or more sets of encoded data slices. The method continues by sending the one or more sets of encoded data slices to at least one of: the consensus threshold number of devices or a dispersed storage network (DSN) memory for storage therein. When the at least the consensus threshold number of devices are not active in the VPN group of devices, each of the devices in the VPN group of devices are prohibited from creating the group specific data or accessing previously created group specific data.

OPTIMISTIC CHECKED WRITES

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and processing circuitry operably coupled to the interface and to the memory. The processing circuitry is configured to execute the operational instructions to perform various operations and functions. The computing device dispersed error encodes a data segment to produce a set of encoded data slices (EDSs) and selects a subset of a set of storage units (SUs). Then, the computing device transmits list slice requests to the subset of the set of SUs and receive list slice responses from at least some of them. The computing device determines a highest current revision level based on list slice responses and select a revision level for the set of EDSs. The computing device generates a set of checked revision slice requests and transmit them to the set of SUs.

BUNDLED WRITES IN A DISTRIBUTED STORAGE SYSTEM

A method for execution by a dispersed storage network (DSN). The method begins by disperse storage error encoding a data object for storage in a set of storage units with mapping to a unique storage unit. The method continues by selecting a storage unit for temporary exclusion, producing a bundled encoded data slice, updating the slice mapping, selecting a subset of storage units of the set of storage units for storage of the plurality of bundled encoded data slices in accordance with the updated slice mapping and issuing a write slice request that includes a group of encoded data slices in accordance with the updated slice mapping. The method continues by determining to conclude the temporary exclusion of the selected storage unit and facilitating migration of the plurality of bundled encoded data slices from the subset of storage units to the selected storage unit.

Disk array and method for reading/writing data from/into disk unit

When a bus is used as a data communication channel, data within a disk unit cannot be reproduced or copied into a spare disk while a control unit is making read/write processing based on a request from a host computer, or vice versa. Thus, a loop is constructed by a fiber channel capable of time division multiplex function, and the processing between the disk unit and the spare is performed not through the control unit so that the data within the disk can be copied into the spare while the processing between the host computer and the disk unit is being performed, or both processing operations can be executed in parallel.

MESSAGE PERSISTENCE IN A ZONED SYSTEM

A plurality of storage controllers configured to initiate an action based on redundant copies of metadata, such that a source authority of one of the plurality of storage controllers receives a message, records the message redundantly throughout the plurality of storage controllers, and delivers the message to a destination authority of a further one of the storage controllers responsive to achieving a level of redundancy for the redundant copies of the metadata regarding the message is provided, wherein at least one of the plurality of storage controllers comprises a zoned storage drive. 1. A storage system , comprising:a plurality of storage controllers configured to initiate an action based on redundant copies of metadata, such that a source authority of one of the plurality of storage controllers receives a message, records the message redundantly throughout the plurality of storage controllers, and delivers the message to a destination authority of a further one of the storage controllers responsive to achieving a level of redundancy for the redundant copies of the metadata regarding the message, and wherein at least one of the plurality of storage controllers comprises a zoned storage drive.2. The storage system of claim 1 , wherein the storage controllers are configured to provide a limit on a number of open zones within the zoned storage drive.3. The storage system of claim 1 , wherein the plurality of storage controllers are distributed across multiple chassis.4. The storage system of claim 1 , wherein the storage controllers are configured to reset an open zone of the zoned storage drive to allow an erase block associated with the open zone to be reused.5. The storage system of claim 1 , further comprising:each of the plurality of storage controllers configured to send a copy of the metadata to others of the plurality of storage controllers.6. The storage system of claim 1 , further comprising:each of the plurality of storage controllers having decision ...

Virtualized data storage vaults on a dispersed data storage network

A system, method, and apparatus for implementing a plurality of dispersed data storage networks using a set of slice servers are disclosed. A plurality of information records are maintained, with each information record corresponding to a dispersed data storage network. The information record maintains what slice servers are used to implement the dispersed data storage network, as well as other information needed to administer a DDSN, such as the information dispersal algorithm used, how data is stored, and whether data is compressed or encrypted.

Memory reallocation during raid rebuild

Apparatus and method for managing data in a multi-device storage system, such as a RAID (redundant array of independent discs) system. Distributed data sets are stored across a plurality of storage devices. A selected storage device is replaced with a new storage device responsive to an anomalous event. A rebuild operation is performed to reconstruct data from the selected storage device to the new storage device. The rebuild process includes accessing a list of distributed data sets in a local memory. For each distributed data set in the list identified as constituting valid data, read commands are issued to the remaining storage devices and a write command is issued to the new storage device. For each distributed data set in the list identified as constituting unused data, a data clear command is issued to each of the remaining storage devices and to the new storage device.

Analyzing found data in a distributed storage and task network

A method begins by a dispersed storage (DS) processing module establishing data identifying criteria for searching data on a network, establishing data analyzing criteria for analyzing found data of the data on the network, and establishing distributed computing criteria. The method continues with the DS processing module distributing the data identifying criteria and the data analyzing criteria to a set of distributed storage and task (DST) units. The method continues with the DS processing module receiving a set of network data partial resultants from the set of DST units, wherein the set of DST units generates the set of network data partial results in accordance with the data identifying criteria to produce found data and analyzing the found data in accordance with the data analyzing criteria. The method continues with the DS processing module processing the set of network data partial resultants to produce a network data resultant.

Data migration between a raid memory and a dispersed storage network memory

A method begins by a processing module determining whether to convert data between a redundant array of independent disks (RAID) format and a dispersed storage network (DSN) format. The method continues with the processing module retrieving the data from a RAID memory to produce retrieved RAID data when the data is to be converted from the RAID format to the DSN format. The method continues with the processing module converting stripe-block data of the retrieved RAID data into a plurality of sets of encoded data slices and outputting the plurality of sets of encoded data slices to at least one of the RAID memory and a DSN memory for storage therein.

HANDLING ZERO FAULT TOLERANCE EVENTS IN MACHINES WHERE FAILURE LIKELY RESULTS IN UNACCEPTABLE LOSS

Provided are a computer program product, system, and method for managing I/O requests to a storage array of storage devices in a machine having a processor node and device adaptor. In response to initiating a rebuild of data in the storage array, the device adaptor determines whether a remaining fault tolerance at the storage array comprises a non-zero fault tolerance that permits at least one further storage device to fail and still allow recovery of data stored in the storage array. In response to determining that the remaining fault tolerance is a zero fault tolerance that does not permit at least one storage device to fail and allow recovery of data, the device adaptor sends a message to the processor node to cause the processor node to initiate an emergency protocol to terminate a mission critical operation when the processor node is performing the mission critical operation.

METHOD, APPARATUS AND COMPUTER PROGRAM PRODUCT FOR MANAGING CACHE

Techniques manage a cache. Such techniques involve creating a primary cache by a cache management module in a storage system. Such techniques further involve: in response to the primary cache being created, sending a first request to a hardware management module to obtain first information about a first virtual disk. Such techniques further involve: in response to receiving the first information from the hardware management module, creating a secondary cache using the first virtual disk. Such techniques further involve: in response to an available capacity of the primary cache being below a predetermined threshold, flushing at least one cache page in the primary cache to the secondary cache. In certain techniques, it is possible to use spare extents in the disk array to create the secondary cache to increase a total capacity of the cache in the system, thereby improving the access performance of the system.

Storage Systems with Peer Data Recovery

Example peer storage systems, storage devices, and methods provide peer data recovery across a peer communication channel. Peer storage devices establish peer communication channels that communicate data among the peer storage devices. A storage device may identify storage media segments from their storage medium for recovery of failed data units. A peer storage device may be identified that contains recovery data for the failed data units. The recovery data may be received over the peer communication channel and the storage media segments may be recovered using the recovery data.

Using parity data for concurrent data authentication, correction, compression, and encryption

A system for software error-correcting code (ECC) protection or compression of original data using ECC data in a first memory is provided. The system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The software ECC protection or compression includes: a data matrix for holding the original data in the first memory; a check matrix for holding the ECC data in the first memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the ECC data; and a thread for executing on the processing core. The thread includes a Galois Field multiplier for multiplying entries of the data matrix by an entry of the encoding matrix, and a sequencer for ordering operations using the Galois Field multiplier to generate the ECC data.

MULTI OPTION REBUILDING IN A DISPERSED STORAGE NETWORK

A method includes identifying an encoded slice for rebuilding. The method further includes determining whether the set of encoded slices is stored in an encrypted section of a vault or within an unencrypted section of the vault. The method further includes, when the set of encoded slices is stored in the unencrypted section of the vault, determining whether the set of storage units have viewing rights. The method further includes, when the set of storage units does not have the viewing rights, enabling a restricted rebuilding process to rebuild the encoded slice. The method further includes, when the set or storage units does have the viewing rights, enable an unrestricted rebuilding process to rebuild the encoded slice. 1. A computing device comprising:an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);memory that stores operational instructions; and identify, via the interface, an encoded data slice (EDS) to be rebuilt, wherein a data object is segmented into a plurality of data segments, wherein a data segment is dispersed storage error encoded to produce a set of encoded data slices (EDSs), which includes the EDS, wherein a set of storage units (SUs) of the DSN stores the set of EDSs, and wherein a first storage unit (SU) of the set of SUs is assigned to store the EDS;', 'determine whether the set of EDSs is stored in an encrypted section of a vault or within an unencrypted section of the vault;', 'based on a first determination that the set of EDSs is stored in the encrypted section of the vault, enable an unrestricted rebuilding process to rebuild the EDS;, 'processing circuitry operably coupled to the interface and to the memory, wherein the processing circuitry is configured to execute the operational instructions tobased on a second determination that the set of EDSs is stored in the unencrypted section of the vault, determine whether the set of SUs have viewing rights; andbased on a third ...

Methods and systems for using a write cache in a storage system

Methods and systems for storing data at a storage device of a storage system are provided. The data is first temporarily stored at a first write cache and an input/output request for a persistence storage device used as a second write cache is generated, when an I/O request size including the received data has reached a threshold value. The data from the first cache is transferred to the persistence storage device and a recovery control block with a location of the data stored at the persistence storage device is updated. An entry is added to a linked list that is used to track valid data stored at the persistence storage device and then the data is transferred from the persistence storage device to the storage device of the storage system.

REBUILDING A STORAGE ARRAY

The present invention discloses a method and apparatus for rebuilding a storage array. The method comprises: replacing, in response to a failing storage device occurs in the storage array, the failing storage device with a first spare storage device; performing component rebuild by using other storage devices in the storage array than the first spare storage device, so as to restore within the first spare storage device data in the failing storage device; performing, in parallel with the component rebuild, smart rebuild by using a second spare storage device, so as to copy data in the failing storage device to the second spare storage device; replacing the first spare storage device with the second spare storage device in the storage array upon completion of the smart rebuild.

METHOD AND SYSTEM FOR MANAGING STORAGE SYSTEM

A method and system for managing a storage system, the method comprising: creating a storage system including a first number of stripes, wherein each of the first number of stripes includes a second number of extents, and each of the second number of extents is distributed over a different storage device in a group of storage devices; with respect to a stripe among the first number of stripes, generating stripe information descriptive of a mapping relation between the second number of extents included in the stripe and the different storage devices in the group of storage devices; and with respect to a storage device where an extent among the second number of extents is located, building extent information about a mapping relation between an extent in the storage device and the stripe. In one embodiment of the present invention, there is provided a corresponding device and system.

Expanding information dispersal algorithm width without rebuilding through imposter slices

A method and system for expanding a set of storage units. Expanding a set of storage units includes facilitating storage of a set of encoded data slices in a set of storage units within a first time frame. When detecting expansion of the set of storage units with new storage units within a second timeframe, primary storage units of the expanded set of storage units based are identified based on a write threshold value associated with the expanded set of storage units. For each new primary storage unit, storage of an imposter encoded data slice sourced from another storage unit that is not a primary storage unit but holds an encoded data slice of the set of encoded data slices is facilitated. The other storage unit then deletes the imposter encoded data slice.

DISTRIBUTED REDUNDANT STORAGE SYSTEM

A device implementing a system for object placement includes at least one processor configured to identify an object stored on two of a plurality of storage devices, and to determine a pairing value for each pair of the plurality of storage devices, the pairing value corresponding to a number of stored objects common to the pair. The at least one processor is further configured to select, based on the determined pairing values, two storage devices having a lowest pairing value relative to the determined pairing values, and to move the object to each of the two selected storage devices.

MULTI OPTION REBUILDING IN A DISPERSED STORAGE NETWORK

A method includes identifying an encoded slice for rebuilding. The method further includes determining whether the set of encoded slices is stored in an encrypted section of a vault or within an unencrypted section of the vault. The method further includes, when the set of encoded slices is stored in the unencrypted section of the vault, determining whether the set of storage units have viewing rights. The method further includes, when the set of storage units does not have the viewing rights, enabling a restricted rebuilding process to rebuild the encoded slice. The method further includes, when the set or storage units does have the viewing rights, enable an unrestricted rebuilding process to rebuild the encoded slice.

REPORTING INCIDENTS OF DATA LOSS IN RAID ARRAYS

A method for reporting incidents of data loss in a storage environment comprising redundant arrays of independent disks (RAIDs) is disclosed. In one embodiment, such a method monitors storage drive failures in a storage environment. For a storage drive failure detected in the storage environment, the method reports the RAID type in which the storage drive failure occurred and whether data loss occurred in the RAID as a result of the storage drive failure. In certain embodiments, the method reports whether the data loss could have been prevented had the RAID type been converted to a more robust RAID type. In other or the same embodiments, the method reports whether the data loss was prevented by the RAID type. A corresponding system and computer program product are also disclosed.

Managing memory utilization in a distributed storage and task network

A method begins by a distributed storage (DS) processing module retrieving a data slice from a local memory and performing a partial task on the data slice. When the performing of the partial task is complete, the method continues with the DS processing module determining whether at least a retrieval number of slices of a set of slices of a data segment that includes the data slice is available from a set of DST execution units. When the at least a retrieval number of slices is available, the method continues with the DS processing module deleting the data slice from the local memory. When the at least a retrieval number of slices of the set of slices is not available, the method continues with the DS processing module determining whether execution of a task on the data segment is complete and deleting the data slice when the execution is complete.

Storing indexed data to a dispersed storage network

A method begins by a dispersed storage (DS) processing module storing a data portion in a dispersed storage network (DSN), where the data portion is associated with a set of data index keys, a version index key, and with a set of portion index keys. The method continues with the DS processing module traversing, based on the set of data index keys, a plurality of index structures to determine whether an object retrieval structure exists for the data portion. When the object retrieval structure exists, the method continues with the DS processing module updating the object retrieval structure by determining whether a version record exists that corresponds to the version index key and when the version record exists, creating, within the version record, a new data portion record for the data portion to include an identifier of the data portion and the set of portion index keys.

PARTIAL REBUILDING TECHNIQUES IN A DISPERSED STORAGE UNIT

Methods for use in a dispersed storage network (DSN) to rebuild data. In various examples, at least one data segment of a data object is dispersed storage error encoded to produce a set of encoded data slices that includes an information dispersal algorithm (IDA) width number of encoded data slices, wherein the IDA width number is at least twice the number of storage units of a set of storage units utilized to store the data object. At least two encoded data slices are stored in each of the storage units. In response to detecting a storage error indicating an error slice, a partial threshold number of partial encoded data slices (generated by performing a partial encoding function on stored encoded data slices) are generated. The partial threshold number of partial encoded data slice responses are combined to produce a rebuilt encoded data slice corresponding to the error slice.

RECOVERY OF AN INFECTED AND QUARANTILED FILE IN A PRIMARY STORAGE CONTROLLER FROM A SECONDARY STORAGE CONTROLLER

A primary storage controller determines that a quarantined area of the primary storage controller cannot be repaired, wherein the quarantined area is infected with a virus. A query is sent to a secondary storage controller to determine whether the secondary storage controller has data that is free of virus in an area of the secondary storage controller corresponding to the quarantined area of the primary storage controller. In response to receiving a notification that the secondary storage controller has data that is free of virus, the primary storage controller is repaired to remove the virus.

IDENTIFYING A FAULT DOMAIN FOR A DELTA COMPONENT OF A DISTRIBUTED DATA OBJECT

The disclosure herein describes placing a delta component of a base component in a target fault domain. A delta component associated with a base component is generated. The generation includes selecting a first fault domain as a target fault domain for the delta component based on the first fault domain including a witness component associated with the distributed data object of the base component. Otherwise, the generation includes selecting a second fault domain as the target fault domain based on the second fault domain including at least one data component that includes a different address space than the base component. Otherwise, the generation includes selecting a third fault domain as the target fault domain based on the third fault domain being unused. Then, the delta component is placed on the target fault domain, whereby data durability of the distributed data object is enhanced, and available fault domains are preserved.

CO-LOCATE OBJECTS REQUEST

A method begins by a dispersed storage (DS) processing module receiving a data object co-locate write request. The method continues with the DS processing module obtaining a plurality of sets of encoded data slices for a data object to co-locate. The method continues with the DS processing module generating a plurality of sets of slice names for the data object to co-locate based on another plurality of sets of slice names associated with a data object to be co-located with. The method continues with the DS processing module storing the plurality of sets of encoded data slices in the DSN using the generated plurality of sets of slice names for the data object co-locate.

Wirelessly communicating a data file

A dispersed storage (DS) processing module sends a plurality of undecodeable portions of a plurality of data files via a public wireless communication network to one or more targeted devices of a private wireless communication network. The DS processing module continues processing by sending data content indicators regarding the plurality of data files and in response to a selection of a data file of the plurality of data files based on a corresponding one of the data content indicators, sending, via the private wireless communication network, one or more encoded data slices of each of one or more sets of encoded data slices of the data file such that, for each of the one or more sets of encoded data slices, the one or more targeted devices obtains at least a decode threshold number of encoded data slices to decode the data file.

Data reduction techniques in a flash-based key/value cluster storage

In one aspect, a method includes splitting empty RAID stripes into sub-stripes and storing pages into the sub-stripes based on a compressibility score. In another aspect, a method includes reading pages from 1-stripes, storing compressed data in a temporary location, reading multiple stripes, determining compressibility score for each stripe and filling stripes based on the compressibility score. In a further aspect, a method includes scanning a dirty queue in a system cache, compressing pages ready for destaging, combining compressed pages in to one aggregated page, writing one aggregated page to one stripe and storing pages with same compressibility score in a stripe.

INTELLIGENT SLICE CACHING

Systems and methods for intelligent slice caching in a dispersed storage network. The methods include determining a minimum slice access rate for encoded data slices to be stored, determining a least access rate of a least accessed encoded data slice stored, determining an estimated access rate for an encoded data slice and determining whether to store the encoded data slice in small fast memory as a cached encoded data slice based on the minimum slice access rate, the least access rate, and the estimated access rate. The method further includes facilitating storage of the encoded data slice in small fast memory. The method may also include updating the minimum slice access rate and transferring an encoded data slice stored in small fast memory to large slow memory when an actual access rate is less than the minimum slice access rate or is less than the least access rate.

Estimating a number of entries in a dispersed hierarchical index

Methods and systems for estimating a number of entries in a dispersed hierarchical index. The method and systems involve determining a number of random walks N to perform on the dispersed hierarchical index, conducting N walkthroughs based on the number of walkthroughs, determining a number of walk entries for each of the N random walks and averaging the number of walk entries for each of the N random walks to produce an estimated total number of entries for the dispersed hierarchical index. The determining may be based on one or more of a number of levels, a desired confidence interval, a predetermination, and interpretation of system registry information, and an interpretation of a request. Each random walk starts at a root node and ends at a leaf node through L levels of the dispersed hierarchical index.

ADAPTIVE DISPERSED STORAGE NETWORK (DSN) AND SYSTEM

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and to the memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. When additional unique pillar combinations of at least read threshold number of encoded data slices (EDSs) supported by EDSs may be needed, the computing device dispersed error encodes each data segment in accordance to generate additional pluralities of EDSs and distributedly stores the additional pluralities of EDSs associated respectively with the data object across the plurality of SUs within the DSN to support the additional unique pillar combinations of the at least read threshold number of EDSs.

Verifying and migrating data slices in a storage network

A method for execution by a storage network includes receiving a request pertaining to a data object. Metadata associated with the data object is determined and used to identify data segments associated with the data object and a set of storage units associated with the data segments. Based on a set of query requests, a response is received from a storage unit from the set of storage units. When the response indicates a storage environment that is unfavorable as compared to predetermined performance metrics, the storage network facilitates migration of encoded data slices associated with the storage unit to another storage unit.

Encoding and rebuilding the data content of up to two unavailable DASDs in a DASD array

The data contents of up to two concurrently failed or erased DASDs can be reconstituted where the data is distributed across M DASDs as an (M-1)*M block array and where (1) the (M-1)st DASD contains the simple parity taken over each of the array diagonals in diagonal major order in the same mode (odd/even) as that exhibited by the major diagonal of the array and (2) where the M-th DASD contains the simple even parity over each of the rows in row major order. Relatedly, short write updates require fewer operations for data blocks located off the major data array diagonal. ...

On-line Wiederherstellung eines fehlerhaften redundanten Gruppensystems

Reducing rebuild time in a computing storage environment

Embodiments for reducing rebuild time in a computing storage environment in by a processor. One or more disk drive failures in a Redundant Array of Independent Disks (RAID)-6 may be rebuilt by holding at least three parity strips per stripe while using one or more of the at least three parity strips according the one or more disk drive failures.

Autonomic parity exchange in data storage systems

Video storage

Autonomic parity exchange in data storage systems

Servicing data storage devices in a data storage array

Systems and methods for replacing and testing a data storage device are disclosed. In disclosed embodiments, a system including a data storage array (DSA) including a plurality of data storage devices (DSDs) in an enclosure. The system further includes an I/O server coupling the DSA to a client node and configured to provide data access between the client node and the DSA. The system further includes a management server coupled to the DSA, configured to detect a failed DSD in the DSA, detect a replacement DSD in the enclosure that replaces the failed DSD, and add the replacement DSD to a logical path of the DSA. The management server is further configured to display an indication of a state of the DSA based on the comparing.

PERFORMING A PREEMPTIVE RECONSTRUCT OF A FAULT-TOLERANT RAID ARRAY

Rebuilding drive data

Fast data recovery from HDD failure

The invention relates to a fast data recovery from HDD failure. The storage system comprises a first storage device having a first plurality of hard disk drives and a first controller. The first controller stores data in the first plurality of hard disk drives by stripes. Each stripe includes M data and N parity data allocated to M+N hard disk drives of the first plurality of hard disk drives. A first hard disk drive includes data or parity data of both a first stripe of the stripes and a second stripe of the stripes, while a second hard disk drive includes data or parity data of only one of the first stripe or the second stripe. During data recovery involving failure of one of the first plurality of hard disk drives, the data in the failed hard disk drive is recovered for each stripe by calculation using data and parity data in other hard disk drives for each stripe.

SYSTEM AND METHOD FOR RECOVERING DATA AND COMPUTER READABLE MEDIUM THEREOF CAPABLE OF RESTORING DATA STORED IN STORING DEVICES

PURPOSE: A system and method for recovering data and computer readable medium thereof are provided to optimized redundancy recovery in a distributed layout environment by using a recovery order. CONSTITUTION: A priority module(210) is combined to storage devices(102). If the critical data failure occurs, the priority order module determines a recovery order of a first and second data part. A recovery module(212) restores the first and second data part based on the recovery order and is combined to the priority order module and the storing devices. The first and second data part is respectively connected to the first and second data word. The priority order module determines a recovery order of the first and second data part based on data failure which is connected to the first and second word. COPYRIGHT KIPO 2012 ...

METHOD AND A DEVICE FOR STORING DATA TO AN HDD(HARD DISK DRIVE) BY DISTRIBUTING THE DATA

PURPOSE: A method and a device for storing data are provided to realize highly-convenient usage patterns of various types of data storage devices by distributing the data in an HDD. CONSTITUTION: The first divider divides the data to be stored into a plurality of divided data. The first parity data generator generates parity data corresponding to each divided data. The first storing part distributes/stores the divided data and parity data to the plurality of storage units. A restorer restores the data stored before one storage unit become inaccessible state based on the data stored in the remaining storage units when one storage unit become inaccessible state. The second divider divides the restored data into the plurality of divided data. The second parity data generator generates the parity data corresponding to each divided restored data. The second storing part distributes/stores the divided restored data and the parity data to the plurality of accessible storage units. © KIPO 2007 ...

Method and apparatus for performing dynamic recovery management regarding redundant array of independent disks

A method and apparatus for performing dynamic recovery management regarding a RAID are provided. The method includes: writing a first set of protected data into a first protected access unit of multiple protected access units of the RAID, and recording a first set of management information corresponding to the first set of protected data, for data recovery of the first set of protected data; and when any storage device of multiple storage devices of the RAID malfunctions, writing a second set of protected data into a second protected access unit of the protected access units, and recording a second set of management information corresponding to the second set of protected data, for data recovery of the second set of protected data. Any set of the first set of protected data and the second set of protected data includes data and multiple parity-check codes.

PARITY BASED METHOD FOR RECONSTRUCTING A DATA SUBSET WHEREIN SAID SUBSET BELONGS TO MULTIPLE PARITY SETS

Method and apparatus for protecting data comprising dividing data into a plurality of data subsets. Generating parity data for each data subset and another data subset of the plurality of data subsets. Generating parity data for each data subset and a second other data subset of the plurality of data subsets. Method and apparatus for recovering data comprising retrieving one or more data subsets. Retrieving two or more different parity data sets generated for one data subset. Regenerating any missing data subsets using the retrieved one or more data subset and two or more retrieved parity data sets. Combining the one or more retrieved data subset and regenerated missing data subsets to form the recovered data.

ARCHIVAL STORAGE AND RETRIEVAL SYSTEM

A highly reliable data archival and retrieval method that enables fine grained control over data availability is implemented across a Quality of Service driven archival system, configured to fragment the data into data and parity chunks for storing onto the storage node. The technique employed by the archival system enables files to be read without having need to access any metadata, thereby tolerating complete loss of such metadata. Further, the Quality of Service driven system architecture improves upon the system performance and throughput by means of a storage node regeneration process which ensures balanced load on participating storage node during various storage, retrieval and regeneration operations.

Method of handling unreadable blocks during rebuilding of a RAID device

Disclosed is a method for continuing a rebuilding process of a RAID system by flagging a block of data as being bad when a media error or other error occurs that prohibits the reconstruction of data. The block of data may be flagged by writing a bad error correction code to the block of data, by keeping a log of bad blocks of data, or by otherwise indicating that the block of data is known bad.

Parity based method for reconstructing a data subset wherein said subset belongs to multiple parity sets

Method and apparatus for protecting data comprising dividing data into a plurality of data subsets. Generating parity data for each data subset and another data subset of the plurality of data subsets. Generating parity data for each data subset and a second other data subset of the plurality of data subsets. Method and apparatus for recovering data comprising retrieving one or more data subsets. Retrieving two or more different parity data sets generated for one data subset. Regenerating any missing data subsets using the retrieved one or more data subset and two or more retrieved parity data sets. Combining the one or more retrieved data subset and regenerated missing data subsets to form the recovered data.

Raid 10 reads optimized for solid state drives

A mechanism is provided in a data processing system. The mechanism determines a maximum queue depth of a queue for each solid state drive in a plurality of solid state drives. A given data block is mirrored between a group of solid state drives within the plurality of solid state drives. The mechanism tracks outstanding input/output operations in a queue for each of the plurality of solid state drives. For a given read operation to read the given data block, the mechanism identifies a solid state drive within the group of solid state drives based on a number of empty slots in the queue of each solid state drive within the group of solid state drives.

IP multicast message transmission for event notifications

A method for a dispersed storage network (DSN) begins when storing data associated with a vault, by identifying a multicast IP address associated with the vault. The method continues by generating and sending a multicast IP message to a plurality of storage units of the DSN and extracting, by each storage unit receiving the multicast IP message, the multicast IP address. The method continues when the extracted IP address compares favorably to a subscribed IP address, by storing, by the storage unit, the one or more encoded data slices for storage and the corresponding slice name for each encoded data slice. The method continues by each storage unit issuing a status multicast IP message. The method continues when an extracted IP address of a plurality of received status multicast IP messages compares favorably to a subscribed IP address, by executing an associated task.

File system driven raid rebuild technique

Embodiments described herein are directed to a file system driven RAID rebuild technique. A layered file system may organize storage of data as segments spanning one or more sets of storage devices, such as solid state drives (SSDs), of a storage array, wherein each set of SSDs may form a RAID group configured to provide data redundancy for a segment. The file system may then drive (i.e., initiate) rebuild of a RAID configuration of the SSDs on a segment-by-segment basis in response to cleaning of the segment (i.e., segment cleaning). Each segment may include one or more RAID stripes that provide a level of data redundancy (e.g., single parity RAID 5 or double parity RAID 6) as well as RAID organization (i.e., distribution of data and parity) for the segment. Notably, the level of data redundancy and RAID organization may differ among the segments of the array.

Accelerated erasure coding system and method

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.

FASTER REBUILDING OF 2-DISK FAILURE IN RAID BY EFFICIENT, MODULAR, DIAGONAL, CONCURRENT PARITY

In response to receiving data to be stored in a storage system that includes a plurality of N drives, a method may include storing a first quantity of the data in a first N portions across a first half of the N drives; storing a second quantity of the data in a second N portions across a second half of the N drives; determining first parities to be stored on a first parity drive; determining second parities and storing them on a second parity drive; and calculating third parities based on both the first quantity of the data and the second quantity of the data, wherein the third parities are calculated in a diagonal fashion based on distinct elements of the first N portions and distinct elements of the second N portions, and storing the third parities on a third parity drive. 1. A method comprising: storing a first quantity of the data in a first N portions across a first half of the N drives;', 'storing a second quantity of the data in a second N portions across a second half of the N drives;', 'for each of the first N portions of the first quantity of the data, determining a first parity, and storing each first parity on a first parity drive;', 'for each of the second N portions of the second quantity of the data, determining a second parity, and storing each second parity on a second parity drive; and', 'calculating third parities based on both the first quantity of the data and the second quantity of the data, wherein the third parities are calculated in a diagonal fashion based on distinct elements of the first N portions and distinct elements of the second N portions, and storing the third parities on a third parity drive., 'in response to receiving data to be stored in a storage system that includes a plurality of N drives, performing data storing operations comprising2. The method of claim 1 , wherein the first claim 1 , second claim 1 , and third parities are calculated based on an XOR operation.3. The method of claim 1 , wherein at least one of the first ...

Method, system, apparatus, and computer-readable medium for improving disk array performance

A method, system, apparatus, and computer-readable medium are provided for improving storage in a disk array are provided. According to aspects of the invention, a redundant disk array is combined with a mechanism for thin provisioning of the array. Thin provisioning refers to a process of allocating physical capacity to logical volumes on an as-needed basis. Data structures containing a mapping between the logical location of stored data and its actual location on a physical device are maintained. Through the use of the thin provisioning mechanism, physical storage space can be allocated sequentially, regardless of the order of logical writes. In this manner, the data stored on the array grows in a linear manner. The data structures maintained by the thin provisioning mechanism can be used to identify the portions of a device or an array that have been previously written. This information allows redundant arrays, such as RAID arrays, to perform initialization, rebuild, and data migration ...

METHOD AND SYSTEM FOR A PROACTIVE HEALTH CHECK AND RECONSTRUCTION OF DATA

In general, the invention relates to a method for managing data. The method includes identifying a first chunk to be health-checked in a fault domain using storage metadata stored in the fault domain, obtaining the chunk from the fault domain, generating, after obtaining the chunk, a first chunk identifier using the first chunk, making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata, and in response to the determination: obtaining a plurality of chunks associated with the first chunk from a plurality of fault domains, wherein the fault domain is not part of the plurality of fault domains, regenerating the first chunk using the plurality of chunks to generate a new first chunk, and storing the new first chunk in the fault domain.

Information processing system, information processing system management method, and program thereof

An object of the present invention is to provide a redundant information processing system that can continue its operation without stopping as much as possible. To achieve this, the information processing system includes multiple storage nodes including processors, memories, and storage devices, as well as a network for connecting the storage nodes. The processor of at least one storage node performs the steps of: obtaining the configuration information of the information processing system; obtaining first failure information related to a first failure occurred in the information processing system; calculating the availability level of the information processing system when a second failure further occurs, based on the configuration information and the first failure information; and controlling the operation of the system based on the availability level. Then, the processor outputs a notification.

DISK ARRAY SYSTEM AND CONTROL METHOD THEREOF

A disk array system, upon detecting a failure in any data disk from among a plurality of data disks providing one or more RAID groups, conducting a correction copy to any spare disk, using one or more other data disks belonging to the same RAID group as the data disk causing the failure. When the data disk causing the failure has been replaced with a new data disk, the disk array system alters the management so that the data disk can be managed as a spare disk, and the spare disk can be managed as a data disk.

Identifying a slice name information error in a dispersed storage network

A method begins by a processing module sending list digest requests to a set of dispersed storage (DS) units. The method continues with the processing module receiving list digest responses from at least some of the set of DS units and determining whether an inconsistency exists between first and second list digest responses of the list digest responses. The method continues with the processing module requesting at least a portion of each of the slice name information lists from first and second DS units of the set of DS units and identifying a slice name information error associated with the inconsistency based on the at least a portion of each of the slices name information lists of the first and second DS units when the inconsistency exists between first and second list digest responses of the list digest responses.

Method and system for a proactive health check and reconstruction of data

In general, the invention relates to a method for managing data. The method includes identifying a first chunk to be health-checked in a fault domain using storage metadata stored in the fault domain, obtaining the chunk from the fault domain, generating, after obtaining the chunk, a first chunk identifier using the first chunk, making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata, and in response to the determination: obtaining a plurality of chunks associated with the first chunk from a plurality of fault domains, wherein the fault domain is not part of the plurality of fault domains, regenerating the first chunk using the plurality of chunks to generate a new first chunk, and storing the new first chunk in the fault domain.

REDUCING INCIDENTS OF DATA LOSS IN RAID ARRAYS OF DIFFERING RAID LEVELS

A method for reducing incidents of data loss in RAIDs of differing RAID levels is disclosed. Such a method identifies, in a data storage environment, first and second sets of RAIDs. The first set contains RAIDs providing more robust data protection (e.g., RAID-6 arrays), and the second set contains RAIDs providing less robust data protection (e.g., RAID-5 arrays). The method identifies, in the data storage environment, higher risk storage drives having a failure risk above a threshold and lower risk storage drives having a failure risk below the threshold. The method swaps higher risk storage drives from RAIDs of the second set with lower risk storage drives from RAIDs of the first set. In certain embodiments, the swapping may be performed such that no RAID of the second set includes more than a selected number of higher risk storage drives. A corresponding system and computer program product are also disclosed.

Rebuilding process for storage array

When a drive in a redundant array of independent disks (RAID) fails, for a first stripe unit stored in the failed drive, a RAID controller obtains from a bitmap a first indicator indicating whether the first stripe unit contains written data, wherein the bitmap contains a plurality of indicators corresponding to stripe units stored in the failed drive, and the first stripe unit belongs to a first stripe stored in the RAID system; if the first indicator indicates that the first stripe unit does not contain written data, write zeros into a first replacement stripe unit in a replacement drive, wherein the first replacement stripe unit replaces the first stripe unit in the first stripe. Without performing XOR operations for the stripe units indicated no data by the table, a time consumed for rebuilding can be significantly reduced, recovering the RAID array healthy sooner and reducing the impact on system bandwidth usage and host I/O performance impact during the rebuild process.

DISPERSED STORAGE WRITE PROCESS

A method includes issuing write commands to DS storage units regarding encoded data slices. A write command causes a storage unit to store an encoded data slice, but not allow access to the stored encoded data slices. The method includes receiving write acknowledgements from at least some of DS storage units to produce received write acknowledgements. The method includes, when a write threshold number of the received write acknowledgements have been received within a given time frame, issuing commit commands to the DS storage units. A commit command causes the storage unit to allow access to the stored encoded data slice. The method includes, when the write threshold number of the received write acknowledgements have not been received within the given time frame, issuing rollback commands to the DS units. A rollback command causes the storage unit to continue to not allow access to the stored encoded data slice.

DISTRIBUTED STORAGE NETWORK FOR MODIFICATION OF A DATA OBJECT

A distributed storage network generates a plurality of data segments from a data object and stores each of the plurality of data segments as a plurality of encoded data slices generated from an error encoding dispersal function. When the distributed storage network receives a modification request for the data object, it determines a size of the plurality of data segments of the data object from a segment size field and identifies one of the plurality of data segments requiring modification. The identified data segment is reconstructed from the plurality of encoded data slices and modified in accordance with the modification request.

DETECTING STORAGE ERRORS IN A DISPERSED STORAGE NETWORK

A method includes dividing a data object into data partitions. The method further includes, for each data partition: dividing the data partition into data segments; dispersed storage error encoding the data segments to produce sets of encoded data slices; storing the sets of encoded data slices in a first set of storage units; and generating a segment allocation table regarding storage information of the sets of encoded data slices. The method further includes generating a directory of segment allocation tables. The method further includes receiving an access request regarding at least a portion of the data object. The method further includes accessing the directory to identify one or more segment allocation tables containing storage information for the at least a portion of the data object. The method further includes accessing encoded data slices of the at least the portion of the data object based on the storage information.

DISASTER RECOVERY SYSTEM

Disclosed herein is a computer implemented method of performing recovery for a customer server system that has an associated backup of server system data of the customer server system, the method comprising the steps of: receiving a server recovery request at a portal for a rebuild of at least part of the customer server system; and, sending a request from the portal to a cloud-based data centre for on-demand provisioning of cloud-based server resources, wherein the request includes information on the location of at least part of the backup of the server system data to enable the deployment of a rebuild of at least part of the customer server system at the cloud-based data centre. Advantages include a user being able to easily manage disaster recovery testing as well as actual live recovery operations. The use of temporary servers in the cloud is an efficient, and inexpensive, use of resources as the servers can be rented and used only when required.

LOCALIZED DISPERSED STORAGE MEMORY SYSTEM

A method includes a processing module receiving data to store and determining error coding dispersal storage function parameters based on an error profile of one or more hard drives. The method continues with the processing module encoding at least a portion of the data in accordance with the error coding dispersal storage function parameters to produce a set of data slices. The method continues with the processing module defining addressable storage sectors within the one or more hard drives based on a number of data slices within the set of data slices to produce a set of addressable storage sectors. The method continues with the processing module storing data slices of the set of data slices in corresponding addressable storage sectors of the set of addressable storage sectors.

REBALANCING DATA STORAGE IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) execution unit includes generating location weight data that includes a plurality of location weights assigned to a plurality of memory devices of the DST execution unit. A first one of the plurality of memory devices and a second one of the plurality of memory devices are selected for reallocation based on the location weight data. The reallocation is executed by removing a data slice from the first one of the plurality of memory devices and storing the data slice in the second one of the plurality of memory devices.

RAID array systems and operations using mapping information

An apparatus may include a redundant array of independent disks (RAID) array including a plurality of solid state drives (SSDs). The apparatus may further include a RAID array controller coupled to the plurality of SSDs. The RAID array controller may be configured to determine whether one or more logical block addresses (LBAs) of a stripe of the RAID array are unmapped. The one or more LBAs may be associated with one or more SSDs of the plurality of SSDs. The RAID array controller may be configured to determine data corresponding to the stripe based on the determination of whether the one or more LBAs are unmapped. RAID operations (such has Rebuild, Exposed Mode Read, and/or Parity Resync operations) may be optimized based on the knowledge of which LBAs are mapped and unmapped.

Redundant array of inexpensive disks configuration using simple network management protocol

A method of configuring a storage device is disclosed. The method generally includes the steps of (A) receiving a single data unit over a communication network, the data unit (i) being transferred via the communication network using a standard communication protocol, (ii) defining both (a) a plurality of new configuration items that define a new configuration of the storage device and (b) a command to be performed by the storage device and (iii) having a standard markup language format, (B) calculating at least one configuration change from a plurality of current configuration items to the new configuration items, the current configuration items defining a current configuration of the storage device, (C) adjusting the storage device into the new configuration based on the at least one configuration change and (D) performing a requested operation with the storage device in the new configuration in response to the command.

FLEXIBLE REDUNDANT ARRAY OF INDEPENDENT DISKS (RAID) COMPUTATION DEVICE

A method for calculating a plurality (M) of redundancy blocks for multiple (N) data blocks of a plurality (D) of words each, the method comprises: receiving the number (M) of redundancy blocks by a calculator that comprises multiple (R) calculation units; configuring the calculator according to M and R; concurrently calculating, if M equals R, by the multiple (R) calculation units of the calculator, R sets of parity vectors, each set includes a plurality (D) of parity vectors; and calculating the plurality (M) of the redundancy blocks based on the R sets of parity vectors.

Combining deduplication with locality for efficient and fast storage

A method begins by receiving a storage request to store one or more portions of a data object. The method continues by determining whether the one or more portions are stored within a dispersed storage network memory. When stored, the method continues by determining a retrieval demand for each portion. When the retrieval demand for a first portion is below a first threshold, the method continues by indicating that an existing copy of the first portion will be used for retrieval requests of the first portion. When the retrieval demand is above the first threshold, the method continues by copying the existing copy to create a first portion copy and indicating that the existing copy will be used for retrieval requests from a first group of user computing devices and indicating that the first portion copy will be used for retrieval requests from a second group of user computing devices.

Retrieving data in a storage network

A method for execution by a dispersed storage and task (DST) client module includes issuing a read threshold number of read slice requests are issued to storage units of the set of storage units. One or more encoded slices of a selected read threshold number of encoded slices are received. When a next encoded data slice of a decode threshold number of encoded data slices is received within a response timeframe, outputting of the next encoded data slice is initiated. When the next encoded data slice is not received within the response timeframe, receiving of another decode threshold number of encoded slices of the set of encoded slices is facilitated. The other decode threshold number of encoded slices are decoded to produce recovered encoded data slices, where the recovered encoded data slices includes at least a recovered next encoded data. Outputting of the recovered next encoded data slice is initiated.

COORDINATED SHARED STORAGE ARCHITECTURE

ストレージシステム、ストレージ制御装置およびストレージ制御方法

Optimized reconstruction and copyback methodology for a disconnected drive in the presence of a global hot spare disk

The present invention is a system for optimizing the reconstruction and copyback of data contained on a failed disk in a multi-disk mass storage system. A system in accordance with the present invention may comprise the following: a processing unit requiting mass-storage; one or more disks configured as a RAID system; an associated global hot spare disk; and interconnections linking the processing unit, the RAID and the global hot spare disk. In a further aspect of the present invention, a method for the reconstruction and copyback of a disconnected RAID disk utilizing a global hot spare disk is disclosed. The method includes: disconnecting a RAID component disk; reconstructing data from the disconnected RAID disk onto a global hot spare disk; reconnecting the disconnected RAID component disk; and copying the reconstructed data from the global hot spare disk back to the reconnected RAID component disk.

Optimized reconstruction and copyback methodology for a failed drive in the presence of a global hot spare disk

Dynamic authorization batching in a dispersed storage network

Data reconstruction

Systems and methods for preventing data loss

VIDEO STORAGE

Video data in the form of a plurality of digitised frames, is stored on a plurality of magnetic disks. Each image frame is striped across a plurality of disks and redundant parity information, derived from the stripes, is written to an additional disk. Disk failure is detected and in response to this detection missing data is regenerated from the parity information. This allows the transfer of video data in real time to be maintained for output s o that the system remains operational. While data is being read in real time, derived from regenerated data, the regenerated data is written to an operational disk, thereby reprotecting the data in the event of a subsequent failure. Frame supplied to output are labelled as being protected or unprotected and application programs may respond to this status information as considered appropriate.

RAID survey device

Method for updating control program of physical storage device in storage virtualization system

Parity-based vital product data backup

A method for maintaining vital product data (VPD) of each field replaceable unit (FRUs) in a computer system, the computer system including a first FRU and a second FRU operatively coupled with the first FRU. The method includes calculating a parity for the VPD of the second FRU, and upon detecting a failure of the second FRU, regenerating the VPD for the failed second FRU using the parity.

Network-coding-based distributed file system

A network-coding-based distributed file system (NCFS) is disclosed. The NCFS may include a file system layer, a disk layer, and a coding layer. The file system layer may be configured to receive a request, for an operation on data within a data block, to specify the data block to be accessed in a storage node of a plurality of storage nodes. The disk layer may provide an interface to the file system to provide access the plurality of storage nodes via a network. The coding layer may be connected between the file system layer and the disk layer, to encode and/or decode functions of fault-tolerant storage schemes based on a class of maximum distance separable (MDS) codes. Additional apparatus, systems, and methods are disclosed.

Identifying a slice error in a dispersed storage network

A method begins by a dispersed storage (DS) processing module identifying a set of collections of records corresponding to a data segment that is stored in a dispersed storage network (DSN) as a set of encoded data slices, wherein a collection of records includes an event record including information regarding an event, a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, and a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests. The method continues with the DS processing module determining whether an error exists for one of the set of encoded data slices based on at least some of the set of collections of records and when the error exists, flagging the one of the set of encoded data slices for potential rebuilding.

Method for improving performance in raid systems

A method for computing and storing parity information in a RAID system includes dividing each segment in a stripe into a data block and a parity block, and storing in each parity block, parity information for a limited number of other data blocks in the stripe. A method for rebuilding data in a RAID system includes rebuilding the data from parity information and storing the rebuilt data on reserve portions of the remaining disks in the system.

Storage system, storage control device, and storage control method

A control device manages a plurality of storage devices so that data to be recorded is redundantly recorded in different storage devices. An error monitoring unit monitors an occurrence of an error in each of the plurality of storage devices to register information indicative of error occurrence conditions in an error information storage unit for each storage device. When the use of one of the plurality of storage devices is stopped, a rebuild controller determines a timing to perform rebuild processing based on past error occurrence conditions in the storage devices other than the one storage device of the plurality of storage devices by referring to information registered in the error information storage unit.

Method of exact repair of pairs of failed storage nodes in a distributed data storage system and corresponding device

The invention proposes a method and corresponding device for exact repair of pairs of failed storage nodes interconnected in a distributed data storage system, which method and device are particularly efficient with respect to reliability while keeping the use of resources of the distributed storage network low.

Acquiring a Trusted Set of Encoded Data Slices

A method begins by a dispersed storage (DS) processing module receiving a decode threshold number of encoded data slices of a set of encoded data slices. The method continues with the DS processing module determining whether to evoke a trust verification function and when the trust verification function is to be evoked, selecting one or more encoded data slices of the set of encoded data slices for trust verification to produce one or more selected encoded data slices. The method continues with the DS processing module sending, to a trusted source, a request to receive the one or more selected encoded data slices and when the one or more selected encoded data slices are received from the trusted source, determining that a trusted set of encoded data slices is available based on the decode threshold number of encoded data slices and the received one or more selected encoded data slices.

Computing Core Application Access Utilizing Dispersed Storage

A computing core includes a processing module, main memory, and a memory controller. The memory controller receives a request to fetch an instruction from the processing module and determines whether the instruction is currently stored in the main memory. When the instruction is not currently stored in the main memory, the memory controller determines whether the instruction is stored in a distributed storage network (DSN) memory as one or more sets of encoded instruction slices; and, when it is, the memory controller addresses the DSN memory to retrieve the one or more sets of encoded instruction slices. When at least a threshold number of encoded instruction slices are retrieved for each of the one or more sets of encoded instruction slices, the one or more sets of encoded instruction slices are decoded using a dispersed storage error coding function to reconstruct the instruction, which is provided to the processing module.

Method and apparatus for dispersed storage of streaming data

A method begins by receiving streaming data. The method continues by partitioning the streaming data into a first data stream and a second data stream. For the first data stream, the method continues by encoding, in accordance with error coding dispersed storage function parameters, the first data stream into a first encoded data slices. The method continues by determining first memory of a DSN to store the first encoded data slices and facilitating storage of the first encoded data slices in the first memory. For the second data stream, the method continues by encoding, in accordance with the error coding dispersed storage function parameters, the second data stream into a second encoded data slices. The method continues by determining second memory of the DSN to store the second encoded data slices and facilitating storage of the second encoded data slices in the second memory.

Secure data parser method and system

A secure data parser is provided that may be integrated into any suitable system for securely storing and communicating data. The secure data parser parses data and then splits the data into multiple portions that are stored or communicated distinctly. Encryption of the original data, the portions of data, or both may be employed for additional security. The secure data parser may be used to protect data in motion by splitting original data into portions of data, that may be communicated using multiple communications paths.

Distributed storage network for modification of a data object

A distributed storage network generates a plurality of data segments from a data object and stores each of the plurality of data segments as a plurality of encoded data slices generated from an error encoding dispersal function. When the distributed storage network receives a modification request for the data object, it determines a size of the plurality of data segments of the data object from a segment size field and identifies one of the plurality of data segments requiring modification. The identified data segment is reconstructed from the plurality of encoded data slices and modified in accordance with the modification request.

Storage system for restoring data stored in failed storage device

A storage system including: a controller; and, a plurality of physical storage devices coupled to the controller and constituting a RAID group; wherein the controller provides one or more logical volumes belonging to the RAID group, each of the one or more logical volumes having a plurality of storage areas; and one or more virtual volumes, wherein, when receiving a write request to a virtual volume of the one or more virtual volumes, the controller is configured to: allocate a storage area in a logical volume to the virtual volume, as an allocated storage area, where reliability of the storage area is used as a criterion in a selection of the storage area for allocation to the virtual volume; and write data to the allocated storage area, wherein, the controller is configured to selectively perform a RAID allocation process to a storage area allocated to a virtual volume.

Storage system and data management method

Storage system comprises a second storage apparatus, which is coupled to multiple first storage apparatuses and is of a different type from the first storage apparatuses, and a first control device, which exists either inside or outside of the second storage apparatus. Row of stripes comprising multiple data elements obtained by segmenting a prescribed data unit, and a redundancy code for rebuilding the data elements, is distributively stored in multiple first storage apparatuses, which are more numerous than the total number of stripe data elements, which are either the data elements or redundancy code, in the row of stripes. The row of stripes is configured to enable the rebuilding of the stripe data elements even when a failure has occurred in up to a prescribed allowable number, which is two or more, of the first storage apparatuses storing the stripe data elements of the relevant row of stripes.

Archival storage and retrieval system

A highly reliable data archival and retrieval method that enables fine grained control over data availability is implemented across a Quality of Service driven archival system, configured to fragment the data into data and parity chunks for storing onto the storage node. The technique employed by the archival system enables files to be read without having need to access any metadata, thereby tolerating complete loss of such metadata. Further, the Quality of Service driven system architecture improves upon the system performance and throughput by means of a storage node regeneration process which ensures balanced load on participating storage node during various storage, retrieval and regeneration operations.

Adaptive rebuilding rates based on sampling and inference

A method for execution by one or more processing modules of a dispersed storage network (DSN), the method begins by monitoring an encoded data slice access rate to produce an encoded data slice access rate for an associated rebuilding rate of a set of rebuilding rates. The method continues by applying a learning function to the encoded data slice access rate based on a previous encoded data slice access rate associated with the rebuilding rate to produce an updated previous encoded data slice access rate of a set of previous encoded data slice access rates. The method continues by updating a score value associated with the updated previous encoded data slice access rate and the rebuilding rate and selecting a slice access scheme based on the updated score value where a rebuild rate selection will maximize a score value associated with an expected slice access rate.

System, method, and computer program for protecting data in persistent memory

According to one embodiment, a computer program embodied on a tangible computer readable medium includes computer code for identifying a first write to a first persistent memory on a first channel and a second write to a second persistent memory on a second channel, computer code for performing a third write to a third persistent memory on a third channel, where the third write includes parity data associated with the first write and the second write, computer code for identifying a failure of the second persistent memory, and computer code for generating recovery data for the second persistent memory, using the first channel, the third channel, and the parity data.

STORAGE UTILIZING A DISTRIBUTED CACHE CHAIN AND A CHECKPOINT DRIVE IN RESPONSE TO A DATA DRIVE CORRUPTION

A storage method and device for a solid-state drive is provided in embodiments of the present disclosure. The method includes: configuring a checkpoint drive and a cache drive; backing up data blocks from a data drive into the checkpoint drive; and in response to the data drives being corrupted, writing into a further data drive part of the data blocks backed up into the checkpoint drive and part of data blocks in the cache drive. The number of required SSD drives can be significantly reduced with the method and device without losing the data restoration capability. In addition, the degrading performance can also be maintained at a relatively high level. 1. A storage method for a solid-state disk , comprising:configuring a checkpoint drive and a cache drive, wherein the cache drive comprises a plurality of solid-state disks, wherein the cache drive includes a cache chain distributed in each solid-state disk drive of the plurality of solid-state disk drives;backing up data blocks from a data drive into the checkpoint drive; andin response to the data drive being corrupted, writing into a further data drive part of the data blocks backed up into the checkpoint drive and part of the data blocks in the cache drive.2. The method of claim 1 , wherein the checkpoint drive comprises a hard disk drive claim 1 , and the data drive comprises a solid-state disk.3. The method of claim 1 , wherein the cache drive is part of the data drive.4. The method of claim 1 , further comprising:before backing up the data blocks from the data drive into the checkpoint drive:assigning a version number to the data drive for each write operation into the data drive and the cache drive; andwriting the data blocks into the data drive and the cache drive.5. The method of claim 4 , wherein backing up the data blocks from the data drive into the checkpoint drive comprises:sequentially backing up the data blocks from the data drive into the checkpoint drive; andrecording a latest version number as a ...

PROVIDING REDUNDANCY IN A VIRTUALIZED STORAGE SYSTEM FOR A COMPUTER SYSTEM

A method for providing redundancy in a virtualized storage system for a computer system is provided. The method includes determining a first set of first logical addresses to provide a virtual storage volume. A redundancy schema is then selected to provide redundancy data for primary data stored in the first set of first logical addresses. A second set of second logical addresses is determined to provide logical storage for the primary data and for the redundancy data. The first set of first logical addresses and the second set of second logical addresses are then mapped and a set of physical addresses is selected from a set of physical storage elements. Mapping between the second set of second logical addresses and the set of physical addresses is then performed to provide physical storage for the primary data and the redundancy data stored in the virtual storage volume. 1. A method comprising:identifying a failed physical storage device;identifying a stride of a virtual volume that has a mapping to a first physical logical address on the failed physical storage device;selecting a second physical logical address on a different physical storage device than the failed physical storage device;reconstructing data stored in the first physical logical address based on redundancy data for the stride; andstoring the reconstructed data in the second physical logical address.2. The method of claim 1 , wherein the stride is identified based on an address-mapping table.3. The method of claim 2 , further comprising:writing the second physical logical address and an internal virtual logical address corresponding to the first physical logical address to non-volatile memory.4. The method of claim 3 , further comprising:updating the internal virtual logical address with the second physical logical address to update the address-mapping table with the reconstructed data.5. The method of claim 4 , further comprising:clearing the second physical logical address and the internal virtual ...

METHOD, DEVICE AND COMPUTER PROGRAM PRODUCT FOR MANAGING STORAGE SYSTEM

Embodiments of the present disclosure relate to a method, device and a computer program product for managing a storage system. The storage system includes a disk array. A method includes determining, based on a first number of disks in the disk array, a second number of spare disks for the disk array. The method further includes creating a spare disk array with the second number of spare disks. The method further includes, in response to a first disk in the disk array failing, allocating a spare logic storage unit from the spare disk array for rebuilding the first disk. In addition, the method further includes rebuilding the first disk with the spare logic storage unit. 1. A method for managing a storage system including a disk array , the method comprising:determining, based on a first number of disks in the disk array, a second number of spare disks for the disk array;creating a spare disk array with the second number of spare disks;in response to a first disk in the disk array failing, allocating a spare logic storage unit from the spare disk array for rebuilding the first disk; andrebuilding the first disk with the spare logic storage unit.2. The method according to claim 1 , wherein the spare disks and the disks in the disk array are of different types claim 1 , and the spare disks have lower latency and higher throughput than the disks in the disk array.3. The method according to claim 1 , wherein the disks in the disk array include a magnetic disk and the spare disks include a solid state disk with a lower Write Per Day (WPD).4. The method according to claim 1 , wherein creating the spare disk array comprises:determining a type of the spare disk array based on the second number; andcreating the spare disk array of the type.5. The method according to claim 1 , wherein allocating the spare logic storage unit comprises:allocating, from the spare disk array, the spare logic storage unit having a same size as the first disk.6. The method according to claim 1 , ...

Method, device and computer program product for managing a storage system

Embodiments of the present disclosure provide method, device and computer program product for managing a storage system. The storage system includes a disk having a plurality of extents. The method comprises obtaining metadata associated with an RAID stripe in a first RAID of a first type, the first RAID including at least a part of extents from the plurality of extents. The method also comprises allocating an additional extent to the RAID stripe. The method further comprises converting, based on the allocated additional extent and by modifying the metadata, the first RAID of the first type into a second RAID of a second type in a degraded mode. Additionally, the method comprises initiating, based on the modified metadata, a rebuilding process for the second RAID, so as to convert the second RAID from the degraded mode to a normal mode.

STORING RELATED DATA IN A DISPERSED STORAGE NETWORK

A method includes determining, by a first client and a second client module, that a first data object and a second data objects are related. The method further includes dispersed storage error encoding the first data object to produce a first plurality of sets of encoded data slices and the second data object to produce a second plurality of sets of encoded data slices. The method further includes sending, by the first client module, first requests regarding the first plurality of sets of encoded data slices to a set of storage units. The method further includes sending, by the second client module, second requests regarding the second plurality of sets of encoded data slices to the set of storage units. The method further includes processing, by a binding module, remaining phases of the first and second write requests as a single write operation for the first and second data objects. 1. A method for execution within a dispersed storage network (DSN) , the method comprises:determining, by a first client and a second client module, that a first data object and a second data objects are related;dispersed storage error encoding, by the first client module, the first data object to produce a first plurality of sets of encoded data slices;dispersed storage error encoding, by the second client module, the second data object to produce a second plurality of sets of encoded data slices;sending, by the first client module, first write requests regarding the first plurality of sets of encoded data slices to a set of storage units of the DSN, wherein the first write requests include a first transaction number and a first indication that the first data object is related to the second data object;sending, by the second client module, second write requests regarding the second plurality of sets of encoded data slices to the set of storage units, wherein the second write requests include a second transaction number and a second indication that the second data object is related to ...

SYSTEM AND METHOD FOR TRANSFERRING TRADITIONAL RAID TO DISTRIBUTED ARRAY

In one embodiment, a method includes iteratively selecting an original, unmoved stride from an original array until all original strides have been moved, determining a target stride location in a distributed array, determining a state of the target stride from the following states: an old state indicating unmoved data, a blank state indicating no data, and a new state indicating migrated data, determining that all target stripes are blank, moving data from the original stride to the target stripes when all the target stripes are blank, delaying the moving of the data from the original stride to the target stripes in the target stride when any of the target stripes of the target stride are in the old state and waiting until all the target stripes of the target stride are in the blank state, and determining that all original strides from the original array have been moved. 1. A computer program product for migrating data to a distributed array , the computer program product comprising a computer readable storage medium having program instructions embodied therewith , the program instructions readable and/or executable by a processor to cause the processor to:iteratively select, by the processor, an original, unmoved stride from an original array until all original strides have been moved;determine, by the processor, a target stride location in a distributed array:determine, by the processor, a state of the target stride, the state being chosen from: an old state indicating unmoved data is stored therein, a blank state indicating no data is stored therein, and a new state indicating that migrated data is stored therein:determine, by the processor, whether all target stripes of the target stride are in the blank state;move, by the processor, data from the original stride to the target stripes in the target stride when all the target stripes of the target stride are in the blank state;delay, by the processor, the moving of the data from the original stride to the target ...

DISPERSED STORAGE NETWORK SECURE HIERARCHICAL FILE DIRECTORY

A method includes creating a file directory entry in a directory file of a secure hierarchical file directory system for a file. The file directory entry includes a path name, an encryption access control list, and a source name. The file is encrypted with a key and the key is encrypted with each public key of user devices authorized to access the file. The encryption access control list includes identities of the set of user devices and the set of object content keys. The method further includes encrypting the directory file using a second key. The method further includes generating second object content keys based on the second key and public keys of second user devices authorized to access the directory file. The method further includes creating a next level directory file entry in a next higher directory file of the secure hierarchical file directory system for the directory file. 1. A method for execution by a processing module to generate a secure hierarchical file directory system , the method comprises:encrypting a directory file using a second key to produce an encrypted directory file of the secure hierarchical file directory system for a file, wherein the directory file includes a file directory entry having a path name for the file, an encryption access control list, and a source name of the file, wherein the file is encrypted with a key, wherein the key is encrypted with each public key of a set of user devices that is authorized to access the file to produce a set of object content keys, and wherein the encryption access control list includes identities of the set of user devices and the set of object content keys;generating a second set of object content keys based on the second key and public keys of a second set of user devices authorized to access the directory file; andcreating a next level directory file entry in a next higher directory file of the secure hierarchical file directory system for the directory file, wherein the next level file ...

STORING DATA IN ACCORDANCE WITH A PERFORMANCE THRESHOLD

A computer readable storage medium includes a first memory section that stores operational instructions that, when executed by a computing device of a dispersed storage network (DSN), cause the computing device to, for a first phase of a multiple phase write operation, determine whether to send a write performance threshold number of write requests to storage units. When yes, a second memory section storing operational instructions causes the computing device to, send them to storage units. When no, a third memory section storing operational instructions causes the computing device to determine whether to change the write performance threshold number for a second phase of the multiple write operation. When the write performance threshold number is to be changed, change the write performance threshold number and send the changed write performance threshold number of write commit requests to a second subset of the set of storage units. 1. A computer readable storage medium comprises: 'for a first phase of a multiple phase write operation to write a set of encoded data slices to a set of storage units of the DSN, determine whether to send a write performance threshold number of write requests to a subset of the set of storage units regarding a write performance threshold number of encoded data slices of the set of encoded data slices, wherein a data segment of data is dispersed storage error encoded into the set of encoded data slices, wherein a decode threshold number of encoded data slices of the set of encoded data slices is required to recover the data segment, wherein a write threshold number of favorable write responses is required to move to a next phase of the multiple phase write operation, wherein the set of encoded data slices includes a total number of encoded data slices, and wherein the write performance threshold number is less than the total number and equal to the write threshold number;', 'a first memory section that stores operational instructions ...

RAID WITH HETEROGENEOUS COMBINATIONS OF SEGMENTS

The presently disclosed subject matter includes various inventive aspects, which are directed to protecting data stored in a data storage system using a RAID protection scheme, which enables, responsive to a storage device failure, to execute a rebuild process that uses a total number of storage devices which is greater than the number of segments in a RAID stripe. 1. A computer implemented method of data protection in a storage system using RAID; the method comprising:allocating N storage devices constituting a RAID pool; each storage device is dedicated for storing a plurality of segments, where a sequence of segments spanning across all N storage devices constitute a RAID row, each RAID row comprises M RAID stripes, where M>1;responsive to a need to write data in the storage system, using at least one processor for:generating one or more RAID rows, and for each RAID row:generating M RAID stripes, each RAID stripe comprising at least K segments, where K*M≤N, the at least K segments including a plurality of data segments and at least one parity segment;allocating each one of the K*M segments of the M RAID stripes to a respective storage device from among the N storage devices in the RAID pool, such that segments from different RAID stripes are mixed across the RAID row; andwriting the K*M segments of the M RAID stripes to the respective storage devices as a single RAID row; wherein mixing of segments varies in different RAID rows, such that when data of two or more RAID rows is written, different combinations of segments from different stripes are written to different storage devices.2. The computer implemented method of further comprising implementing a randomized distribution scheme for randomly distributing the at least K segments of the M RAID stripes to respective storage devices from among the N storage devices.3. The computer implemented method of further comprising:responsive to a storage device failure, using at least one processor for executing a rebuild ...

MECHANISM FOR PERSISTING MESSAGES IN A STORAGE SYSTEM

A plurality of storage nodes is provided. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes is configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided. 1. A plurality of storage nodes , comprising:the plurality of storage nodes configured to initiate an action based on redundant copies of metadata, such that a source authority of one of the plurality of storage nodes receives a message, records the message redundantly throughout the plurality of storage nodes, then delivers the message to a destination authority of a further one of the storage nodes responsive to achieving a level of redundancy for the redundant copies of the metadata regarding the message, and wherein the destination authority acts upon the message.2. The plurality of storage nodes of claim 1 , wherein the action includes one of a data recovery claim 1 , or a data rebuild claim 1 , wherein the single chassis has a communications interconnect that couples the plurality of storage nodes.3. The plurality of storage nodes of claim 1 , further comprising:each of the plurality of storage nodes having a capacitor coupled to a RAM (random access memory), the capacitor having sufficient capacity to power the RAM for a predefined time period in event of a power loss.4. The plurality of storage nodes of claim 1 , wherein the plurality of storage nodes are distributed across multiple ...

REBUILDING DATA PREVIOUSLY STORED ON A FAILED DATA STORAGE DRIVE

In response to detecting the failure of a data storage drive, a drive rebuild operation is performed. The drive rebuild operation is performed by periodically i) calculating a target rebuild rate that enables data previously stored on the failed data storage drive to be completely rebuilt on at least one other data storage drive within an expected drive rebuild time window, ii) calculating a new value for at least one drive rebuild parameter based on a value for the drive rebuild parameter corresponding to the target rebuild rate indicated by a static rebuild rate data structure and a current level of host I/O (Input/Output) activity, and iii) rebuilding the data previously stored on the failed data storage drive on the at least one other data storage drive for a predetermined time period with the at least one drive rebuild parameter set to the new value for the drive rebuild parameter. 1. A method comprising: calculating a target rebuild rate that enables data previously stored on the failed data storage drive to be completely rebuilt on at least one other data storage drive within an expected drive rebuild time window;', 'calculating a new value for at least one drive rebuild parameter based on a value for the drive rebuild parameter corresponding to the target rebuild rate indicated by a static rebuild rate data structure and a current level of host I/O (Input/Output) activity; and', 'rebuilding the data previously stored on the failed data storage drive on the replacement data storage drive for a predetermined time period with the at least one drive rebuild parameter set to the new value for the drive rebuild parameter., 'in response to detecting failure of a data storage drive, performing a drive rebuild operation by periodically2. The method of claim 1 , wherein rebuilding the data previously stored on the failed data storage drive on the replacement data storage drive during the predetermined time period includes executing a plurality of rebuild jobs in ...

OPTICAL COLD STORAGE

Various embodiments (“systems”) are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. “Migration worker” components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file “aggregates” (collections of portions of one or more data files) are to be distributed across optical disks (“disks”) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity. 1. A computer-implemented method for transferring data to an optical data storage system , comprising:dividing a file stored in a primary storage into a multiple chunks including at least a first chunk and a second chunk;transferring the chunks onto multiple optical disks, at least the first chunk and the second chunk on different optical disks;maintaining a database for storing file allocation information for association between the chunks and the optical disks; andin response to the file allocation information of the database being unavailable, rebuilding the database including the file allocation information based on information stored in headers or footers appended to the data chunks stored on the optical disks.2. The computer-implemented method of claim 1 , wherein encoding at least one of the chunks comprises applying a Reed-Solomon encoding.3. The computer-implemented method of claim 1 , the method further comprising deleting the file from the primary storage system.4. The ...

Methods and systems for storing data in a redundant manner on a plurality of storage units of a storage system

Described herein are techniques for storing data in a redundant manner on a plurality of storage units of a storage system. While all of the storage units are operating without failure, only error-correction blocks are stored on a first one of the storage units, while a combination of data blocks and error-correction blocks are stored on a second one of the storage units. Upon failure of the second storage unit, one or more data blocks and one or more error-correction blocks formerly stored on the second storage unit are reconstructed, and the one or more reconstructed data blocks and the one or more reconstructed error-correction blocks are stored on the first storage unit.

PRIORITIZED DATA REBUILDING IN A DISPERSED STORAGE NETWORK

A method begins with a processing module querying distributed storage network (DSN) storage units regarding storage errors associated with a data segment. The method continues with the processing module receiving query responses and depending on the responses, assigning a first threshold priority or a second threshold priority to encoded data slices (EDSs) associated with the data segment. The method proceeds with the processing module, depending on the assigned threshold priority, issuing read slice requests and rebuilding EDS associated with the data segment. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:transmitting a plurality of queries to a plurality of storage units of the DSN, wherein each query of the plurality of queries is directed at a storage unit of a plurality of storage units of the DSN associated with a first data segment;receiving one or more query response message of a plurality of query response messages from the plurality of storage units of the DSN, wherein each query response message of the plurality of query response messages is associated with a query of the plurality of queries;determining, based on the plurality of query response messages received from the plurality of storage units of the DSN, whether a first threshold number of error-free dispersed storage error encoded data slices (EDSs) has been stored in the plurality of storage units of the DSN, wherein the first threshold number of error-free dispersed storage error EDSs were produced from the first data segment;when the first threshold number of error-free dispersed storage error EDSs has been stored in the plurality of storage units of the DSN, assigning a first rebuilding priority to the first data segment;when the first threshold number of error-free dispersed storage error EDSs has not been stored in the plurality of storage units of the DSN, determining Whether a second ...

DYNAMIC STORAGE MAP CONSTRUCTION IN A DISTRIBUTED STORAGE SYSTEM

A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method begins by generating a dispersed storage (DS) unit access request based on a DSN address, identifying a target DS unit based on the DSN address, outputting the DS unit access request to the target DS unit, identifying a DSN address range associated with the target DS unit in accordance with a storage map when receiving an access response addressing error. The method continues by issuing a range owner request to the target DS unit that includes the identified DSN address range associated with the target DS unit and updating the storage map based on received range owner response. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:generating a dispersed storage (DS) unit access request based on a DSN address;identifying a target DS unit based on the DSN address;outputting the DS unit access request to the target DS unit;identifying a DSN address range associated with the target DS unit in accordance with a storage map when receiving an access response addressing error;issuing a range owner request to the target DS unit that includes the identified DSN address range associated with the target DS unit; andupdating the storage map based on received range owner response.2. The method of claim 1 , wherein the generating a dispersed storage (DS) unit access request includes determining the DSN address based on one or more of: a directory lookup claim 1 , a dispersed hierarchical index lookup claim 1 , or when writing new data.3. The method of claim 2 , wherein the generating a dispersed storage (DS) unit access request further includes generating a slice name based on the DSN address for inclusion in the DS unit access request.4. The method of claim 1 , wherein the identifying a target DS unit based on the DSN address includes one or more ...

USING DELETABLE USER DATA STORAGE SPACE TO RECOVER FROM DRIVE ARRAY FAILURE

A plurality of stripe zones are mapped across drives in an array. A capacity of each of the stripe zones is smaller than or equal to an average size of the drives. A failed drive is determined in the array. A deletable stripe zone is selected that is being used for user data storage. The deletable stripe zone is taken offline and used to rebuild a subset of the plurality of the stripe zones affected by the failed drive. 1. A method , comprising:mapping a plurality of stripe zones across drives in an array, wherein a capacity of each of the stripe zones is smaller than or equal to an average size of the drives;identifying a failed drive in the array;selecting a deleteable stripe zone that is being used for user data storage;taking the deletable stripe zone offline; andusing the deletable stripe zones to rebuild a subset of the plurality of the stripe zones affected by the failed drive.2. The method of claim 1 , wherein the deletable stripe zone is part of a selected logical volume that includes two or more stripe zones claim 1 , and wherein taking the deletable stripe zone offline comprises taking the two or more stripe zones offline.3. The method of claim 2 , wherein a portion of the capacity of the selected logical volume is not needed to rebuild the subset of the stripe zones affected by the failed drive claim 2 , the method further comprising using the portion of the capacity of the selected logical volume as spare capacity.4. The method of claim 3 , wherein the selected logical volume is selected to minimize the spare capacity.5. The method of claim 1 , wherein identifying the failed drive comprises determining that only part of the failed drive has failed.6. The method of claim 1 , further comprising utilizing a user-defined policy that is automatically used to select the deletable stripe zone in response to identifying the failed drive.7. The method of claim 1 , wherein the deletable stripe zone is selected based on a priority of user data stored on the ...

READ-IF-NOT-REVISION-EQUALS PROTOCOL MESSAGE

A method for a dispersed storage network begins by receiving one of at least a threshold number of DSTN concurrency requests that includes a header section and a payload section. The method continues by determining, in response to the one of at least the threshold number of DSTN concurrency requests, a most recent slice revision number based on a slice name contained in the slice name section. The method continues by determining whether the last known revision number is less than the most recent revision number. The method continues by generating a DSTN concurrency response to return at least one slice of a greater revision number than the last known revision number of the request. The method continues by determining whether an error condition exists and generating an DSTN concurrency response to indicate that an unfavorable revision mismatch does not exist. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:receiving one of at least a threshold number of DSTN concurrency requests that includes a header section and a payload section, wherein the payload section includes a transaction number, a last known slice revision number, and a slice name section;determining, in response to the one of at least a threshold number of DSTN concurrency requests, a most recent slice revision number based on a slice name contained in the slice name section;determining whether the last known slice revision number is less than the most recent slice revision number;when the last known slice revision number is less than the most recent slice revision number, generating a DSTN concurrency response to return at least one slice of a greater revision number than the last known slice revision number of the request;when the last known slice revision number is not less than the most recent slice revision number, generating the DSTN concurrency response to indicate that an unfavorable slice ...

COOPERATIVE DECENTRALIZED REBUILD SCANNING

A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method begins by receiving one or more list requests within a timeframe. The method continues by identifying one or more slice name ranges associated with the one or more list requests to produce active address ranges. The method continues by determining a scan address range based on the active address ranges utilizing a range selection approach. The method continues by generating a pillar width number of list requests for an associated set of distributed storage and task (DST) execution units based on the scan address range and sending the pillar width number of list requests to the associated set of DST execution units. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:receiving one or more list requests within a timeframe;identifying one or more slice name ranges associated with the one or more list requests to produce active address ranges;determining a scan address range based on the active address ranges utilizing a range selection approach;generating a pillar width number of list requests for an associated set of distributed storage and task (DST) execution units based on the scan address range; andsending the pillar width number of list requests to the associated set of DST execution units.2. The method of claim 1 , wherein receiving one or more list requests includes receiving a list request from another DST execution unit claim 1 , where a storage set of DST execution units includes the other DST execution unit and the DST execution unit.3. The method of claim 2 , wherein the one or more list requests include one or more of a list range request and a list digest request.4. The method of claim 3 , wherein the list range request and a list digest request includes one or more of: a start slice name range claim 3 , a maximum ...

PRIORITIZING REBUILDING BASED ON A LONGEVITY ESTIMATE OF THE REBUILT SLICE

A method begins by determining to rebuild a slice or slices to at least one of multiple distributed storage network (DSN) memory locations. The method continues by calculating a future risk estimation of each one of the multiple DSN memory locations, the future risk estimation including one or more risk factors. The method continues by determining a selection of which of the multiple DSN memory locations to rebuild and rebuilding the slice in the DSN memory of the selection. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:determining, by a rebuild module, to rebuild a slice to at least one of multiple DSN memory locations;calculating, by the rebuild module, a future risk estimation of each one of the multiple DSN memory locations, the future risk estimation including one or more risk factors;determining, based on the future risk estimation, a selection of which of the multiple DSN memory locations to rebuild; andrebuilding the slice in the DSN memory of the selection.2. The method of claim 1 , wherein the selection of which of the multiple DSN memory locations to rebuild comprises selecting a location that equates to a lower probability of impending data loss.3. The method of claim 1 , wherein the selection of which of the multiple DSN memory locations to rebuild comprises: evaluating two or more of the multiple DSN memory locations for rebuilding the slice and prioritizing rebuilding of the slice to a location that yielded a greatest longevity estimate for a slice rebuilt there.4. The method of claim 1 , wherein the selection of which of the multiple DSN memory locations to rebuild comprises selecting a location that equates to longevity due to replacement.5. The method of claim 4 , wherein longevity due to replacement comprises a lower likelihood of any of: updating claim 4 , overwriting or replacement.6. The method of claim 5 , wherein a frequency of the ...

ASSIGNING PRIORITIZED REBUILD RESOURCES OPTIMALLY

A method begins by determining to rebuild one or more encoded data slices to a dispersed storage network (DSN) memory unit. The method continues by determining a rebuild rate of the DSN memory unit. The method continues by determining, based on the rebuild rate, a rebuild rate status of the DSN memory unit. The method continues by when the rebuild rate status is a high rebuild rate status, reducing the rebuild rate to the DSN memory unit. The method continues by rebuilding, when the rebuild rate is not zero, the one or more encoded data slices in the DSN memory unit. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:determining, by a plurality of rebuild modules, to rebuild one or more encoded data slices to a DSN memory unit;determining, by at least one of the plurality of rebuild modules, a rebuild rate of the DSN memory unit;determining, based on the rebuild rate, a rebuild rate status of the DSN memory unit; andwhen the rebuild rate status is a high rebuild status, reducing the rebuild rate to the DSN memory unit; andrebuilding, when the rebuild rate is not reduced to zero, the one or more encoded data slices in the DSN memory unit.2. The method of claim 1 , wherein the rebuild rate is based on one or more of: a number of the plurality of rebuild modules concurrently rebuilding the one or more encoded slices in the DSN memory unit claim 1 , rate of received rebuilds by the DSN memory unit claim 1 , regular access traffic rate of the DSN memory unit claim 1 , or network capacity of the DSN memory unit.3. The method of further comprising claim 1 , when the rebuild rate status is a low rebuild rate status claim 1 , increasing the rebuild rate to the DSN memory unit.4. The method of further comprising claim 1 , when the rebuild rate status is a high rebuild rate status claim 1 , reducing the rebuild rate to zero and delaying the rebuilding of the one or more ...

EFFICIENT METHOD FOR REBUILDING A SET OF ENCODED DATA SLICES

A method begins by a dispersed storage (DS) processing module identifying an encoded slice requiring rebuilding. The method continues by the DS processing module determining whether the encoded data slice is part of a fan-out encoded data slice group and, when it is part of a fan-out encoded data slice group determining by the DS processing module whether a valid encoded data slice of the fan-out data slice group is available. When a copy of the encoded data slice of the fan-out encoded data slice group is not available, the method continues by the DS processing module rebuilding the encoded data slice. A storage unit then stores the rebuilt encoded data slice and creates copies of the rebuilt encoded data slice to produce a rebuilt fan-out encoded data slice group. 1. A method comprises:identifying, by a device of a dispersed storage network (DSN), an encoded data slice requiring rebuilding;determining, by the device, whether the encoded data slice is part of a fan-out encoded data slice group; determining, by the device, whether a copy of a valid encoded data slice of the fan-out encoded data slice group is available;', 'when a copy of the encoded data slice of the fan-out encoded data slice group is not available, rebuilding, by the device or another device, the encoded data slice to produce a rebuilt encoded data slice;', 'storing, by a storage unit, the rebuilt encoded data slice;', 'and creating, by the storage unit, one or more copies of the rebuilt encoded data slice to produce a rebuilt fan-out encoded data slice group., 'when the encoded data slice is part of the fan-out encoded data slice group2. The method of claim 1 , further comprising propagating storage of copies of the rebuilt encoded data slice within the storage unit when the valid encoded data slice of the fan-out encoded data slice group is not available.3. The method of claim 1 , further comprising propagating storage of copies of the rebuilt encoded data slice to other storage units with same ...

WRITING ENCODED DATA SLICES IN A DISPERSED STORAGE NETWORK

A computing device of a dispersed storage network (DSN) includes a memory, interface and a processing module operable to identify an encoded data slice of the set of encoded data slices to produce an identified encoded data slice. The processing module generates a set of first write requests regarding the set of encoded data slices less the identified encoded data slice, and generates a set of second write requests regarding the identified encoded data slice. The set of second write requests include the identified encoded data slice and replications of the identified encoded data slice. The processing module sends the set of first write requests to storage units of the DSN, and sends the set of second write requests to a set of storage units of the DSN, where each storage unit of the set of storage units is sent a corresponding one of the set of second write requests. 1. A computing device of a dispersed storage network (DSN) , the computing device comprises:an interface;memory; and identifying an encoded data slice of the set of encoded data slices for a redundant write operation to produce an identified encoded data slice;', 'generating a set of first write requests regarding the set of encoded data slices less the identified encoded data slice;', 'generating a set of second write requests regarding the identified encoded data slice, the set of second write requests including the identified encoded data slice and replications of the identified encoded data slice;', 'sending the set of first write requests to storage units of the DSN memory; and', 'sending the set of second write requests to a set of storage units of the DSN memory,', 'wherein each storage unit of the set of storage units is sent a corresponding one of the set of second write requests., 'a processing module operably coupled to the memory and the interface, wherein the processing module is operable to write a set of encoded data slices to DSN memory by2. The computing device of claim 1 , wherein the ...

Adaptive extra write issuance within a dispersed storage network (dsn)

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and to the memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. For example, the computing device determines storage unit (SU)-based write transfer rates and/or a DSN-based write transfer rate for a write request of encoded data slices (EDSs) to the SUs within the DSN. Then, the computing device processes the SU-based write transfer rates and/or DSN-based write transfer rate to determine characteristics of the various SUs as well as operation of the DSN. The computing device then selectively modifies parameters (e.g., write request time interval) and/or issues additional write request(s) for some or all of the EDSs based on such characteristics.

Identifying a slice name information error in a dispersed storage network

A method begins by a processing module sending list digest requests to a set of dispersed storage (DS) units. The method continues with the processing module receiving list digest responses from at least some of the set of DS units and determining whether an inconsistency exists between first and second list digest responses of the list digest responses. The method continues with the processing module requesting at least a portion of each of the slice name information lists from first and second DS units of the set of DS units and identifying a slice name information error associated with the inconsistency based on the at least a portion of each of the slices name information lists of the first and second DS units when the inconsistency exists between first and second list digest responses of the list digest responses.

Methods and systems for storing information that facilitates the reconstruction of at least some of the contents of a storage unit on a storage system

The failure of a storage unit in a storage array of a storage system may render the storage unit unresponsive to any requests. Any writes to the storage system that occur during the failure of the storage unit will not be reflected on the failed unit, rendering some of the failed unit's data stale. Assuming the failed unit's data is not corrupted but is just stale, a partial rebuild may be performed on the failed unit, selectively reconstructing only data that is needed to replace the stale data. Described herein are techniques for storing information that identifies the data that needs to be rebuilt. When the storage unit fails, the segment identifier associated with the last data segment written to the storage system may be stored. Upon the storage unit recovering, the storage system can rebuild only those data segments whose identifier is greater than the stored segment identifier.

Methods and systems for storing information that facilitates the reconstruction of at least some of the contents of a storage unit on a storage system

The failure of a storage unit in a storage array of a storage system may render the storage unit unresponsive to any requests. Any writes to the storage system that occur during the failure of the storage unit will not be reflected on the failed unit, rendering some of the failed unit's data stale. Assuming the failed unit's data is not corrupted but is just stale, a partial rebuild may be performed on the failed unit, selectively reconstructing only data that is needed to replace the stale data. Described herein are techniques for storing information that identifies the data that needs to be rebuilt. When the storage unit fails, the segment identifier associated with the last data segment written to the storage system may be stored. Upon the storage unit recovering, the storage system can rebuild only those data segments whose identifier is greater than the stored segment identifier.

Prioritizing Locations For Error Scanning In A Storage Network

A method includes obtaining, by a computing device of a storage network, provenance information for data associated with a set of storage units of the storage network, where the data is error encoded into a set of encoded data slices, in accordance with error encoding parameters, for storage in the set of storage units. The method further includes determining, by the computing device, probable error locations associated with the set of storage units based on the provenance information. The method further includes scanning, by the computing device, the probable error locations to determine whether an error exists for the set of encoded data slices. 1. A method comprises:obtaining, by a computing device of a storage network, provenance information for data associated with a set of storage units of the storage network, wherein a data segment of the data is error encoded into a set of encoded data slices in accordance with error encoding parameters and stored in the set of storage units;determining, by the computing device, probable error locations associated with the set of storage units based on the provenance information; andscanning, by the computing device, the probable error locations to determine whether an error is associated with an encoded data slice of the set of encoded data slices.2. The method of claim 1 , wherein the scanning is performed in accordance with a rebuild scanning schedule.3. The method of further comprises:updating a rebuild scanning schedule with the probable error locations to produce an updated rebuild scanning schedule, wherein the probable error locations are prioritized within the updated rebuild scanning schedule.4. The method of further comprises:generating at least a portion of the provenance information regarding a plurality of tasks that are performed for the data.5. The method of further comprises:determining that the error is the encoded data slice of the set of encoded data slices needs rebuilding; andrebuilding the encoded data ...

SELECTING STORAGE RESOURCES IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) client module includes determining a storage unit performance level for storage units of a set of storage units. Storage resources of the set of storage units are temporarily selected based on the storage unit performance levels to produce identities of candidate primary storage slots. Identities of candidate primary storage slots are exchanged with another DST client module. Selection of primary storage slots of the candidate primary storage slots is coordinated with the other non-transitory computer readable storage medium to produce identities of selected primary storage slots. Data stored in the set of storage units is accessed using the selected primary storage slots. 1. A method for execution by a dispersed storage and task (DST) client module that includes a processor , the method comprises:determining a storage unit performance level for storage units of a set of storage units;temporarily selecting storage resources of the set of storage units based on the storage unit performance levels to produce identities of candidate primary storage slots;exchanging identities of candidate primary storage slots with another DST client module;coordinating selection of primary storage slots of the candidate primary storage slots with the another DST client module to produce identities of selected primary storage slots; andaccessing data stored in the set of storage units using the selected primary storage slots.2. The method of claim 1 , wherein the data accessed using the selected primary storage slots includes a data segment claim 1 , and wherein the data segment was dispersed storage error encoded to produce a set of encoded data slices for storage in the selected primary storage slots.3. The method of claim 2 , wherein the identities of the candidate primary storage slots are determined such that a decode threshold number of encoded data slices of the set of encoded data slices are available from any two ...

Client provided request prioritization hints

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. For example, the computing device generates a prioritized request that includes at least one of a task for execution or a priority level based on information stored within a storage unit (SU) of a plurality of storage units (SUs) implemented within the DSN. Note that the information corresponds to a data object that is related to a set of encoded data slices (EDSs) that are distributedly stored within the DSN. The computing device then transmits the prioritized request to the SU and receives, from the SU, a response to the prioritized request.

Combining deduplication with locality for efficient and fast storage

A method begins by receiving a storage request to store one or more portions of a data object and determining whether the one or more portions are stored within a dispersed storage network memory. When stored, the method continues by determining a retrieval demand for a portion of the one or more portions. When below a first threshold, the method continues by indicating that an existing copy of the one or more portions will be used for retrieval requests for the one or more portions. When above the first threshold, the method continues by copying the existing copy to create a first one or more portion copies and indicating the one or more portions will be used for retrieval requests from a first group of user computing devices and indicating the first one or more portion copies will be used for retrieval requests from a second group.

RESOLVING WRITE CONFLICTS IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) execution unit includes receiving a write slice request from a requesting entity, where the write slice request includes a plurality of encoded data slices and a corresponding plurality of slice names. The plurality of slice names are interpreted to produce a slice name range. Whether a write lock conflict exists is determined based on the slice name range. An unfavorable write slice response is issued to the requesting entity when the write lock conflict is determined to exist. When the write lock conflict is determined to not exist, local storage of the plurality of encoded data slices is initiated, a favorable write slice response is issued to the requesting entity, and the processing system indicates that the slice name range is not locked when completing the local storage of the plurality of encoded data slices. 1. A method for execution by a dispersed storage and task (DST) execution unit that includes a processor , the method comprises:receiving a write slice request from a requesting entity, wherein the write slice request includes a plurality of encoded data slices and a corresponding plurality of slice names;interpreting the plurality of slice names to produce a slice name range;determining whether a write lock conflict exists based on the slice name range;issuing an unfavorable write slice response to the requesting entity when the write lock conflict is determined to exist; initiating local storage of the plurality of encoded data slices;', 'issuing a favorable write slice response to the requesting entity; and', 'indicating that the slice name range is not locked when completing the local storage of the plurality of encoded data slices., 'when the write lock conflict is determined to not exist2. The method of claim 1 , wherein the plurality of encoded data slices are associated with at least one data segment claim 1 , and wherein the at least one data segment was dispersed storage error ...

STORING A PLURALITY OF CORRELATED DATA IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) client module includes obtaining a plurality of sorted data entries. A data access performance goal level associated with the plurality of sorted data entries is obtained, and obtaining DSN performance information is obtained. Compression parameters are selected based on the data access performance goal level and the DSN performance information. Sorted data entries of the plurality of sorted data entries are selected based on the selected compression parameters to produce a data object. The data object is compressed to produce a compressed data object using the selected compression parameters. The compressed data object is dispersed storage error encoded to produce one or more sets of encoded data slices for storage in a set of storage units. 1. A method for execution by a dispersed storage and task (DST) client module that includes a processor , the method comprises:obtaining a plurality of sorted data entries;obtaining a data access performance goal level associated with the plurality of sorted data entries;obtaining DSN performance information;selecting compression parameters based on the data access performance goal level and the DSN performance information;selecting sorted data entries of the plurality of sorted data entries based on the selected compression parameters to produce a data object;compressing the data object to produce a compressed data object using the selected compression parameters; anddispersed storage error encoding the compressed data object to produce one or more sets of encoded data slices for storage in a set of storage units.2. The method of claim 1 , wherein the sorted data entries share a common affiliation claim 1 , and wherein the common affiliation includes at least one of: belonging to a common index node of a dispersed hierarchical index or belonging to a common storage vault.3. The method of claim 1 , wherein the sorted data entries share a common affiliation claim 1 , ...

DETERMINING MISSING ENCODED DATA SLICES

A method includes determining, by a computing device of a dispersed storage network (DSN), a source name for a data object to be scanned for missing encoded data slices. The method further includes issuing list source requests to the set of storage units. When a list source response is not received from a storage unit of the set of storage units within a response timeframe, the method further includes identifying one or more encoded data slices stored on the storage unit as potentially missing encoded data slices; determining a next level missing encoded data slice determination approach for the storage unit based on one or more of: a number of potentially missing encoded data slices, a performance goal, a network loading level, a rebuilding loading level, a predetermination, and an entry of a system registry; and executing the next level missing encoded data slice determination approach. 1. A method comprises:determining, by a computing device of a dispersed storage network (DSN), a source name for a data object to be scanned for missing encoded data slices, wherein the data object is stored as a plurality of sets of encoded data slices on a set of storage units of the DSN;issuing, by the computing device, list source requests to the set of storage units; and identifying, by the computing device, one or more encoded data slices stored on the storage unit as potentially missing encoded data slices;', 'determining, by the computing device, a next level missing encoded data slice determination approach for the storage unit based on one or more of: a number of potentially missing encoded data slices, a performance goal, a network loading level, a rebuilding loading level, a predetermination, and an entry of a system registry; and', 'executing, by the computing device, the next level missing encoded data slice determination approach., 'when a list source response is not received from a storage unit of the set of storage units within a response timeframe2. The method of ...

REBUILDING DATA WHILE READING DATA IN A DISPERSED STORAGE NETWORK

A method includes executing a read request regarding a set of encoded data slices stored in storage units of the DSN, where a data segment of data is encoded in accordance with a dispersed storage error coding function to produce the set of encoded data slices, and where a decode threshold number of encoded data slices of the set of encoded data slices is required to rebuild the data segment. The method further includes reconstructing the data segment from at least the decode threshold number of encoded data slices of the set of encoded data slices, where the at least the decode threshold number of encoded data slices are included in read responses from the storage units. The method further includes determining an encoded data slice not used in the reconstructing needs to be rebuilt and rebuilding the encoded data slice using the reconstructed data segment. 1. A method for execution by one or more computing devices of a dispersed storage network (DSN) , the method comprises:executing a read request regarding a set of encoded data slices stored in storage units of the DSN, wherein a data segment of data is encoded in accordance with a dispersed storage error coding function to produce the set of encoded data slices, and wherein a decode threshold number of encoded data slices of the set of encoded data slices is required to rebuild the data segment;reconstructing the data segment from at least the decode threshold number of encoded data slices of the set of encoded data slices, wherein the at least the decode threshold number of encoded data slices are included in read responses from the storage units;determining an encoded data slice of the set of encoded data slices not used in the reconstructing needs to be rebuilt; andrebuilding the encoded data slice using the reconstructed data segment.2. The method of claim 1 , wherein the determining the encoded data slice requires rebuilding comprises:interpreting a read response of the read responses to determine a revision ...

MANAGING MAPPED RAID EXTENTS IN DATA STORAGE SYSTEMS

A technique for use in managing mapped RAID extents is disclosed. A neighborhood matrix denoting a distribution of neighboring physical storage portions across the N physical storage devices of the pool is created and initialized. A disk extent consumed array denoting a disk consumption metric is created and initialized. Disks in an extent pool are analyzed to identify a least consumed disk in the extent pool and a disk extent is allocated from the identified disk. The allocated disk extent is configured as a first disk extent for a first RAID extent. The neighborhood matrix and disk extent consumed array are updated appropriately. All the disks in the extent pool are analyzed to identify a valid disk extent that will result in the lowest variance value in the neighborhood matrix. The identified valid disk extent is allocated and the neighborhood matrix and disk extent consumed array are again updated. 1. A method for use in managing mapped RAID extents in data storage systems , the method comprising:creating and initializing a neighborhood matrix denoting a distribution of neighboring physical storage portions across N physical storage devices of a pool;using a disk utilization ratio to evaluate a first disk extent choice for each raid extent;using the neighborhood matrix to evaluate the first disk extent choice for each raid extent;analyzing all the disks in the extent pool to identify a valid disk extent that will result in a lowest variance value of neighborhood matrix values; andallocating the identified valid disk extent and updating the neighborhood matrix and the disk extent consumed array.2. The method of claim 1 , wherein using the disk utilization ratio to evaluate the first disk extent choice comprises:analyzing disks in an extent pool to identify a disk servicing fewest input output operations (I/Os) in the extent pool and allocating a disk extent from the identified disk.3. The method of claim 1 , wherein using the neighborhood matrix to evaluate the ...

DISTRIBUTED STORAGE NETWORK WITH ALTERNATIVE FOSTER STORAGE APPROACHES AND METHODS FOR USE THEREWITH

A method includes encoding input data into a plurality of slices. The plurality of slices are sent to a plurality of distributed storage and task execution units for storage, the plurality of distributed storage and task execution units being located at a corresponding plurality of sites A storage failure is detected corresponding to at least one of the plurality of slices corresponding to at least one of the plurality of the distributed storage and task execution units and at least one of the corresponding plurality of sites A foster storage approach is determined. At least one alternative distributed storage and task execution unit is selected in accordance with the foster storage approach. At least one foster slice is generated corresponding to the at least one of the plurality of slices. The at least one foster slice is sent to the at least one alternative distributed storage and task execution unit. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:encoding input data into a plurality of slices;sending the plurality of slices to a plurality of distributed storage and task execution units for storage, the plurality of distributed storage and task execution units being located at a corresponding plurality of sites;detecting a storage failure corresponding to at least one of the plurality of slices corresponding to at least one of the plurality of the distributed storage and task execution units and at least one of the corresponding plurality of sites;determining a foster storage approach;selecting at least one alternative distributed storage and task execution unit in accordance with the foster storage approach;generating at least one foster slice corresponding to the at least one of the plurality of slices; andsending the at least one foster slice to the at least one alternative distributed storage and task execution unit.2. The method of wherein determining ...

STORAGE SYSTEM RESOURCE REBUILD BASED ON INPUT-OUTPUT OPERATION INDICATOR

An apparatus comprises a storage system comprising at least one processing device and a plurality of storage devices. The at least one processing device is configured to obtain a given input-output operation from a host device and to determine that the given input-output operation comprises an indicator having a particular value. The particular value indicates that the given input-output operation is a repeat of a prior input-output operation. The at least one processing device is further configured to rebuild at least one resource of the storage system that is designated for servicing the given input-output operation based at least in part on the determination that the given input-output operation comprises the indicator having the particular value. 1. An apparatus comprising: to obtain a given input-output operation from a host device;', 'to determine that the given input-output operation comprises an indicator having a particular value, the particular value indicating that the given input-output operation is a repeat of a prior input-output operation; and', 'to rebuild at least one resource of the storage system that is designated for servicing the given input-output operation based at least in part on the determination that the given input-output operation comprises the indicator having the particular value., 'a storage system comprising at least one processing device and a plurality of storage devices, the at least one processing device configured2. The apparatus of wherein the indicator comprises at least one bit of the given input-output operation.3. The apparatus of wherein the indicator comprises information identifying the at least one resource as a corrupted resource that requires a rebuild.4. The apparatus of wherein the at least one resource comprises at least one of metadata claim 1 , a cache claim 1 , a cache buffer claim 1 , a fibre channel (FC) exchange claim 1 , an input-output control block (IOCB) claim 1 , an IO record claim 1 , a masking table ...

STORAGE CONTROL DEVICE AND STORAGE CONTROL METHOD

A storage control apparatus includes a controller to, when more storage devices, among a plurality of storage devices across which a plurality of information areas storing information representing redundant data and one or more spare areas are distributed, than the number of the spare areas fail, perform a rebuild process of information stored in a plurality of information areas of a failed first storage device included in the plurality of combinations of the plurality of information areas and the one or more spare areas, the rebuild process including restoring information corresponding to one information area of the failed first storage device included in one combination among the plurality of combinations, and determining a write destination storage device to which the restored information is to be written in accordance with the number of times information is read from a non-failed second storage device. 1. A storage control apparatus comprising: determine whether more storage devices, among a plurality of storage devices across which a plurality of information areas storing information representing redundant data and one or more spare areas are distributed, than the number of the spare areas fail, and', 'perform a rebuild process of information stored in a plurality of information areas of a failed first storage device included in the plurality of combinations of the plurality of information areas and the one or more spare areas, when determining that the more storage devices than the spare areas fail,', restoring information corresponding to one information area of the failed first storage device included in one combination among the plurality of combinations, and', 'determining a write destination storage device to which the restored information is to be written in accordance with the number of times information is read from a non-failed second storage device., 'the rebuild process including], 'a controller configured to2. The storage control apparatus ...

Buffered Automated Flash Controller Connected Directly to Processor Memory Bus

A mechanism is provided for direct memory access in a storage device. Responsive to the buffered flash memory module receiving from a memory bus of a processor a memory command specifying a write operation, the mechanism initializes a first memory buffer in the buffered flash memory module. The mechanism writes to the first memory buffer based on the memory command. Responsive to the buffer being full, the mechanism deterministically maps addresses from the first memory buffer to a plurality of solid state drives in the buffered flash memory module using a modular mask based on a stripe size. The mechanism builds a plurality of input/output commands to persist contents of the first memory buffer to the plurality of solid state drives according to the deterministic mapping and writes the contents of the first memory buffer to the plurality of solid state drives in the buffered flash memory module according to the plurality of input/output commands. 1. A computer program product comprising a computer readable storage medium having a computer readable program stored therein , wherein the computer readable program , when executed on a controller in a buffered flash memory module , causes the controller to:responsive to the buffered flash memory module receiving from a memory bus of a processor a memory command specifying a write operation, initialize a first memory buffer in the buffered flash memory module;write to the first memory buffer based on the memory command;responsive to the buffer being full, deterministically map addresses from the first memory buffer to a plurality of solid state drives in the buffered flash memory module using a modular mask based on a stripe size;build a plurality of input/output commands to persist contents of the first memory buffer to the plurality of solid state drives according to the deterministic mapping; andwrite the contents of the first memory buffer to the plurality of solid state drives in the buffered flash memory module ...

Systems and methods for storing, maintaining, and accessing objects in storage system clusters

Systems and methods providing for storing fragments for one or more source objects at storage nodes of multiple cluster instances comprising a cluster set are disclosed. Repair of the stored data is operated within cluster instances autonomously or semi-autonomously of other cluster instances of the cluster set according to embodiments. Embodiments may provide a storage process operable to generate fragments for a first source object to be stored in a storage system using a first encoding, wherein a different plurality of fragments of the generated fragments are stored at different cluster instances of a cluster set. A repair process may be enacted at each cluster instance of the cluster set, wherein a repair process enacted at a cluster instance is used to maintain the recoverability of the fragments of the first source object stored at that cluster instance using a second encoding.

System and method for generating field replaceable unit information files

Technologies are described herein for generating field replaceable unit (FRU) information files in a format that is readable by a management controller in accordance with IPMI such that the FRU and the management controller are interoperable. In particular, a FRU installation station is in operative communication with a general purpose computer comprising a FRU information conversion module. A script utilized by the FRU information conversion module is configured to receive FRU information relating to a specified FRU and convert the information FRU binary files or a FRU image binary. The FRU binary files or FRU image binary are then received by the FRU installation station where they are subsequently transmitted to the inventory device of the specified FRU storage space according to the specified IPMI standard.

Storage Systems with Peer Data Scrub

Example peer storage systems, storage devices, and methods provide data scrub using a peer communication channel. Peer storage devices establish peer communication channels that communicate data among the peer storage devices. A storage device may identify data segments from their storage medium for a data scrub process. A peer storage device may be identified that contains corresponding data segments to the data segment being scrubbed. A corresponding lock command may be sent over the peer communication channel to lock the corresponding data segments during the data scrub process. A data scrub error report may be generated from the data scrub process. If an error is discovered during the data scrub process the storage device may use the peer communication channel to retrieve recovery data from peer storage devices to rebuild the data segment with the error. 1. A storage device , comprising:a processor;a memory;at least one storage medium;an interface configured to communicate with a plurality of peer storage devices over a peer communication channel; and selecting a first data segment on the at least one storage medium for a data scrub process;', 'identifying at least one peer storage device including at least one corresponding data segment;', 'sending a corresponding data lock command to the at least one peer storage device over the peer communication channel;', 'performing a data scrub operation on the first data segment; and', 'generating a data scrub error report for the first data segment in response to performing the data scrub operation., 'a data scrub module stored in the memory and executable by the processor to perform operations comprising2. The storage device of claim 1 , wherein the peer communication channel bypasses a storage control plane for managing the storage device.3. The storage device of claim 1 , further comprising: a peer storage device identifier for each of the plurality of peer storage devices, wherein the peer storage device identifier ...

MAINTAINING LOGICAL TO PHYSICAL ADDRESS MAPPING DURING IN PLACE SECTOR REBUILD

An apparatus comprises at least one processing device comprising a processor coupled to a memory that is configured to initiate a read data request utilizing a logical address of a content addressable storage system that maps to a physical address comprising an offset on a storage device that internally maps the offset to a first sector. The processing device is also configured to determine a health of the first sector responsive to the read data request failing, to recover data stored in the first sector responsive to the first sector being a bad sector, and to overwrite the recovered data to the logical address while maintaining the mapping to the physical address by directing a write of the recovered data to the offset to update the internal mapping of the offset in the storage device to a new physical location corresponding to a second sector different than the first sector. 1. An apparatus comprising:at least one processing device comprising a processor coupled to a memory; to initiate a read data request utilizing a given logical address of a content addressable storage system, the given logical address mapping to a given physical address comprising a designated physical address offset on a given storage device of the content addressable storage system, the given storage device internally mapping the designated physical address offset to a first sector;', 'to determine a health of the first sector responsive to the read data request failing;', 'to recover data stored in the first sector responsive to the determined health indicating that the first sector is a bad sector; and', 'to overwrite the recovered data to the given logical address while maintaining the mapping of the given logical address to the given physical address in the content addressable storage system by directing a write of the recovered data to the designated physical address offset to update the internal mapping of the designated physical address offset in the given storage device to a new ...

RAID Storage System with Logical Data Group Rebuild

Example redundant array of independent disks (RAID) storage systems and methods provide rebuild of logical data groups. Storage devices are configured as a storage array for storing logical data groups distributed among the storage devices. The logical data groups are written in a configuration of RAID stripes in the storage devices. A failed storage device may be rebuilt using the RAID stripes and completed rebuilds of logical blocks may be tracked during the device rebuild process. A logical group rebuild status may be determined by comparing the completed rebuilds of logical blocks to a logical group map. The logical group rebuild status for each logical data group may be provided as complete in response to all logical blocks in the logical data group having been rebuilt. In the event the array rebuild fails, the logical groups that did complete rebuild may be brought online as a partially completed rebuild to prevent the loss of the entire array. 1. A storage system , comprising:a plurality of storage devices configured as a storage array for storing a plurality of logical data groups distributed among the plurality of storage devices; a rebuild module configured to rebuild a failed storage device from the plurality of storage devices based on the configuration of RAID stripes; and', 'a rebuild progress monitor configured to track completed rebuilds of logical blocks from the failed storage device during a device rebuild process;, 'a redundant array of independent disks (RAID) controller configured to write the plurality of logical data groups in a configuration of RAID stripes across at least one RAID group in the plurality of storage devices, the RAID controller comprisinga logical group index comprising a logical group map for each of the plurality of logical data groups, wherein the logical group map identifies the logical blocks corresponding to each of the plurality of logical data groups; and determine a logical group rebuild status by comparing the ...

Rebuild Assist Using Failed Storage Device

Methods, systems, and other aspects for reconstructing data and rebuilding a failed storage device in a storage system using one or more functioning compute resources and/or storage resources of the failed storage device. For example, a method may include, responsive to a detection of a failed storage device in a storage system, locating data and redundancy information in functioning storage device(s) in the storage system for reconstructing data of the failed storage device; issuing peer-to-peer commands to the functioning storage device(s) to obtain the data and the redundancy information from the functioning storage device(s); and reconstructing the data of the failed storage device based on the data and the redundancy information obtained from the functioning storage device(s), wherein a functioning compute resource of the failed computing device at least partially performs one or more of the locating, issuing, and reconstructing. 1. A computer-implemented method , comprising:responsive to a detection of a failed storage device in a storage system, locating data and redundancy information in one or more functioning storage devices in the storage system for reconstructing data of the failed storage device;issuing one or more peer-to-peer commands to the one or more functioning storage devices to obtain the data and the redundancy information from the one or more functioning storage devices; andreconstructing the data of the failed storage device based on the data and the redundancy information obtained from the one or more functioning storage devices,wherein a functioning compute resource of the failed computing device at least partially performs one or more of the locating, issuing, and reconstructing.2. The computer-implemented method of claim 1 , further comprising:writing the reconstructed data of the failed storage device to one or more available locations in the storage system.3. The computer-implemented method of claim 1 , wherein the functioning compute ...

INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING SYSTEM MANAGEMENT METHOD, AND PROGRAM THEREOF

An object of the present invention is to provide a redundant information processing system that can continue its operation without stopping as much as possible. To achieve this, the information processing system includes multiple storage nodes including processors, memories, and storage devices, as well as a network for connecting the storage nodes. The processor of at least one storage node performs the steps of: obtaining the configuration information of the information processing system; obtaining first failure information related to a first failure occurred in the information processing system; calculating the availability level of the information processing system when a second failure further occurs, based on the configuration information and the first failure information; and controlling the operation of the system based on the availability level. Then, the processor outputs a notification. 1. An information processing system comprising:a plurality of storage nodes including processors, memories, and storage devices; anda network for connecting the plurality of storage nodes,wherein the processor of at least one storage node performs the steps of:obtaining the configuration information of the information processing system;obtaining first failure information related to a first failure occurred in the information processing system;calculating the availability level of the information processing system when a second failure further occurs, based on the configuration information and the first failure information; andcontrolling the operation based on the availability level.2. The information processing system according to claim 1 , wherein the processor calculates the availability level by using the rebuild time of the configuration of the information processing system for the first failure information claim 1 , as well as the failure occurrence probability that the second failure will occur.3. The information processing system according to claim 2 , wherein the ...

DISPERSED STORAGE NETWORK SECURE HIERARCHICAL FILE DIRECTORY

A method includes creating a file directory entry in a directory file of a secure hierarchical file directory system for a file. The file directory entry includes a path name, an encryption access control list, and a source name. The file is encrypted with a key and the key is encrypted with each public key of user devices authorized to access the file. The encryption access control list includes identities of the set of user devices and the set of object content keys. The method further includes encrypting the directory file using a second key. The method further includes generating second object content keys based on the second key and public keys of second user devices authorized to access the directory file. The method further includes creating a next level directory file entry in a next higher directory file of the secure hierarchical file directory system for the directory file. 1. A method for execution by a processing module to generate a secure hierarchical file directory system , the method comprises:creating a file directory entry in a directory file of the secure hierarchical file directory system for a file, wherein the file directory entry includes a path name for the file, an encryption access control list, and a source name of the file, wherein the file is encrypted with a key, wherein the key is encrypted with each public key of a set of user devices that is authorized to access the file to produce a set of object content keys, and wherein the encryption access control list includes identities of the set of user devices and the set of object content keys;encrypting the directory file using a second key to produce an encrypted directory file;generating a second set of object content keys based on the second key and public keys of a second set of user devices authorized to access the directory file; andcreating a next level directory file entry in a next higher directory file of the secure hierarchical file directory system for the directory file, ...

STORAGE CONTAINER DS UNIT REASSIGNMENT BASED ON DYNAMIC PARAMETERS

A dispersed storage network (DSN) includes storage units storing dispersed error-encoded data slices, and logically grouped into a container served by a computing device configured as a container controller. A method for use in such a DSN, includes obtaining by the container controller, a container status of the container, the container status including a status indicator associated with a first storage unit included in the container. The container controller determines whether the container status compares favorably to a status threshold, and in response to an unfavorable comparison, determines a data slice to be migrated from the first storage unit. The method further includes determining, at the container controller, a second storage unit to receive the data slice to be migrated, and facilitating, at the container controller, migration of the data slice to be migrated from the first storage unit to the second storage unit. 1. A method for use in a dispersed storage network including a plurality of storage units storing dispersed error-encoded data slices , the storage units logically grouped into a container served by a computing device configured as a container controller , the method comprising:obtaining, by the container controller, a container status of the container, the container status including a status indicator associated with a first storage unit included in the container;determining, at the container controller, whether the container status compares favorably to a status threshold;in response to an unfavorable comparison, determining, at the container controller, a data slice to be migrated from the first storage unit;determining, at the container controller, a second storage unit to receive the data slice to be migrated; andfacilitating, at the container controller, migration of the data slice to be migrated from the first storage unit to the second storage unit.2. The method of claim 1 , wherein:at least one of the plurality of storage units is ...

Data processing method for AFA storage device to restore data protection capability and the AFA storage device utilizing the same

An all flash array storage device includes a flash memory array including multiple flash memories and a microprocessor. The flash memories correspond to multiple logical aggregation units. Each logical aggregation unit includes multiple stripes. Each stripe includes multiple storage units, including multiple data units and at least one parity unit. The microprocessor detects a status of the flash memories. In response to a detection result indicating that one of the flash memories has been removed from the flash memory array, the microprocessor sequentially performs a repair operation on the stripes comprised in one or more logical aggregation units that have been written with data. In the repair operation of one stripe, the microprocessor recalculates protection information of the stripe according to content stored in a portion of data units of the stripe and writes the recalculated protection information in one or more storage units of the stripe. 1. An all flash array storage device , comprising:a flash memory array, comprising a plurality of flash memories, wherein the flash memories correspond to a plurality of logical aggregation units, each logical aggregation unit comprises a plurality stripes, each stripe comprises a plurality of storage units, and the storage units comprise a plurality of data units for storing data and at least one parity unit for storing protection information; anda microprocessor, coupled to the flash memory array and configured to detect a status of the flash memories, wherein when the microprocessor detects one of the flash memories has been removed from the flash memory array, the microprocessor is configured to sequentially perform a repair operation on the stripes comprised in one or more logical aggregation units that have been written with data, andwherein in the repair operation corresponding to one stripe, the microprocessor is configured to recalculate protection information of the stripe according to content stored in a ...

FALLBACK DELEGATES FOR MODIFICATION OF AN INDEX STRUCTURE

A method includes identifying a fallback delegate device of a plurality of delegate devices for changing one or more nodes of a plurality of nodes of a hierarchical index structure, where a primary delegate device of the plurality of delegate devices is responsible for changing the one or more nodes and where each delegate device of a plurality of delegate devices is assigned an individual global namespace address that is partially based on a location within the DSN. The method further includes determining to process a change to a node of the one or more nodes using the fallback delegate device. 1. A method for execution by a device of a dispersed storage network (DSN) , the method comprises:identifying a fallback delegate device of a plurality of delegate devices for changing one or more nodes of a plurality of nodes of a hierarchical index structure, wherein a primary delegate device of the plurality of delegate devices is responsible for changing the one or more nodes, wherein each delegate device of a plurality of delegate devices is assigned an individual global namespace address that is partially based on a location within the DSN, and wherein the hierarchical index structure is used to identify particular data stored in the DSN; anddetermining to process a change to a node of the one or more nodes using the fallback delegate device.2. The method of claim 1 , wherein the identifying the fallback delegate device comprises:performing a first modification of a global namespace address of the primary delegate device to produce a first modified address identifier;determining whether another delegate device of the plurality of delegate devices has the global namespace address corresponding to the first modified address identifier; andwhen the global namespace address of other delegate device corresponds to the first modified address identifier, identifying the other delegate device as the fallback delegate device for the one or more nodes.3. The method of further ...

Updating an encoded data slice

A distributed storage (DS) processing unit distributes an initial set of encoded data slices and an initial parity slice, for storage in multiple DS units. The initial parity slice is associated with an initial encoded data slice stored in a first DS unit. The DS processing unit transmits an updated encoded data slice reflecting changes to the initial encoded data slice, and obtains, from the first DS unit, delta parity information associated with a delta parity slice. The delta parity slice reflects differences between parity values calculated using the updated data slice and the initial data slice. An updated parity slice is generated by performing an exclusive OR (XOR) operation on the initial parity slice and the delta parity slice. A message transmitted to a second DS unit, which currently stores the initial parity slice, directs the second DS unit to store the updated parity slice.

INITIALIZING STORAGE UNIT PERFORMANCE RANKINGS IN NEW COMPUTING DEVICES OF A DISPERSED STORAGE NETWORK

Methods for use in a dispersed storage network (DSN) to enable sharing of storage unit performance ranking information between computing devices. In one example, a new computing device of the DSN requests performance ranking information from one or more established computing devices, or from a database of such information that is curated by the DSN. After receiving and storing such information, the new computing device utilizes it to select one or more storage units/sets of storage units for performing dispersed storage operations (e.g., retrieval or storage of dispersed storage error encoded data). The computing device may then update the stored performance ranking information with information regarding such dispersed storage operations, and subsequently share the updated performance ranking information with other computing devices of the DSN and/or a database maintained by the DSN. 1. A method for execution by one or more processing modules of a first computing device of a dispersed storage network (DSN) , the DSN having storage resources including a plurality of storage units , the method comprises:determining that the first computing device and the plurality of storage units have not engaged in dispersed storage operations;requesting, from at least a second computing device of the DSN, performance ranking information relating to the plurality of storage units; andreceiving responsive performance ranking information, the received performance ranking information including data generated by the at least a second computing device based on dispersed storage operations between the at least a second computing device and the plurality of storage units.2. The method of further comprises:performing a dispersed storage operation involving the first computing device and one or more of the plurality of storage units;generating, by the first computing device, additional performance ranking information based on the dispersed storage operation; andupdating the responsive ...

EVALUATION FOR REBUILDING PERFORMANCE OF REDUNDANT ARRAYS OF INDEPENDENT DISKS

Embodiments of the present disclosure provide a solution of evaluating a rebuilding performance of a redundant array of independent disks. In some embodiments, there is provided a computer-implemented method, comprising: simulating, based on a first group of redundant arrays of independent disks, a rebuilding process for a second group of redundant arrays of independent disks; obtaining a first performance metric of the simulated rebuilding process; and identifying a factor associated with the rebuilding performance of the second group of redundant arrays of independent disks based on the first performance metric. 1. A computer-implemented method , comprising:simulating, using a first group of redundant arrays of independent disks (RAID), at least one input output generator, and one or more spare disks, a rebuilding process for a second group of redundant arrays of independent disks;obtaining a first performance metric, of a plurality of performance metrics, of the simulated rebuilding process;identifying, based on the first performance metric, a factor associated with rebuilding performance of the second group of redundant arrays of independent disks; andre-simulating the rebuilding process for the second group of redundant arrays of independent disks by changing at least one of the following: a number of the input output generators, and a number of the one or more spare disks.2. The method according to claim 1 , wherein the first group of redundant arrays of independent disks includes a group of conventional redundant arrays of independent disks claim 1 , and the second group of redundant arrays of independent disks includes a group of mapped redundant arrays of independent disks.3. The method according to claim 1 , wherein simulating the rebuilding process for the second group of redundant arrays of independent disks comprises:disabling a disk in the first group of redundant arrays of independent disks and disabling rebuilding of the first group of redundant ...

DETECTING STORAGE ERRORS IN A DISPERSED STORAGE NETWORK

A method includes determining a root cause for a rebuilding request of an encoded data slice of a set of encoded data slices, where the rebuilding request includes a slice name of the encoded data slice corresponding to a slice error. The method further includes establishing a pricing level as a user pricing level when the root cause is a user-centric root cause, and establishing the pricing level as a non-user pricing level when the root cause is a non-user-centric root cause. The method further includes facilitating the rebuilding of the encoded data slice, and generating billing information for the rebuilding based on the pricing level. 1. A method for execution by one or more computing devices of a dispersed storage network (DSN) , the method comprises:determining a root cause for a rebuilding request of an encoded data slice of a set of encoded data slices, wherein the rebuilding request includes a slice name of the encoded data slice corresponding to a slice error;establishing a pricing level as a user pricing level when the root cause is a user-centric root cause;establishing the pricing level as a non-user pricing level when the root cause is a non-user-centric root cause;facilitating the rebuilding of the encoded data slice; andgenerating billing information for the rebuilding based on the pricing level.2. The method of claim 1 , wherein the determining the root cause for the rebuilding request is based on one or more of:an error message;an event log;a query; anda test.3. The method of claim 1 , wherein the slice error comprises at least one of:an indication that the encoded data slice is missing; andan indication that the encoded data slice is associated with an unfavorable integrity value.4. The method of claim 1 , wherein determining that the root cause is the user-centric root cause comprises at least one of:detecting that the encoded data slice is not written in DSN memory;detecting that the encoded data slice was not successfully committed in a write ...

DYNAMIC MEMORY CELL REPLACEMENT USING COLUMN REDUNDANCY

A memory chip comprises a main memory array having a plurality of memory columns, a redundancy memory column associated with the main memory array, and a hit logic circuitry configured to generate a plurality of hit logic signals by a plurality of hit logic units in the hit logic circuitry to enable dynamic replacement of a defective memory cell in one of the memory columns for dynamic replacement by the redundancy memory column when the memory array is in operation. 1. A method performed with a memory chip having a memory array having a plurality of memory columns , comprising:generating a plurality of hit logic signals by a plurality of hit logic units in a hit logic circuitry associated with the memory array to enable dynamic replacement a defective memory cell in one of the memory columns;designating the memory column containing the defective memory cell for replacement if the defective memory cell is identified;replacing the designated memory column with a redundancy memory column by dynamically reasserting the address of the designated column to the redundancy memory column when the memory array is in operation;wherein the dynamic replacement of the designated column having the defective memory cell with the redundancy column is completed before the next read/write operation is performed on the memory array.2. The method of claim 1 , further comprising:testing one of the plurality of memory columns for defective memory cell when the corresponding hit logic signal is at a first logic state.3. The method of claim 1 , further comprising:replacing the defective memory cell when clock signal of the memory chip is at a second logic state.4. The method of claim 1 , further comprising:adopting nested thermometer coding of input signals to the hit logic circuitry to generate the plurality of hit logic signals to enable more than one of the memory columns be tested for defects at the same time.5. The method of claim 4 , further comprising:organizing the input signals to ...

RECOVERING DATA IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) client module includes issuing a read threshold number of read slice requests are issued to storage units of the set of storage units. One or more encoded slices of a selected read threshold number of encoded slices are received. When a next encoded data slice of a decode threshold number of encoded data slices is received within a response timeframe, outputting of the next encoded data slice is initiated. When the next encoded data slice is not received within the response timeframe, receiving of another decode threshold number of encoded slices of the set of encoded slices is facilitated. The other decode threshold number of encoded slices are decoded to produce recovered encoded data slices, where the recovered encoded data slices includes at least a recovered next encoded data. 1. A method for execution by a dispersed storage and task (DST) client module that includes a processor , the method comprises:determining a selected read threshold number of encoded slices of each set of encoded slices of a plurality of sets of encoded slices stored in a set of storage units, wherein each set of the a plurality of sets of encoded slices corresponds to one of a plurality of data segments;issuing a read threshold number of read slice requests to storage units of the set of storage units, where the read threshold number of read slice requests includes identities of the selected read threshold number of encoded slices;receiving one or more encoded slices of the selected read threshold number of encoded slices;for each data segment of the plurality of data segments, when a next encoded data slice of a decode threshold number of encoded data slices is received within a response timeframe, initiating outputting of the next encoded data slice;when the next encoded data slice is not received within the response timeframe, facilitating receiving of another decode threshold number of encoded slices of the set of encoded ...

UTILIZING FAST MEMORY DEVICES TO OPTIMIZE DIFFERENT FUNCTIONS

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. A computing device receives a data access request for an encoded data slice (EDS) associated with a data object. The computing device compares a slice name of the data access request with slice names stored within RAM. When the data access request slice name compares unfavorably with those stored slice names, the computing device transmits an empty data access response that includes no EDS to the other computing device without needing to access a hard disk drive (HDD) that stores EDSs. Alternatively, the computing device transmits a data access response that includes the EDS. 1. A computing device comprising:an interface configured to interface and communicate with a dispersed storage network (DSN);memory that stores operational instructions; and transmit, to a storage unit (SU), a data access request for an encoded data slice (EDS) associated with a data object, wherein the data access request includes a data access request slice name for the EDS associated with the data object, wherein the SU includes another memory that includes random access memory (RAM) and a hard disk drive (HDD); and', 'receive, from the SU, an empty data access response that includes no EDS based on an unfavorable comparison, as performed by the SU, of the data access request slice name with a plurality of slice names stored within the RAM, wherein the plurality of slice names are respectively associated with a plurality of encoded data slices (EDSs) stored within the HDD of the SU., 'a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable ...

MANAGING MIGRATION OF ENCODED DATA SLICES IN A DISPERSED STORAGE NETWORK

A method begins by a processing module of a dispersed storage network (DSN) determining to modify a configuration of a set of storage units by obtaining a first DSN address range set and first storage information for the set of storage units based on the current configuration. The method continues with the processing module producing a modified and modifying the first DSN address range set to produce a second DSN address range set, where the second DSN address range set is based on the modified configuration and the first storage information. The method continues by transmitting the second DSN address range set to the set of storage units; and facilitating migration of encoded data slices from each storage unit of the set of storage units in accordance with the modified configuration and the second DSN address range set. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises: in response to determining to modify the configuration of the set of storage units, obtaining a first DSN address range set and first storage information for the set of storage units based on the configuration, wherein the first DSN address range set includes a first plurality of address range assignments for the set of storage units such that each address range assignment thereof corresponds to a respective one storage unit of the set of storage units;', 'modifying the configuration of the set of storage units to produce a modified configuration;', 'modifying the first DSN address range set to produce a second DSN address range set, wherein the second DSN address range set is based on the modified configuration and the first storage information, wherein the second DSN address range set includes a second plurality of address range assignment for the set of storage units such that each address range assignment thereof corresponds to the respective one storage unit of the set of storage units;', ' ...

MANAGING UNAVAILABLE STORAGE IN A DISPERSED STORAGE NETWORK

A method begins by a processing module of a dispersed storage network (DSN) identifying a data object of a group of data objects for storage in the DSN and determining micro slice encoding parameters for the encoding the data object. The method continues by identifying a set of distributed storage (DS) units for storing encoded micro slices (EMSs) and generating a set of meta-slices from the set of EMSs, followed by determining whether DS units are available to store the meta-slices, determining a mapping scheme for storing the set of EMSs and mapping each meta-slice associated with a DS unit available to store the EMSs. The method continues by transmitting each meta-slice associated with a DS unit available to store the EMSs and when certain DS units of are not available to store the EMSs, transmitting each meta-slice associated with a DS unit not available to the DS units available to store the EMSs. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:identifying a data object of a group of data objects for storage in the DSN;determining micro slice encoding parameters for the encoding the data object, wherein the data object is segmented into a plurality of data segments, wherein a data segment is encoded into a plurality of encoded micro slices in accordance with micro slice encoding parameters, wherein a micro slice decode threshold number of encoded micro slices of the plurality of encoded micro slices is needed to recover the data segment;dispersed storage error encoding the data segment into a set of encoded micro slices in accordance with micro slice encoding parameters;identifying a set of distributed storage (DS) units for storing the set of encoded micro slices;generating a set of meta-slices from the set of encoded micro slices, wherein each meta-slice is generated according to dispersal parameters;determining whether one or more DS units of the set of ...

Data storage architecture for storing metadata with data

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

Rebuilding a data object using portions of the data object

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

RECONCILIATION IN SYNC REPLICATION

A distributed storage system replicates data for a primary logical storage object on a primary node of the storage system to a secondary logical storage object on a secondary node on the distributed storage system. Failures in writing data to the primary logical storage object or failures in the replication of the data to the secondary logical storage object can cause data that should be synchronized to become divergent. In cases where the data may be divergent, reconciliation operations can be performed to resynchronize the data. 1. A method comprising:receiving, by a first node, a first request to write data for a first logical storage object to a first storage unit coupled to the first node;in response to the first request, performing a first write operation to the first storage unit;replicating the first request to a second node, wherein the replicated first request cause the second node to perform a second write operation to write the data for a second logical storage object on a second storage unit coupled to the second node; anddetermining a combined response based, at least in part, on a first result of the first write operation and a second result of the second write operation, wherein the first node defers responding to the first request until receipt of the first result and the second result.2. The method of claim 1 , further comprising:in response to determining that the first write operation failed to write the data, reading by the first node, old data from a location targeted by the first write operation and forwarding the old data to the second node.3. The method of claim 2 , further comprising:writing, by the second node, the old data, wherein the old data overwrites the data stored on the second storage unit.4. The method of claim 1 , further comprising:in response to determining that the second write operation failed to write the data to the second storage unit, terminating data replication between the first node and the second node.5. The method ...

DEFERRED REBUILDING OF A DATA OBJECT IN A MULTI-STORAGE DEVICE STORAGE ARCHITECTURE

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device. 1. A computer-implemented method comprising:obtaining, at a storage management computer node of a storage management system, historical information regarding a failure rate of a storage device, the storage device being of a type of multiple storage devices associated with the storage management system, the storage management computer node encoding a data object to generate a first specified number of encoded data segments, the encoded data segments stored at multiple storage computer nodes, the first specified number of encoded data segments including a second specified number of redundant encoded data segments, which are generated to provide a storage resiliency to the data object, the first specified number being a function of the second specified number;determining, using the storage management computer node, predicted information regarding a failure rate of the storage devices based on the historical information;determining, using the storage management computer node, a lifespan of the storage devices as a function of ...

Deterministically determining affinity for a source name range

Systems and methods for storing encoded data slices in a dispersed storage network (DSN) involve receiving a write data object request, determining a source name for the data object, identifying a source name range corresponding to the source name, and identifying a storage pattern associated with the source name range. Based on the source name range, a set of DS units can be identified as affinity DS units that correspond to the storage pattern. The data object is encoded to produce encoded affinity data slices which are output to the affinity DS units. A determination is made that an encoded data slice was improperly stored. One or more other encoded data slices corresponding to the data object may then be output to one or more DS units of the set of units other than the affinity DS units. 1. A method of storing a data object in a dispersed storage network , the dispersed storage network including a plurality of dispersed storage (DS) units , the method comprising:receiving a write data object request;determining a source name for the data object;identifying a source name range corresponding to the source name;identifying a storage pattern associated with the source name range;identifying a set of DS units of the plurality of DS units corresponding to the source name;identifying affinity DS units of the set of DS units that correspond to the storage pattern;encoding the data object to produce encoded affinity data slices;outputting the encoded affinity data slices to the affinity DS units;determining an encoded affinity data slice of the encoded affinity data slices was improperly stored; andoutputting another encoded data slice corresponding to the data object to a DS unit of the set of DS units other than one of the affinity DS units.2. The method of claim 1 , wherein the data object includes a data name.3. The method of claim 2 , wherein the source name is based on the data name.4. The method of claim 1 , wherein the step of identifying the storage pattern ...

Multiplying width and threshold for improved performance and efficiency

A method for execution by a dispersed storage network (DSN), the method begins by receiving data for storage in the DSN. The method continues by selecting a set of storage units, identifying a baseline pillar width and a baseline decode threshold, determining an estimated performance of the set of storage units, determining a parameter multiple based on the estimated performance and the baseline pillar width and the baseline decode threshold, multiplying the parameter multiple by each of the baseline pillar width and the baseline decode threshold to produce a pillar width and a decode threshold, encoding the data using a dispersed storage error coding function in accordance with the pillar width and the decode threshold to produce a plurality of sets of encoded data slices, facilitating storage of the plurality of sets of encoded data slices in the set of storage units and storing the parameter multiple. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:receiving data for storage in the dispersed storage network (DSN);selecting a set of storage units;identifying a baseline pillar width and a baseline decode threshold;determining an estimated performance of the set of storage units;determining a parameter multiple based on the estimated performance and the baseline pillar width and the baseline decode threshold;multiplying the parameter multiple by each of the baseline pillar width and the baseline decode threshold to produce a pillar width and a decode threshold respectively;encoding the data using a dispersed storage error coding function in accordance with the pillar width and the decode threshold to produce a plurality of sets of encoded data slices;facilitating storage of the plurality of sets of encoded data slices in the set of storage units; andstoring the parameter multiple.2. The method of claim 1 , wherein the receiving further includes receiving one or ...

SALTED ZERO EXPANSION ALL OR NOTHING TRANSFORMATION

A method for execution by a dispersed storage network (DSN), the method begins by injecting generated data into a data segment to produce mixed data, partitioning the mixed data to produce first and second data partitions, performing a deterministic function on the first data partition to produce a first key, encrypting the second data partition using the first key to produce an encrypted second data partition, performing the deterministic function on the encrypted second data partition to produce a second key, encrypting the first data partition using the second key to produce an encrypted first data partition, performing the deterministic function on the encrypted first data partition to produce a third key, encrypting the encrypted second data partition to produce a re-encrypted second data partition, aggregating the encrypted first data partition and the re-encrypted second data partition to produce a secure package, and encoding the secure package and storing. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:injecting generated data into a data segment to produce mixed data in accordance with a data injection scheme;partitioning the mixed data in accordance with a partitioning approach to produce first and second data partitions;performing a deterministic function on the first data partition to produce a first key;encrypting the second data partition using the first key to produce an encrypted second data partition;performing the deterministic function on the encrypted second data partition to produce a second key;encrypting the first data partition using the second key to produce an encrypted first data partition;performing the deterministic function on the encrypted first data partition to produce a third key;encrypting the encrypted second data partition using the third key to produce a re-encrypted second data partition;aggregating the encrypted first ...

Extra write scaling for performance and reliability

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. For example, the computing device monitors storage unit (SU)-based write transfer rates and SU-based write failure rates associated with each of the SUs for a write request of encoded data slices (EDSs) to the SUs within the DSN. The computing device generates and maintains a SU write performance distribution based on monitoring of the SU-based write transfer rates and the SU-based write failure rates and adaptively adjusts a trimmed write threshold number of EDSs and/or a target width of EDSs for write requests of sets of EDSs to the SUs within the DSN.

Encoding data for storage in a dispersed storage network

A method for execution by a dispersed storage and task (DST) processing unit that includes a processor includes receiving a data object for storage in the DSN via a network. Available storage unit data is generated, indicating a subset of a plurality of storage units of the DSN that corresponds to a plurality of available storage units. A shortened encoding matrix is generated based on an original encoding matrix and the available storage unit data. A size of the shortened encoding matrix is based on a number of storage units in the plurality of available storage units. A plurality of encoded slices is generated, each for transmission to one of the plurality of available storage units via the network, by performing an encoding function on the shortened encoding matrix and the data obj ect.

VARYING REBUILD TASK PRIORITIES

A method begins by determining whether at least one encoded data slice of a corresponding set of encoded data slices associated with a primary storage unit requires rebuilding and includes one or more excess encoded data slices of the set of encoded data slices stored in a secondary storage unit. The method continues by identifying the excess encoded data slices based on scan response messages from the secondary storage units. The method continues by assigning, for each data segment associated with at least one of an encoded data slice requiring rebuilding and an excess encoded data slice, a priority level in accordance with a prioritization scheme. The method continues by facilitating, for each data segment, rebuilding of the encoded data slices requiring rebuilding and deletion of excess encoded data slices requiring deletion in accordance with the assigned priority level of the data segment. 1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) , the method comprises:determining, for each data segment of a plurality of data segments stored as a plurality of sets of encoded data slices in one or more of primary storage units and secondary storage units of a DSN, whether at least one encoded data slice of a corresponding set of encoded data slices associated with a primary storage unit requires rebuilding and includes one or more excess encoded data slices of the plurality of sets of encoded data slices stored in the secondary storage units;identifying the excess encoded data slices based on scan response messages from the secondary storage units;assigning, for each data segment associated with at least one of an encoded data slice requiring rebuilding and an excess encoded data slice, a priority level in accordance with a prioritization scheme; andfacilitating, for each data segment, rebuilding of the at least one encoded data slice requiring rebuilding and deletion of excess encoded data ...

REBUILDING DATA WHILE READING DATA IN A DISPERSED STORAGE NETWORK

A method includes sending, in response to read operation, read requests regarding a read threshold number of encoded data slices of a set of encoded data slices to storage units, where the read threshold number is between a decode threshold number and a total number of encoded data slices of the set of encoded data slices. The method further includes sending status inquiries regarding a remaining number of encoded data slices of the set of encoded slices to other storage units, where the remaining number equals the total number minus the read threshold number. The method further includes receiving responses from the storage units regarding the read requests and the status inquires. The method further includes determining, based on the responses, that an encoded data slice of the set of encoded data slices requires rebuilding, and rebuilding the encoded data slice while processing the read operation. 1. A method for execution by one or more computing devices of a dispersed storage network (DSN) , the method comprises:sending, in response to a read operation, read requests regarding a read threshold number of encoded data slices of a set of encoded data slices to storage units of the DSN, wherein a data segment of a data object is encoded in accordance with a dispersed storage error coding function to produce the set of encoded data slices, wherein a decode threshold number of encoded data slices of the set of encoded data slices is required to rebuild the data segment, wherein the set of encoded data slices includes a total number of encoded data slices, and wherein the read threshold number is between the decode threshold number and the total number;sending status inquiries regarding a remaining number of encoded data slices of the set of encoded slices to other storage units of the DSN, wherein the remaining number equals the total number minus the read threshold number;receiving responses from the storage units regarding the read requests and the status inquires; ...

Utilizing site write thresholds in a dispersed storage network

A method for execution by a dispersed storage and task (DST) processing unit that includes a processor includes receiving a data object for storage via a network. A plurality of encoded slices are generated by executing an encoding function on the data object based on an information dispersal algorithm (IDA) threshold value. Site dispersal data indicating a plurality of site slice sets is generated, where each site slice set includes a number of unique encoded slices of the plurality of encoded slices that is greater than or equal to a site write threshold value. The site dispersal data further indicates a designated one of a plurality storage sites for each of the plurality of site slice sets. Each of the plurality of site slice sets is transmitted to the corresponding designated one of the plurality of storage sites via the network.

REBUILDING SLICES IN A DISPERSED STORAGE NETWORK

A method for use in a dispersed storage network operates to select a recovery of selected ones of one or more first sets of encoded data slices in response to detecting a storage error associated with the selected ones of the one or more first sets of encoded data slices; issue requests for a second decode threshold number of encoded data slices of selected ones of one or more second sets of encoded data slices corresponding to the selected ones of the one or more first sets of encoded data slices; decode the second decode threshold number of encoded data slices to produce recovered data in response to receiving the second decode threshold number of encoded data slices; encode the recovered data utilizing first IDA parameters associated with the first IDA to produce one or more rebuilt encoded data slices corresponding to the selected ones of the one or more first sets of encoded data slices; and facilitate storage of the one or more rebuilt encoded data slices. 1. A method for execution by a processing system of a dispersed storage and task (DST) integrity processing unit that includes a processor , the method comprises:detecting a storage error associated with the selected ones of the one or more first sets of encoded data slices, wherein data is stored as the one or more first sets of encoded data slices in a plurality of dispersed storage and task execution (DSTE) units, the one or more first sets of encoded data slices encoded utilizing a first information dispersal algorithm (IDA) having a first decode threshold number, wherein the data is also stored as one or more second sets of encoded data slices in a subset of the plurality of DSTE units, the one or more second sets of encoded data slices encoded utilizing a second IDA having a second decode threshold number;issuing requests for a second decode threshold number of encoded data slices of selected ones of the one or more second sets of encoded data slices corresponding to the selected ones of the one or ...

CYCLING OUT DISPERSED STORAGE PROCESSING UNITS FROM ACCESS POOLS TO PERFORM EXPENSIVE OPERATIONS

A method includes identifying, by a dispersed storage (DS) processing unit of a plurality of DS processing units of a dispersed storage network (DSN), a DSN operation to be performed that will adversely interfere with processing of DSN access requests. The method further includes sending, by the DS processing unit, a suspension request to a process balancing module of the DSN where the suspension request is for temporarily suspending processing of DSN access requests. The method further includes determining, by the process balancing module, whether sufficient processing resources of active DS processing units of the plurality of DS processing units are available for processing DSN access requests on behalf of the DS processing unit. When determined that sufficient processing resources are available, the method further includes granting the suspension request, and allocating DSN access requests of the DS processing unit to one ore of the active DS processing units. 1. A method comprises:identifying, by a dispersed storage (DS) processing unit of a plurality of DS processing units of a dispersed storage network (DSN), a DSN operation to be performed that will adversely interfere with processing of DSN access requests;sending, by the DS processing unit, a suspension request to a process balancing module of the DSN, wherein the suspension request is for temporarily suspending processing of DSN access requests;determining, by the process balancing module, whether sufficient processing resources of active DS processing units of the plurality of DS processing units are available for processing DSN access requests on behalf of the DS processing unit; and granting, by the process balancing module, the suspension request; and', 'allocating, by the process balancing module, DSN access requests of the DS processing unit to one or more of the active DS processing units., 'when determined that sufficient processing resources are available2. The method of further comprises: ...

PROXYING READ REQUESTS WHEN PERFORMANCE OR AVAILABILITY FAILURE IS ANTICIPATED

A method includes receiving, by a read threshold number of storage units of a dispersed storage network (DSN), the read threshold number of read requests regarding the read threshold number of encoded data slices of a set of encoded data slices. The method further includes determining, by each storage unit of the read threshold number of storage units, whether the storage unit is capable of processing a respective read request. When a particular storage unit is not capable of processing the respective read request, the method further includes sending, by the particular storage unit, a proxy read request to another storage unit that is not in the read threshold number of storage units. The method further includes determining, by the other storage unit, whether the other storage unit is capable of processing the proxy read request and, when it is, processing the proxy read request. 1. A method comprises:identifying, by a computing device of a dispersed storage network (DSN), a read threshold number of storage units of a set of storage units, wherein other storage units in the set of storage units are available as proxy storage units;sending, by the computing device, a read threshold number of read requests regarding a read threshold number of encoded data slices of a set of encoded data slices to the read threshold number of storage units, wherein a data segment of a data object is dispersed storage error encoded into the set of encoded data slices, wherein the set of encoded data slices is stored in the set of storage units, and wherein the read threshold number is less than a total number of encoded data slices in the set of encoded data slices and is equal to or greater than a decode threshold number;determining, by a storage unit of the read threshold number of storage units, whether the storage unit is capable of processing a respective read request of the read threshold number of read requests; sending, by the storage unit, a proxy read request to one of the ...

APPORTIONING STORAGE UNITS AMONGST STORAGE SITES IN A DISPERSED STORAGE NETWORK

A method of apportioning storage units in a dispersed storage network (DSN) includes generating storage unit apportioning data indicating a mapping of a plurality of desired numbers of storage units to a plurality of storage sites based on site reliability data. The mapping includes a first desired number of storage units corresponding to a first one of the plurality of storage sites that is greater than a second desired number of storage units corresponding to a second one of the plurality of storage sites in response to the site reliability data indicating that a first reliability score corresponding to the first one of the plurality of storage sites is more favorable than a second reliability score corresponding to the second one of the plurality of storage sites. A plurality of storage units are allocated to the plurality of storage sites based on the storage unit apportioning data. 1. A method of apportioning storage units in a dispersed storage network (DSN) , the method comprising:generating storage unit apportioning data indicating a mapping of a plurality of desired numbers of storage units to a plurality of storage sites based on site reliability data, wherein the storage unit apportioning data represents each of the plurality of desired numbers of storage units is a numerical value, wherein when the site reliability data includes a first reliability score for the first one of the plurality of storage sites and a second reliability score for the second one of the plurality of storage sites and the first reliability score is more favorable than the second reliability score, the mapping includes a first desired number of storage units mapped to to a first one of the plurality of storage sites that is greater than a second desired number of storage units mapped to a second one of the plurality of storage sites; andallocating a plurality of storage units to the plurality of storage sites based on the storage unit apportioning data, wherein each of the ...

END-TO-END SECURE DATA STORAGE IN A DISPERSED STORAGE NETWORK

A method includes a first computing device encrypting a portion of a data matrix based on a set of encryption keys to produce an encrypted data matrix. The method further includes the first computing device sending the encrypted data matrix to a second computing device. The method further includes the second computing device dispersed storage error encoding the data matrix to produce a set of encrypted encoded data slices. The method further includes the second computing device sending the set of encrypted encoded data slices to a set of storage units of the DSN for storage therein. 1. A method comprises:encrypting, by a first computing device of a dispersed storage network (DSN), a portion of a data matrix based on a set of encryption keys to produce an encrypted data matrix, wherein the data matrix includes data blocks of a data segment of a data object;sending, by the first computing device, the encrypted data matrix and to a second computing device of the DSN;dispersed storage error encoding, by the second computing device, the encrypted data matrix to produce a set of encrypted encoded data slices; andsending, by the second computing device, the set of encrypted encoded data slices to a set of storage units of the DSN for storage therein.2. The method of further comprises:generating, by the first computing device, a set of temporary encryption keys as the set of encryption keys;obtaining, by the set of storage units, the set of temporary encryption keys, wherein a first storage unit of the set of storage units obtains a first temporary encryption key of the set of temporary encryption keys;decrypting, by the first storage unit, a first encrypted encoded data slice of the set of encrypted encoded data slices based on the first temporary encryption key to produce a first encoded data slice; andstoring, by the first storage unit, the first encoded data slice.3. The method of claim 2 , wherein the decrypting the first encrypted encoded data slice comprises: ...

CONCATENATING DATA OBJECTS FOR STORAGE IN A DISPERSED STORAGE NETWORK

A method includes identifying an independent data object of a plurality of independent data objects for retrieval from dispersed storage network (DSN) memory. The method further includes determining a mapping of the plurality of independent data objects into a data matrix, wherein the mapping is in accordance with the dispersed storage error encoding function. The method further includes identifying, based on the mapping, an encoded data slice of the set of encoded data slices corresponding to the independent data object. The method further includes sending a retrieval request to a storage unit of the DSN memory regarding the encoded data slice. When the encoded data slice is received, the method further includes decoding the encoding data slice in accordance with the dispersed storage error encoding function and the mapping to reproduce the independent data object. 1. A method for execution by a computing device of a dispersed storage network (DSN) , the method comprises:identifying an independent data object of a plurality of independent data objects for retrieval from DSN memory of the DSN, wherein the plurality of independent data objects is combined to produce a concatenated data object and wherein the concatenated data object is encoded in accordance with a dispersed storage error encoding function to produce a set of encoded data slices;identifying an encoded data slice of the set of encoded data slices corresponding to the independent data object based on a mapping of the plurality of independent data;sending a retrieval request to a storage unit of the DSN memory regarding the encoded data slice; andwhen the encoded data slice is received, decoding the encoded data slice in accordance with the dispersed storage error encoding function and the mapping to reproduce the independent data object.2. The method of wherein the mapping of the plurality of independent data objects includes a mapping of the plurality of independent data objects to data blocks in a ...

MEMORY

A memory device may operate in multiple modes. In a first mode, writes are not committed. In a second mode, writes are committed. 1. A method , comprising:instructing a media controller of a memory device to invalidate each memory region of a set of memory regions;instructing a set of redundancy controllers to include the memory device in a redundant set of memory devices;after instructing the set of redundancy controllers to include the media controller, instructing the media controller to enable writes.2. The method of claim 1 , further comprising:after instructing the media controller to enable writes, instructing a redundancy controller to begin a rebuild operation.3. The method of claim 1 , further comprising:prior to instructing the set of redundancy controller to include the memory device, instructing the media controller to ignore writes.4. The method of claim 1 , further comprising instructing the media controller to enable writes after a period of time subsequent to instructing a last redundancy controller to include the memory device claim 1 , the period of time being sufficient for any in-flight degraded mode operations to complete.5. A method claim 1 , comprising:tagging a set of memory regions as invalid;in a first mode of operation, receiving but not committing write commands for memory regions of the set of memory regions;transitioning to a second mode of operation; andin the second mode of operation, receiving and committing write commands for memory regions of the set of memory regions.6. The method of claim 5 , wherein the set of memory regions are a set of cache line sized memory regions.7. The method of claim 5 , further comprising tagging the set of memory regions as invalid by setting a poison bit associated with each element of the set of memory regions.8. The method of claim 7 , further comprising not committing write commands by not unsetting corresponding poison bits after writing data specified by the uncommitted write commands.9. The ...

IDENTIFYING ENCODED DATA SLICES FOR REBUILDING

A method for identifying encoded data slices for rebuilding includes determining, by a computing device of a dispersed storage network (DSN), a partial scanning approach based on an event, where the event is one of a plurality of possible events. When the event is a memory device issue, the method further includes selecting a first partial scanning approach that includes: sending a scan memory device request to the storage unit to scan the memory device for encoded data slices affected by the memory device issue, receiving a scan memory device response from the storage unit, and identifying the encoded data slices indicated in the scan memory device response for rebuilding. 1. A method for identifying encoded data slices for rebuilding , the method comprises:determining, by a computing device of a dispersed storage network (DSN), an event, wherein the event is one of a plurality of possible events;determining, by the computing device, a partial scanning approach based on the event; sending, by the computing device, a scan memory device request to the storage unit to scan the memory device for encoded data slices affected by the memory device issue, wherein the scan memory device request includes a request to list slice names of encoded data slices stored in the memory device;', 'receiving, by the computing device, a scan memory device response from the storage unit; and', 'identifying, by the computing device, the encoded data slices indicated in the scan memory device response for rebuilding., 'when the event is a memory device issue of a memory device of a storage unit of a set of storage units of the DSN, selecting, by the computing device, a first partial scanning approach that includes2. The method of further comprises: sending, by the computing device, a slice integrity scan request to the storage unit;', 'receiving, by the computing device, a slice integrity scan response from the storage unit; and', 'identifying, by the computing device, encoded data slices ...

UTILIZING REQUEST DEADLINES IN A DISPERSED STORAGE NETWORK

A method for execution by a dispersed storage and task (DST) processing unit includes generating a plurality of access requests that include an execution deadline time for transmission via a network to a corresponding subset of a plurality of storage units. A first deadline error notification is received via the network from a first storage unit of the first subset. A new one of the plurality of storage units not included in the first subset is selected in response to receiving the first deadline error notification. A new access request that includes an updated execution deadline time is generated for transmission to the new one of the plurality of storage units via the network. The new access request is based on a one of the first plurality of access requests sent to the first storage unit of the first subset. 1. A method for execution by a dispersed storage and task (DST) processing unit that includes a processor , the method comprises:generating a first plurality of access requests that indicate a first execution deadline time, the first plurality of access requests for transmission via a network to a corresponding first subset of a plurality of storage units;receiving a first deadline error notification via the network from a first storage unit of the first subset;calculating a missed deadline cost value in response to receiving the first deadline error notification;comparing the missed deadline cost value to a new request cost threshold; selecting a new one of the plurality of storage units not included in the first subset in response to receiving the first deadline error notification; and', 'generating a new access request for transmission to the new one of the plurality of storage units via the network that includes an updated execution deadline time, wherein the new access request is based on a one of the first plurality of access requests sent to the first storage unit of the first subset; and, 'when the missed deadline cost value compares favorably to the ...