Data storage method and system

04-08-2010 дата публикации

Номер:

CN0101795211A

Автор: JIA LIN, LI YONGLI, LU KAIZHI, MA LIBIN, YANG HONGTAO, ZHANG CHANGHUA, LIN JIA, YONGLI LI, KAIZHI LU, LIBIN MA, HONGTAO YANG, CHANGHUA ZHANG

Принадлежит: Beijing Zhongchuang Telecom Test Co Ltd

Контакты:

Номер заявки: 03-10-20104069

Дата заявки: 13-01-2010

[1]

Technical Field

[2]

The invention relates to the field of data processing, more specifically, relates to a method and system for storage of the data.

[3]

Background Art

[4]

The current telecommunication network monitoring system, is facing magnanimity network signaling data, these need to be stored, the service data of the order of magnitude up to TB per day. these service data in preserves in the process, usually uses to increase, to delete, changes, Zha Zhong increases, looks up, deletes three kind of visits, has not revised the data nearly the demand, deletes also merely needs to delete the obsolete data forward according to the time. That is to say, data for the business, the vast majority of operation in order to increase and inquiry two. Furthermore, with the passage of time, the continuous development of the telecommunications service, traffic growth has also resulted in ever-increasing data, storage system also must be can be thereupon expanded,

[5]

In the prior art, for such mass data, memory need to be distributed, using a database generally is through distributed data storage (cloud) after slicing the data according to a certain algorithm on scattered storage of the network, in order to meet the demand of the space and performance. Query characteristics usually according to the business or further accurate to a certain a certain business key word.

[6]

However, because the generic distributed data storage (cloud) after the slicing of data will generally be according to a certain algorithm on scattered storage of the network, so the mass data storage, retrieval and mobile, may occupy a large number of network bandwidth. The network communication request will be very high, it is difficult to highly efficient positioning and extraction. In the prior art so that the technical scheme of the memory speed is slow, the large amount of data is very difficult to satisfy the fast writing, at the same time also needs high purchase cost and maintenance cost.

[7]

Content of the invention

[8]

In view of this, the embodiment of the invention provides a method and system of storing data, in order to realize the convenient and efficient access data.

[9]

In order to realize the above-mentioned purpose, the embodiment of the invention provides the following technical scheme:

[10]

A method of storing data, including:

[11]

After receiving the data storage server, according to the data type and data time of the data information to determine the memory contents; to the data stored in the form of data files, the data file is included in the file name of the time information;

[12]

Storage server after receiving the query request, the query according to the request categories of data and time information inquiry to the inquiry data file and the data file in the list in a file;

[13]

According to the resource or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file.

[14]

Preferred, the embodiment of the invention, further comprises:

[15]

Is used for storing the data of the storage server is comprised of a plurality of disk drives in a disk array, pre-to the disk array logical partition;

[16]

The storage server receives the data, for each kind of the data starting a process, to the server when starting the process for storing data in each of the logical partition creating thread; corresponding to the data storage according to the logical partition, the time information of the data type determination of data and data storage contents; to the data stored in the form of data files, the data file is included in the file name of the time information.

[17]

Preferred, the embodiment of the invention, further comprises:

[18]

Before storing data, the policy server, the data needs to be stored according to the set policy distributed to the storage server;

[19]

Storage server before the data query, a load equalizer receive the inquiry request, the load of the server according to the interface used for the implementation of the query task interface server;

[20]

The implementation of the interface of the query task server, according to the kind of the data query request on the policy server for the types of data stored in a storage server, and to the storage server sends the query request.

[21]

Preferred, the embodiment of the invention, further comprises:

[22]

After the logical partitions, each one of the logical partition in the set up according to the set rules file contents;

[23]

The division of the data file to rules established, the stored data the magnitude of the file within the set range;

[24]

When storing data, the storage server before storing data, first data and analysis according to the rules of multi-stage contents of the determined storage contents;

[25]

After setting time the storage data of the index file is set up;

[26]

Storage server query data adopts a segmented multi-thread concurrent technology to scan the path of index and data file.

[27]

Preferred, the embodiment of the invention, the set-up index for the establishment of a multi-level index.

[28]

Preferred, the embodiment of the invention, also includes the steps of: after the index is set up, timing update index.

[29]

Preferred, the embodiment of the invention, the rules are established by setting the multi-level file directory:

[30]

Establishing a level of contents, a level of contents in the delay data includes normal data and two file directory;

[31]

Normal data contents for storage in the setting time is able to reach within the range of data of the storage system; delay data contents is used for storing the system exceeds the set time of the data;

[32]

In each level of the building in the catalog is the same as secondary contents; secondary contents for various business document contents, data classified according to the class of service, each service corresponding to the data stored in the second directory;

[33]

Each secondary directory in the same establishment of the three-level contents; monthly three-level directory is divided into 12 document contents, three levels of contents for the storage of the data in accordance with stored data;

[34]

When storing data, according to the multi-stage contents of the determined storage contents to set up the rule:

[35]

Compared with the current time, to determine is the need to store in one of the normal data contents is also delay data contents; judge according to the types of business needs to be stored in the service directory;

[36]

According to the current month in which month judges the contents;

[37]

According to pre-established rules of the determined divided into divided documents file name, write data to the file;

[38]

The creation of the index in particular to:

[39]

Index includes index corresponding to the data file, the plurality of index file for the SUMMARY rollup index and multi-level index.

[40]

Preferred, the embodiment of the invention, also includes the steps of:

[41]

According to the data storage requirements define various service time necessary to retain, when the disk space is lower than the set value of the configuration of the data file to delete more than retention time in units of the data.

[42]

Preferred, the embodiment of the invention, also includes the steps of: when the existing disk space can not meet the memory requirements increase the disk partition.

[43]

A data storage system, including,

[44]

The logic structure is equipped with:

[45]

For storing business data storage server;

[46]

For the data according to the set policy distribution policy server to the memory of the server;

[47]

Query task for the implementation of the interface server;

[48]

For the interface server balance the load of the load equalizer;

[49]

In the memory in a server on the basis of a plurality of disk drives needs to be logically partitioned;

[50]

Each storage server is the service data needs to be stored for starting a process, process initiation will, according to the server is used for storing data of the number of disk partition creation of a number of thread; each thread based on the corresponding disk partition and service determines that the memory contents, each thread sharing through data distribution technology is the business data processing;

[51]

Load equalizer query request is received, determining interface server; inquiry request to the interface server send an inquiry request;

[52]

Interface server according to the service of the request on the policy server for the service data stored in a storage server, and to the storage server sends the query request;

[53]

The storage server receives the query request inquiry document placed in a the same and will be in the list of files, according to the resources or configuration creation plurality of reading thread, the thread of each read from a queue sequentially extracting file, reading the data in the file.

[54]

In the embodiment of the invention, through the in the disk array divided into a plurality of logical partitions, and the use of multi-thread on different partition for read-write, thereby improving the utilization of the bandwidth IO. IO is usually because bottlenecks of storage, therefore, improves the bandwidth IO, the utilization of the storage server to improve the data throughput capacity. And, in the embodiment of the invention, in a data file in the form of data storage and retrieval, storage and transmission of the data is of the same format, format conversion is not required, so saving the data conversion step, the access efficiency of the the data is improved. Moreover, the embodiment of the invention, is issued by the query conditions to the storage server, from the storage server to perform query task obtaining the required data file, network data file required for transmission only, so as to reduce the data transmission quantity, improve the efficiency. In the prior art is avoided, when the cloud computing system then all the data is read by the query filtering scheme, thus causing a large amount of data needs to be transmitted by the network congestion problem.

[55]

Description of drawings

[56]

In order to more clearly illustrate the embodiment of the invention or a technical proposal in the prior art, will be to the embodiment or the prior art to be used in the description for the simple introduction of the Figure, it is obvious that, in the description below only the Figure is some embodiments of the present invention, for one of ordinary skill in the art is concerned, on the premise of a creative work, can also be obtained according to these with other Figures.

[57]

Figure 1 is the step flow chart of embodiment the method of the present invention;

[58]

Figure 2 is flow chart of the steps of the invention another embodiment the method;

[59]

Figure 3 is the step flow chart of the present invention another embodiment the method;

[60]

Figure 4 is a schematic diagram of an embodiment the system of the present invention.

[61]

Mode of execution

[62]

Will be combined with the embodiment of the invention the Figure, in this embodiment of the invention a clear the technical scheme, complete described, obviously, the embodiment described is only a portion of the embodiment of this invention, , but not all, of the embodiment. Based on the embodiment of the invention, one of ordinary skill in the art without making creative work is obtained on the premise that all of the other embodiments, all belong to the scope of protection of this invention.

[63]

The invention discloses a method of storing data, in order to realize high-efficient storage data of the high efficiency of the query data and, as shown in Figure 1, the specific steps are as follows:

[64]

S11, for storing the data of the memory in the server on the basis of a plurality of disk drives needs to be logically partitioned;

[65]

In the practical application, usually according to the amount of data needs to be stored and the processing capacity of the server, to dispose a plurality of disk drive, through the formation of disk array and on the disk array logical partition to improve the data read/write speed. Moreover, the use of a plurality of disk drives on the disk array of the logical partition may limit the capacity of a single magnetic disk, can greatly enhance the data capacity of a storage server.

[66]

S12, store data, in the storage server after receiving the data, the data for each kind of starting a process, to the server when starting the process for storing data in each of the logical partition creating thread; each thread corresponding to the storage data according to logical partition and data types for the data storage contents; to the data stored in the form of data files, the data file is included in the file name of the time information;

[67]

In the embodiment of the invention, by dividing a plurality of logical partitions, and the use of multi-thread on different logical partitions of the read-write, thereby improving the utilization of the bandwidth IO. IO is usually because bottlenecks of storage, therefore, improves the bandwidth IO, the utilization of the storage server to improve the data throughput capacity. In the prior art memory technologies, such as the database, in the data storage and transmission of different data formats (coding), the stored received data format conversion needs to be done, is returned in the query to the data format conversion is required. In the embodiment of the invention, mode of storing the document, storage and transmission of the data is of the same format, format conversion is not required, so saving the data conversion step, the access efficiency of the the data is improved.

[68]

Furthermore, a time information through the in order to include the name of the name of the data file can be in the query data with time as the inquiry condition, convenient and efficient query the required data.

[69]

S13, the data query, to the query request to the storage server, according to the file name of the data file with the time information included in the query to the query data file and the data file in the list in a file, according to the resources or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file.

[70]

In the embodiment of the invention, is issued by the query conditions to the storage server, storage server executes the query task data file and acquire the needed, network data file required for transmission only, so as to reduce the data transmission quantity, improve the efficiency. In the prior art is avoided, when the cloud computing system then all the data is read by the query filtering scheme, thus causing a large amount of data needs to be transmitted by the network congestion problem.

[71]

In another embodiment, as shown in Figure 2, can also include the following steps:

[72]

S21, in order to be used for storing data in the storage server on the basis of a plurality of disk drives needs to be logically partitioned;

[73]

[74]

S22, data storage the front, using strategy server will need to store data according to the set policy distributed to the storage server;

[75]

In the embodiment of the invention, in the data storage to the storage server the former, the use of policy server of the data needs to be stored according to the set policy distributed to the storage server; the stated setting strategy usually in the storage and processing capacity of the server according to the proportional distribution, this can ensure that all the storage of data in storage of a balanced load of the server.

[76]

S23, store data, in the storage server after receiving the data, the data for each kind of starting a process, to the server when starting the process for storing data in each of the logical partition creating thread; each thread corresponding to the storage data according to logical partition and data types for the data storage contents; to the data stored in the form of data files, the data file is included in the file name of the time information;

[77]

[78]

S24, before the data query, using the load equalizer receive the inquiry request, the load of the server according to the interface used for the implementation of the interface of the query task server, the balanced so as to play the role of the interface server load.

[79]

S25, the implementation of the interface of the query task server, according to the kind of the data query request on the policy server for the types of data stored in a storage server, and to the storage server sends the query request.

[80]

S26, the data query, to the query request to the storage server, according to the file name of the data file with the time information included in the query to the query data file and the data file in the list in a file, according to the resources or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file. In the embodiment of the invention, is issued by the query conditions to the storage server, from the storage server to perform query task data file and acquire the needed, network data file required for transmission only, so as to reduce the data transmission quantity, improve the efficiency. In the prior art is avoided, when the cloud computing system then all the data is read by the query filtering scheme, thus causing a large amount of data needs to be transmitted by the network congestion problem.

[81]

Furthermore, by adopting the multi-thread reading data, improves the storage server to the reading speed of the data, to improve the working efficiency of the storage server.

[82]

In another embodiment of the invention, as shown in Figure 3, can also include the following steps:

[83]

S31, in order to be used for storing data in the storage server on the basis of a plurality of disk drives needs to be logically partitioned;

[84]

[85]

S32, in each logic subregion according to the rules set up a multi-level file directory;

[86]

The set up according to the set rules for file directory:

[87]

Establishing a level of contents, a level of contents in the delay data includes normal data and two file directory;

[88]

[89]

[90]

Each secondary directory in the same establishment of the three-level contents; monthly three-level directory is divided into 12 document contents, three levels of contents for the data in accordance with stored data storage of the month.

[91]

Rules are established according to the set of the directory, the storage data may be stored according to the set classification rules, data storage thus able to establish a corresponding index, and further inquiry of the data. In the embodiment of the invention, the data storage-contents, thereby ensuring that the various contents of the document would not be too much, the efficiency of the file retrieval is ensured. For the mobile or at the same time provides a convenient backup data, when we need some kind of data, only needs to copy a directory instead of many documents in the search.

[92]

S33, the establishment of the rules of the division of the data file, the size of the data file within the set range; according to the data amount can be hourly, daily, weekly, monthly, etc.-dividing mode, one data file will not be excessive. Also can not result in a large amount of small documents, too many small document query efficiency will be affected.

[93]

In the prior art memory technologies, such as the database, in the data storage and transmission of different data formats (coding), the stored received data format conversion needs to be done, is returned in the query to the data format conversion is required. In the embodiment of the invention, mode of storing the document, storage and transmission of the data is of the same format, format conversion is not required, so saving the data conversion step, the access efficiency of the the data is improved.

[94]

S34, data storage the front, using strategy server will need to store data according to the set policy distributed to the storage server;

[95]

[96]

S35, when storing data, the storage server receives the data needs to be stored, for each kind of the data starting a process, to the server when starting the process for storing data in each of the logical partition creating thread; a thread according to the rules of multi-stage contents of the determined storage contents; to the data stored in the form of data files, the data file is included in the file name of the time information;

[97]

According to the multi-stage contents of the determined storage contents to set up the rule:

[98]

[99]

According to the current month in which month judges the contents;

[100]

According to pre-established rules of the determined divided into divided documents file name, the data writing to a file.

[101]

The contents of the rules according to the set of stored data, thus facilitating data query. In the embodiment of the invention, the data storage-contents, thereby ensuring that the various contents of the document would not be too much, the efficiency of the file retrieval is ensured. For the mobile or at the same time provides a convenient backup data, when we need some kind of data, only needs to copy a directory instead of many documents in the search.

[102]

S36, after the set time, the index file for the data file is set up; the index file includes the index of the data files corresponding to, the plurality of index file for the SUMMARY rollup index and multi-level index.

[103]

The set-up index for the establishment of a multi-level index. The index file is set up, timing updates index file.

[104]

Through setting up the index, so that when in the query data file, the number of the scanning document is reduced, improving the performance of the query document, and the multi-level data index further reduce the number of the scanning document, reduce the number of times of the open-close document, improves the efficiency of data query.

[105]

S37, the former data query, using the load equalizer receive the inquiry request, the load of the server according to the interface used for the implementation of the query task interface server;

[106]

S38, the implementation of the interface of the query task server, according to the kind of the data query request on the policy server for the types of data stored in a storage server, and to the storage server sends the query request.

[107]

S39, the data query the storage server, the first segmented multi-thread concurrent technology to scan the path of index and data file. If there is no index file, according to the file name of the data file with the time information included in the query data file.

[108]

In the embodiment of the invention, the process of data inquiries will scan index, if the index file is created, and the inquiry request is associated with the query conditions, the index file can be used. Check the index is divided into multi-stage, check the rollup index, if a rollup index priority in using the rollup index, if aggregation index score index or the old data file, by rollup index, but by scanning data file to query.

[109]

S40, the storage server will query the data file and the in in a file list, according to the resources or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file. In the embodiment of the invention, is issued by the query conditions to the storage server, storage server executes query task, after obtaining the required data file, network data file required for transmission only, so as to reduce the data transmission quantity, improve the efficiency. In the prior art is avoided, when the cloud computing system then all the data is read by the query filtering scheme, thus causing a large amount of data needs to be transmitted by the network congestion problem.

[110]

Furthermore, by adopting the multi-thread reading data, improves the storage server to the reading speed of the data, to improve the working efficiency of the storage server.

[111]

In another embodiment, all of the above embodiment, also includes the steps of:on the basis of the data storage requirements, define various operational needs and the retention time of the free space of the logical partition in the threshold, the free space of the logical partition is lower than the set threshold value, the data of the deleted more than retention time;

[112]

As in the embodiment of the invention, data is stored for the unit in the document, therefore, deleted, in order to document the data for the unit can be deleted.

[113]

In another embodiment, all of the above embodiment, also includes:

[114]

When the existing disk space can not satisfy the logical partitions increase the memory requirements.

[115]

In the embodiment of the invention, when the store on the server the existing space is not sufficient to meet the memory requirements, the logical partitions, a method to realize the expansion of the storage space, when adding new logical partitions, each logical partition because the amount of data received is balanced, the amount of the new logical partitions will be relatively small, the amount of data of the old logical partitions will be relatively large, but as long as the increase in time of the logical partitions, the old logical partitions from sharing the additional data, the data amount of the newly written data to be deleted is the maintenance, it will not cause the old logical partition to the insufficient space and the problem of data loss, with the passage of time, each logical partition data quantity reaches equilibrium.

[116]

Furthermore, as shown in Figure 4, the embodiment of the invention also provides a system for storing data, including, is used for storing the data in a storage server 1, used for strategy will need to be stored according to the set data distribution of the policy server to the memory of the server 2, for execution of the query task interface server 3 and is used for the load of the server according to the interface used for the implementation of the interface of the query task equalizer the load of the server 4;

[117]

First of all, for storing the data in a storage server 1 a plurality of disk drive in the logical partitions according to needs.

[118]

The embodiment of the invention by dividing a plurality of logical partitions, and the use of multi-thread on different logical partitions of the read-write, thereby improving the utilization of the bandwidth IO. IO is usually because bottlenecks of storage, therefore, improves the bandwidth IO, the utilization of the storage server to improve the data throughput capacity.

[119]

Store data, using the policy server 2 will need to store data according to the set policy distributed to the storage server 1 ; in the storage server 1 after receiving the data, the data for each kind of starting a process, to the server when starting the process for storing data in each of the logical partition creating thread; each thread corresponding to the storage data according to logical partition and data types for the data of the memory contents and storing data, each thread sharing through data distribution technology is the business data processing;

[120]

The data query, using the load balancer 4 receive the inquiry request, according to the interface server 3 for determining the load situation of the implementation of the query task interface server 3;

[121]

The implementation of the query task interface service 3 device according to the kind of the data query request to the policy server 2 query on the type data of the storage server 1, and to the storage server 1 sends the query request.

[122]

Storage server 1 to the inquiry request after the inquiry data file and in the list in a file, according to the resources or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file.

[123]

The embodiment of the invention in the query data file of the data file to the segmented multi-thread concurrent technology, this kind of multi-thread concurrent technology the query efficiency can be greatly improved. Moreover, the embodiment of the invention from the file will be read data according to the query conditions in the server filter, filter and then is returned to the interface server, network transmission quantity is greatly reduced.

[124]

In this specification each embodiment described by adopting a progressive manner, each embodiment are highlighted with the other embodiment of the difference, between each embodiment is the same as the similar parts refer to each other.

[125]

For the disclosed embodiment of the above-mentioned description, the field of the professional technical personnel can realize or use of the invention. For these embodiments the multi-kind of modification of the professional technical personnel in this field will be obvious, defined in this text the general principle can be without departing from the spirit of this invention or under the situation of range, is realized in other embodiments. Therefore, the invention will not be limited to the invention as shown in these embodiments, but rather to comply with the principle disclosed herein is consistent and novel characteristics of the most wide range.

[1]

The embodiment of the invention discloses a stored data method and a system. In the invention, a disk array is divided into a plurality of logic subareas, and multiple threads are used for respectively reading and writing different subareas, thereby enhancing the utilization of IO bandwidth. Since IO is usually the bottleneck of storage, so the enhancement of the utilization of IO bandwidth enhances the data throughput of the storage server. In addition, in the embodiment of the invention, data is stored and retrieved in the form of data files; and both the storage and the transmission of the data use the same format and do not need format conversion, thereby saving the step of data conversion and enhancing the data access efficiency. Furthermore, in the embodiment of the invention, by transmitting the query conditions to a storage server, the storage server executes the query task and acquires the required data, and the network only transmits the required data, thereby reducing the data transmission quantity and enhancing the efficiency.

1. A method of storing data, characterized in that comprises:

Storage server after receiving the query request, the query according to the request categories of data and time information inquiry to the inquiry data file and the data file in the list in a file;

According to the resource or configuration creation plurality of reading thread, in order to extract the data file from the list, the data in the read data file.

2. The method according to Claim 1, characterized in that comprises:

Is used for storing the data of the storage server is comprised of a plurality of disk drives in a disk array, pre-to the disk array logical partition;

3. The method according to Claim 1 or 2, characterized in that it also comprises:

Before storing data, the policy server, the data needs to be stored according to the set policy distributed to the storage server;

Storage server before the data query, a load equalizer receive the inquiry request, the load of the server according to the interface used for the implementation of the query task interface server;

4. The method according to Claim 3, characterized in that it also comprises:

After the logical partitions, each one of the logical partition in the set up according to the set rules file contents;

The division of the data file to rules established, the stored data the magnitude of the file within the set range;

When storing data, the storage server before storing data, first data and analysis according to the rules of multi-stage contents of the determined storage contents;

After setting time the storage data of the index file is set up;

Storage server query data adopts a segmented multi-thread concurrent technology to scan the path of index and data file.

5. The method according to Claim 4, characterized in that the set-up index for the establishment of a multi-level index.

6. The method according to Claim 5, characterized in that also comprises the steps of: after the index is set up, timing update index.

7. The method according to Claim 6, characterized in that the set up according to the set rules for file directory:

Establishing a level of contents, a level of contents in the delay data includes normal data and two file directory;

When storing data, according to the multi-stage contents of the determined storage contents to set up the rule:

Compared with the current time, to determine is the need to store in one of the normal data contents is also delay data contents;

According to the service type judgment needs to be stored in the service directory;

According to the current month in which month judges the contents;

According to pre-established rules of the determined divided into divided documents file name, write data to the file;

The creation of the index in particular to:

Index includes index corresponding to the data file, the plurality of index file for the SUMMARY rollup index and multi-level index.

8. The method according to Claim 7, characterized in that also comprises the steps of:

9. The method according to Claim 8, characterized in that also comprises the steps of: when the existing disk space can not meet the memory requirements increase the disk partition.

10. A data storage system, characterized in that comprises:

The logic structure is equipped with:

For storing business data storage server;

For the data according to the set policy distribution policy server to the memory of the server;

Query task for the implementation of the interface server;

For the interface server balance the load of the load equalizer;

In the memory in a server on the basis of a plurality of disk drives needs to be logically partitioned;

Load equalizer query request is received, determining interface server; inquiry request to the interface server send an inquiry request;

Interface server according to the service of the request on the policy server for the service data stored in a storage server, and to the storage server sends the query request;

IPC - классификация

H H0 H04 H04L H04L1 H04L12 H04L12/H04L12/2 H04L12/24

Получить PDF