PERFORMING HASH JOINS USING PARALLEL PROCESSING

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output. 1. A method of joining data records using a computer , the method comprising:identifying a first plurality of data records and a second plurality of data records;computing a hash for each data record in the first and second pluralities of data records;assigning data records of the first plurality of data records to groupings from a first set of groupings based on the computed hashes;assigning data records of the second plurality of records to groupings from a second set of groupings based on the computed hashes, each grouping in the second set of groupings associated with a respective grouping in the first set of groupings;determining, based on the hash values, whether to join respective data records of a grouping from the first set of groupings with respective data records of an associated grouping from the second set of groupings; andresponsive to determining to join respective data records of the grouping from the first set of groupings with respective data records of the associated grouping from the second set of groupings, joining the respective data records; andoutputting the joined data records.2. The method of claim 1 , wherein the data records comprise a plurality of fields having values and wherein computing a hash for each data record in the first and ...

PERFORMING HASH JOINS USING PARALLEL PROCESSING

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output. 1. A computer-implemented method comprising:dividing a plurality of data records into a set of groupings based on a hash value of each of the plurality of data records, each grouping in the set of groupings including a subset of the plurality of data records, each grouping associated with a set of bits that is a subset of bits of the hash value of each data record in the corresponding grouping;determining that a first grouping in the set of groupings is associated with a same set of bits as a second grouping in the set of groupings;assigning the first and second groupings to a worker thread;determining, by the worker thread, whether to join a data record in the first grouping with a data record in the second grouping;responsive to determining to join the data record in the first grouping with the data record in the second grouping, joining the two data records by the worker thread; andoutputting, by the worker thread, the joined data records.2. The computer-implemented method of claim 1 , wherein the plurality of data records are data records of a first data stream and a second data stream claim 1 , the data records in the first grouping are a subset of the data records of the first data stream claim 1 , and the data records in the second grouping are a subset of the ...

METHOD OF DATA AGGREGATION FOR CACHE OPTIMIZATION AND EFFICIENT PROCESSING

A data stream comprising a plurality of data records is retrieved. Portions of the data stream are aggregated to form a plurality of record packets of a predetermined size capacity. Each of the plurality of record packets comprises a number of data records from the plurality of data records. Further, the predetermined size capacity is an order of magnitude of a memory size of a cache memory associated with the data processing apparatus. Each of the plurality of record packets is transferred to respective ones of a plurality of threads associated with one or more processing operations. Each of the plurality of threads run independently on a respective processor from among a plurality of processors associated with the data processing apparatus. 1. A method performed by a data processing apparatus comprising:retrieving a data stream comprising a plurality of data records;aggregating the plurality of data records of the data stream to form a plurality of record packets of a predetermined size capacity, the predetermined size capacity determined responsive to a memory size of a cache memory associated with the data processing apparatus; andtransferring respective ones of the plurality of record packets to respective ones of a plurality of threads associated with one or more processing operations of the data processing apparatus.2. The method of claim 1 , wherein the one or more processing operations are associated with a data analytics workflow executing on the data processing apparatus.3. The method of claim 2 , further comprising:executing each of the one or more processing operations to perform a corresponding data analytics function on the plurality of record packets in a linear order, wherein the linear order is according to an operational sequence set in the data analytics workflow.4. The method of claim 3 , wherein executing each of the one or more processing operations comprises parallel processing performed by executing each respective thread on a respective ...

Method of data aggregation for cache optimization and efficient processing

A data stream comprising a plurality of data records is retrieved. Portions of the data stream are aggregated to form a plurality of record packets of a predetermined size capacity. Each of the plurality of record packets comprises a number of data records from the plurality of data records. Further, the predetermined size capacity is an order of magnitude of a memory size of a cache memory associated with the data processing apparatus. Each of the plurality of record packets is transferred to respective ones of a plurality of threads associated with one or more processing operations. Each of the plurality of threads run independently on a respective processor from among a plurality of processors associated with the data processing apparatus.

Asynchronously processing sequential data blocks

A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines whether the data block was correctly parsed. Responsive to determining that the data block was correctly parsed, the data analytics system provides the one or more data records contained within the data block to a data analytics workflow.

Asynchronously processing sequential data blocks

A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines whether the data block was correctly parsed. Responsive to determining that the data block was correctly parsed, the data analytics system provides the one or more data records contained within the data block to a data analytics workflow.

Method of data aggregation for cache optimization and efficient processing

A data stream comprising a plurality of data records is retrieved. Portions of the data stream are aggregated to form a plurality of record packets of a predetermined size capacity. Each of the plurality of record packets comprises a number of data records from the plurality of data records. Further, the predetermined size capacity is an order of magnitude of a memory size of a cache memory associated with the data processing apparatus. Each of the plurality of record packets is transferred to respective ones of a plurality of threads associated with one or more processing operations. Each of the plurality of threads run independently on a respective processor from among a plurality of processors associated with the data processing apparatus.

Asynchronously processing sequential data blocks

A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines whether the data block was correctly parsed. Responsive to determining that the data block was correctly parsed, the data analytics system provides the one or more data records contained within the data block to a data analytics workflow.

Performing hash joins using parallel processing

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output.

Performing hash joins using parallel processing

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output.

Asynchronously processing sequential data blocks

Memory allocation in a data analytics system

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer.

Memory allocation in a data analytics system

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer.

Memory allocation in a data analytics system

Memory allocation in a data analytics system

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer.

Memory allocation in a data analytics system

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer.

Memory allocation in a data analytics system

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer.