Dynamically modifying the parallelism of a task in a pipeline

Processing data sets in a big data repository

The invention provides for a method for processing a plurality of data sets (105; 106; 108; 110-113; DB1; DB2) in a data repository (104) for storing at least unstructured data, the method comprising: - providing (302) a set of agents (150-168), each agent being operable to trigger the processing of one or more of the data sets, the exe- cution of each of said agents being automatically triggered in case one or more conditions assigned to said agent are met, at least one of the conditions relating to the existence, structure, content and/or annotations of the data set whose processing can be triggered by said agent; - executing (304) a first one of the agents; - updating (306) the annotations (115) of the first data set by the first agent; and - executing (308) a second one of the agents, said execution being triggered by the updated annotations of the first data set meeting the conditions of the second agent, thereby triggering a further up- dating of the annotations of the first data ...

Dynamically modifying parallelism of task in pipeline

In an approach to dynamically identifying and modifying the parallelism of a particular task in a pipeline, the optimal execution time of each stage in a dynamic pipeline is calculated. The actual execution time of each stage in the dynamic pipeline is measured. Whether the actual time of completion of the data processing job will exceed a threshold is determined. If it is determined that the actual time of completion of the data processing job will exceed the threshold, then additional instances of the stages are created.