This invention is a solution for handling large-volume real-time data stream processing tasks of unpredictable and diverse sizes. It will help support thousands of customers’ requirements to process streaming data in the digital marketing industry in a fair and timely manner. The invention will handle not only digital marketing data, but can also be used broadly across areas such as IoT data streaming, smart manufacturing, communication network, cloud computing, etc., to handle streaming data with huge and diverse sizes.