Data processing jobs often consist of large numbers of tasks, which stress different resources and often also fluctuate in their resource demands.
Therefore, a better resource utilization and lower runtimes are possible through an optimized resource allocation, scheduling, and placement of tasks.
Moreover, co-locating processing tasks with complementary resource demands in shared infrastructures can further increase the resource utilization and job throughput.
We, therefore, aim to answer the following questions for different data processing workloads with our research: What kind of resource should be allocated for a job and its tasks? Which job should be run next when resources become available? Where should a specific task be placed in a particular infrastructure? Should certain tasks be co-located onto shared resources?
To answer these questions, we use monitoring data, profiling runs, different performance models, as well as scoring and optimization methods.
We currently work on multiple topics in this area:
Renewable-Aware Edge/Fog/Cloud Computing, where we use co-simulation to identify and exploit synergies between distributed computing and distributed energy generation to improve the carbon footprints and resilience of applications.
Tarema: Adaptive Resource Allocation for Scalable Scientific Workflows in Heterogeneous Clusters. Jonathan Bader, Lauritz Thamsen, Svetlana Kulagina, Jonathan Will, Henning Meyerhenke, and Odej Kao. To appear in the Proceedings of the 2021 IEEE International Conference on Big Data (Big Data). IEEE. 2021. [Open Access]
Let’s Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud. Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Kordian Gontarska, and Lauritz Thamsen. To appear in the Proceedings of the 22nd International Middleware Conference (Middleware). ACM. 2021. [Open Access][code]
LOS: Local-Optimistic Scheduling of Periodic Model Training For Anomaly Detection on Sensor Data Streams in Meshed Edge Networks. Soeren Becker, Florian Schmidt, Lauritz Thamsen, Ana Juan Ferrer, and Odej Kao. To appear in the Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). IEEE. 2021.
LEAF: Simulating Large Energy-Aware Fog Computing Environments. Philipp Wiesner and Lauritz Thamsen. In the Proceedings of the 2021 IEEE 5th International Conference on Fog and Edge Computing (ICFEC). IEEE. 2021. [Open Access][video][code]
Mary, Hugo, and Hugo*: Learning to Schedule Distributed Data-Parallel Processing Jobs on Shared Clusters. Lauritz Thamsen, Jossekin Beilharz, Vinh Thuy Tran, Sasho Nedelkoski, and Odej Kao. In Concurrency and Computation: Practice and Experience (e5823). Wiley. 2020. [Open Access][code]
Hugo: A Cluster Scheduler that Efficiently Learns to Select Complementary Data-Parallel Jobs. Lauritz Thamsen, Ilya Verbitskiy, Sasho Nedelkoski, Vinh Thuy Tran, Vinícius Meyer, Miguel G. Xavier, Odej Kao, and César A. F. De Rose. In the Proceedings of the Euro-Par 2019 Workshops (Euro-Par). Presented at the 1st International Workshop on Parallel Programming Models in High-Performance Cloud. Springer. 2019. [Google Scholar][code]
Scheduling Stream Processing Tasks on Geo-Distributed Heterogeneous Resources. Gerrit Janßen, Ilya Verbitskiy, Thomas Renner, and Lauritz Thamsen. In the Proceedings of the 2018 IEEE International Conference on Big Data (IEEE BigData). Presented at the First International Workshop on the Internet of Things Data Analytics (IoTDA). IEEE. 2018. [Google Scholar]
Learning Efficient Co-locations for Scheduling Distributed Dataflows in Shared Clusters. Lauritz Thamsen, Ilya Verbitskiy, Benjamin Rabier, and Odej Kao. In Services Transactions on Big Data (Vol. 4, No. 1). Services Society. 2018. [Open Access]
Scheduling Recurring Distributed Dataflow Jobs Based on Resource Utilization and Interference. Lauritz Thamsen, Benjamin Rabier, Florian Schmidt, Thomas Renner, and Odej Kao. In the Proceedings of the 6th IEEE BigData Congress. IEEE. 2017. [Google Scholar][code]