DEBS 2017June 19, 2017
The 11th ACM International Conference on Distributed and Event-Based Systems (DEBS 2017)
DEBS is an ACM International Conference, i.e. it is accredited by the Association for Computing Machinery (ACM) , one of the world leading associations of computing professionals. The DEBS conferences series is now in its 10th year and has regular attendance and contributions from world leading event processing experts from academia and industrial research. The DEBS Grand Challenge has always been considered as a highlight of the conference as participants need to demonstrate that their research solves actual real problems.
Slides are available here
At this year’s Distributed Events Processing Systems conference (DEBS 2017) AGT has demonstrated its capability to quantify the performance of anomaly detection algorithms in in real-time by using and extending the HOBBIT bench marking platform.
Dr. Roman Katerinenko and Dr. Martin Strohbach co-organized this year’s Grand Challenge in which participating teams were are asked to correctly detect anomalies in sensor data from injection molding machines provided by Weidmüller . By using the HOBBIT platform, we could for the first time objectively quantify and compare the performance of stream processing analytical systems by measuring the correctness of found anomalies, latency of the system (how long does it take to send a response), and throughput (how much data can be processed in a second). Bench marked systems had to process over 5GB of data that was sent to the systems under test at a rate of 250 data points second. In total participants had to detect more than 100 anomalies and deal with an unknown number of machines that provided measurements at unknown periods. In total data from more than 1000 machines had to be processed.
We implemented a data generator that creates realistic data of any size. The data generator has been carefully designed based on requirements from Weidmüller to hide business relevant aspects of the data and at the same time create measurements that allow to extrapolate the performance of the tested system in a realistic setting. The data generator is based on ML models (k-means and Markov Chains) learned from the actual Weidmüller data.
In total 14 teams registered to participate and 7 teams correctly identified the anomalies. The winning team from the Alexandru Ioan Cuza University, Romania had the best performing solution (average latency of 40ms) winning an award of $ 1000. A team of University Stuttgart, Germany won the audience award for the most elegant solution: they required only little optimization to their system and offered data scientist an elegant abstraction from the details of real-time message processing. There was also participation from a WSO2, a vendor of a commercial stream processing system.
AGT and the HOBBIT platform have been highly appreciated by the DEBS scientific community. The steering committee chair Prof. Hans-Arno Jacobson, from University of Toronto pointed out that using the HOBBIT benchmarking platform constitutes an innovation and major improvement in the way results of the DEBS Grand Challenge are evaluated. Participants praised the improvements that the HOBBIT platform brings to the world of event processing, highlighting for instance its capability to benchmark distributed systems. Associate Professor Annika Hinze from the University of Waikato, New Zealand and organizer of next year’s conference said that she “was impressend with the achievements of the Grand Challenge team in 2017” and “Using the Hobbit platform was very successful and your hard work in setting it up really paid off”. According to Prof. Holger Ziekow (University of Furtwangen) the quality of this year’s submissions was notably higher, which may be due to the objective quantifications performed by the HOBBIT platform.