Login
StreamHub: A Massively Parallel Architecture for High-Performance Content-Based Publish/Subscribe
Résumé By routing messages based on their content, publish/subscribe (pub/sub) systems remove the need to establish and maintain fixed communication channels. Pub/sub is a natural candidate for designing large-scale systems, composed of applications running in different domains and communicating via middleware solutions deployed on a public cloud. Such pub/sub systems must provide high throughput, filtering thousands of publications per second matched against hundreds of thousands of registered subscriptions with low and predictable delays, and must scale horizontally and vertically. As large-scale application composition may require complex publications and subscriptions representations, pub/sub system designs should not rely on the specific characteristics of a particular filtering scheme for implementing scalability.

In this paper, we depart from the use of broker overlays, where each server must support the whole range of operations of a pub/sub service, as well as overlay management and routing functionality. We propose instead a novel and pragmatic tiered approach to obtain high-throughput and scalable pub/sub for clusters and cloud deployments. We separate the three operations involved in pub/sub and leverage their natural potential for parallelization. Our design, named StreamHub, is oblivious to the semantics of subscriptions and publications. It can support any type and number of filtering operations implemented by independent libraries. Experiments on a cluster with up to 384 cores indicate that StreamHub is able to register 150 K subscriptions per second and filter next to 2 K publications against 100 K stored subscriptions, resulting in nearly 400 K notifications sent per second. Comparisons against a broker overlay solution shows an improvement of two orders of magnitude in throughput when using the same number of cores.
   
Mots-clés
   
Citation R. Barazzutti, et al., "StreamHub: A Massively Parallel Architecture for High-Performance Content-Based Publish/Subscribe," in DEBS'13: The 7th ACM International Conference on Distributed Event-Based Systems, Arlington, TX, 2013.
   
Type Actes de congrès (Anglais)
Nom de la conférence DEBS'13: The 7th ACM International Conference on Distributed Event-Based Systems (Arlington, TX)
Date de la conférence 29-6-2013
Editeur commercial ACM
Liée au projet SRT-15: Intelligence Push in the Enterprise Realm