Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine

Raphaël Barazzutti, Thomas Heinze, André Martin, Emanuel Onica, Pascal Felber, Christof Fetzer, Zbigniew Jerzak, Marcelo Pasin & Etienne Rivière

Abstract Publish/subscribe (pub/sub) infrastructures running as a service on cloud environments offer simplicity and flexibility for composing distributed applications. Provisioning them appropriately is however challenging. The amount of stored subscriptions and incoming publications varies over time, and the computational cost depends on the nature of the applications and in particular on the filtering operation they require (e.g., content-based vs. topic-based, encrypted vs. non-encrypted filtering). The ability to elastically adapt the amount of resources required to sustain given throughput and delay requirements is key to achieving cost-effectiveness for a pub/sub service running in a cloud environment. In this paper, we present the design and evaluation of an elastic content-based pub/sub system: E-STREAMHUB. Specific contributions of this paper include: (1) a mechanism for dynamic scaling, both out and in, of stateful and stateless pub/sub operators, (2) a local and global elasticity policy enforcer maintaining high system utilization and stable end-to-end latencies, and (3) an evaluation using real-world tick workload from the Frankfurt Stock Exchange and encrypted content-based filtering.
Citation R. Barazzutti, et al., "Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine," in ICDCS'14: The 34th International Conference on Distributed Computing Systems, Madrid, Spain, 2014, p. 567-576.
Type Conference paper (English)
Name of conference ICDCS'14: The 34th International Conference on Distributed Computing Systems (Madrid, Spain)
Date of conference 30-6-2014
Publisher IEEE
Pages 567-576
URL http://dx.doi.org/10.1109/ICDCS.2014.64
Related project SRT-15: Intelligence Push in the Enterprise Realm