Partial fault tolerance in stream processing applications - methods and evaluation techniques

Jacques da Silva, Gabriela

Permalink

https://hdl.handle.net/2142/18383 Copy

Description

Title

Partial fault tolerance in stream processing applications - methods and evaluation techniques

Author(s)

Jacques da Silva, Gabriela

Issue Date

2011-01-14T22:48:28Z

Director of Research (if dissertation) or Advisor (if thesis)

Iyer, Ravishankar K.

Doctoral Committee Chair(s)

Iyer, Ravishankar K.

Committee Member(s)

• Lumetta, Steven S.
• Nahrstedt, Klara
• Sanders, William H.
• Vasudevan, Shobha

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• fault tolerance
• fault injection
• stream processing
• model-based evaluation
• high-availability

Abstract

Stream processing emerged as a paradigm to continuously process incoming live data streams, such as audio, video, and business feeds. These applications are assembled as dataflow graphs, where each vertex of the graph is a stream operator and each edge is a stream connection. In this environment, a fault in a stream operator can result in massive data loss or in the generation of inaccurate results. Most of the fault tolerance solutions proposed for streaming applications aim at guaranteeing that no data is lost or that no data item is delivered to the application more than once. These techniques result in high performance overhead, given the need to coordinate the state stored in checkpoints of distributed components or maintain consistency between replicas. In this dissertation, we investigate partial fault tolerance methods, which protect only the most critical stream operators of a streaming application. These methods take advantage of the fact that stream processing algorithms are approximate by nature and, as a result, can still achieve acceptable results under data loss and duplicate data delivery. The methods proposed in this dissertation include a checkpoint-based mechanism and a partial graph replication technique. Both techniques were implemented in System S, IBM Research's stream processing middleware. In addition, this dissertation describes two different fault tolerance evaluation techniques. The first technique is based on fault injection and is used to emulate the effects of partial fault tolerance on a streaming application. With the fault injection results, the developers can understand the impact of faults on the application output and identify the most critical operators on their streaming application. The second evaluation technique is a model-based framework which provides generic abstractions for representing streaming applications with the stochastic activity network formalism. The framework allows the comparison of different fault tolerance techniques under varying fault models. Based on the results, the developers can evaluate the trade-offs that a certain technique provides when applied to their target application.

Graduation Semester

2010-12

Permalink

http://hdl.handle.net/2142/18383

Copyright and License Information

Copyright 2010 Gabriela Jacques da Silva

Owning Collections