Details of Grant 

EPSRC Reference:	EP/F035217/1
Title:	DISSP: Dependable Internet-Scale Stream Processing
Principal Investigator:	Pietzuch, Professor PR
Other Investigators:
Researcher Co-Investigators:
Project Partners:
Department:	Computing
Organisation:	Imperial College London
Scheme:	First Grant Scheme
Starts:	01 October 2008	Ends:	31 March 2012	Value (£):	293,569
EPSRC Research Topic Classifications:	Networks & Distributed Systems Software Engineering
EPSRC Industrial Sector Classifications:	Communications Information Technologies
Related Grants:
Panel History:	Panel Date Panel Name Outcome 24 Jan 2008 ICT Prioritisation Panel (Technology) Announced
Summary on Grant Application Form
Real-time stream data has begun to play an increasingly important role on the Internet. One of the causes for this is the proliferation of geographically-distributed stream data sources such as sensor networks, scientific instruments, pervasive computing environments and web feeds connected to the Internet. Potentially millions of users world-wide want to take advantage of the availability of this data. Therefore they require a convenient way to process real-time stream data at a global scale through applications that perform Internet-scale stream processing (ISSP). Analogous to how search engines make static web data useful to users, an ISSP system reliably collects, filters and processes stream data from potentially thousands of data sources on behalf of many users.The research focus of this proposal is to address a major challenge facing all ISSP applications: achieving robustness in the presence of failures. Failures are a fact of life in such systems because of their scale, heterogeneity and reliance on best-effort Internet infrastructure. This conflicts with the requirements of many users that demand a dependable ISSP (DISSP) service --- a service that should continue functioning even during Internet path outages, processing host failures and resource shortages. For example, consider a scientific study that detects and analyses transient events in the sky by correlating high-bandwidth, real-time image streams from dozens of radio telescopes world-wide. After the failure of the network connection to one of the telescopes, a scientist would expect the system to continue operating, perhaps with reduced resolution. Similarly, the failure of a host that processes the sky images should not cause a service interruption, although it may lead to a decrease in detection confidence of anomalies. Unfortunately, mechanisms for building DISSP systems are lacking. Conventional techniques for reliable data processing cannot be applied to a DISSP system because of its requirements of global scalability, of short recovery times in a real-time setting and of resource efficiency due to a shared infrastructure.This proposal addresses these challenges so that ISSP systems needed for interconnecting tomorrow's pervasive sensor systems and global scientific experiments can become a reality. We intend to develop new techniques for building dependable ISSP systems, taking their unique features in terms of scale, failure model and data quality into account. In particular, we will devise approaches that gracefully degrade result quality in response to resource shortage after failure. While result quality is reduced, the system will provide constant feedback to users on the achieved level of service. Feedback will be expressed in a domain-specific way, e.g., by notifying a scientific user about the reduction in detection confidence of events of interest. This feedback will also drive an adaptive fault-tolerance mechanism, allowing the DISSP system to strategise about resource allocation in order to minimise the reduction in service quality of a maximum number of users.
Key Findings
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Potential use in non-academic contexts
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Impacts
Description This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk Summary Date Materialised
Sectors submitted by the Researcher
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Project URL:
Further Information:
Organisation Website:	http://www.imperial.ac.uk