| EPSRC Reference: |
EP/N000110/1 |
| Title: |
Unified Stream and Transaction Processing |
| Principal Investigator: |
Migliavacca, Dr M |
| Other Investigators: |
|
| Researcher Co-Investigators: |
|
| Project Partners: |
|
| Department: |
Sch of Computing |
| Organisation: |
University of Kent |
| Scheme: |
First Grant - Revised 2009 |
| Starts: |
01 January 2016 |
Ends: |
31 December 2017 |
Value (£): |
95,456
|
| EPSRC Research Topic Classifications: |
| Fundamentals of Computing |
Software Engineering |
|
| EPSRC Industrial Sector Classifications: |
|
| Related Grants: |
|
| Panel History: |
| Panel Date | Panel Name | Outcome |
|
14 Apr 2015
|
EPSRC ICT Prioritisation Panel - Apr 2015
|
Announced
|
|
|
Summary on Grant Application Form |
The diversity and complexity data-processing architectures, as used in current large scale e-commerce and social media applications, hinder the integration of new data analytics features and increase maintenance and operational costs. These applications typically span three different layers, each one adopting a different processing model. Online transaction processing (OLTP) systems are first used to access core application data in a synchronous fashion. In parallel, events of significance are propagated to back-ends for data analysis. There, events are stored to persistent storage for offline batch processing, e.g., using Hadoop clusters. Increasingly application providers are also deploying stream processing solutions to process data in near real-time and improve the freshness of data analysis. Results from batch or real-time analysis are then propagated back to serving layers in online front-end systems for applications that require them.
The fragmentation of application logic across the transaction, stream and batch processing models increases operational costs and application development time. Substantial engineering effort is required to export, move, convert, and import data across different systems when performing exploratory analytics for developing new applications. Developers do not have a reliable way of mapping application features on a specific processing model, re- lying on past experience and rule of thumb. Reimplementing complex processing pipelines in a system that adopts a different way of thinking is costly and the performance trade-offs among them are difficult to estimate.
The operation of multiple systems in parallel leads to increased operational costs. Operating different platforms either requires allocating computing resources statically to the different systems, which is not flexible, or to use cluster management platforms
such as Mesos or YARN. However to accommodate diverse systems these platforms necessarily provide a restricted management interface which limits their ability to operate on the basis of detailed application metrics.
The goal of this proposal is to unify two of these models, stream and transaction process- ing by i) exploring performance trade-offs between the two models, ii) designing a processing model that sup- ports both, iii) implementing a system prototype which could provide good performance on both transactional and stream processing workloads, and iv) prototyping mixed processing applications which requires both stream and transaction processing functionalities.
|
|
Key Findings |
|
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
|
|
Potential use in non-academic contexts |
|
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
|
|
Impacts |
|
| Description |
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk |
| Summary |
|
| Date Materialised |
|
|
|
|
Sectors submitted by the Researcher |
|
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
|
| Project URL: |
|
| Further Information: |
|
| Organisation Website: |
http://www.kent.ac.uk |