The scalability of reliable computation in Erlang

Ghaffari, Amir (2015) The scalability of reliable computation in Erlang. PhD thesis, University of Glasgow.

Abstract

With the advent of many-core architectures, scalability is a key property for programming
languages. Actor-based frameworks like Erlang are fundamentally scalable, but in practice
they have some scalability limitations.

The RELEASE project aims to scale the Erlang's radical concurrency-oriented programming
paradigm to build reliable general-purpose software, such as server-based systems, on emergent commodity architectures with 10,000 cores. The RELEASE consortium works to scale Erlang at the virtual machine, language level, infrastructure levels, and to supply profiling and refactoring tools.

This research contributes to the RELEASE project at the language level. Firstly, we study
the provision of scalable persistent storage options for Erlang. We articulate the requirements for scalable and available persistent storage, and evaluate four popular Erlang DBMSs against these requirements. We investigate the scalability limits of the Riak NoSQL DBMS using Basho Bench up to 100 nodes on the Kalkyl cluster and establish for the first time scientifically the scalability limit of Riak as 60 nodes, thereby confirming developer folklore.

We design and implement DE-Bench, a scalable fault-tolerant peer-to-peer benchmarking
tool that measures the throughput and latency of distributed Erlang commands on a cluster of Erlang nodes. We employ DE-Bench to investigate the scalability limits of distributed Erlang on up to 150 nodes and 1200 cores. Our results demonstrate that the frequency of global commands limits the scalability of distributed Erlang. We also show that distributed Erlang scales linearly up to 150 nodes and 1200 cores with relatively heavy data and computation loads when no global commands are used.

As part of the RELEASE project, the Glasgow University team has developed Scalable Distributed Erlang (SD Erlang) to address the scalability limits of distributed Erlang. We evaluate SD Erlang by designing and implementing the first ever demonstrators for SD Erlang, i.e. DE-Bench, Orbit and Ant Colony Optimisation(ACO). We employ DE-Bench to evaluate the performance and scalability of group operations in SD-Erlang up to 100 nodes. Our results show that the alternatives SD-Erlang offers for global commands (i.e. group commands) scale linearly up to 100 nodes. We also develop and evaluate an SD-Erlang implementation of Orbit, a symbolic computing kernel and a generalization of a transitive closure computation. Our evaluation results show that SD Erlang Orbit outperforms the distributed Erlang Orbit on 160 nodes and 1280 cores. Moreover, we develop a reliable distributed version of ACO and show that the reliability of ACO limits its scalability in traditional distributed Erlang. We use SD-Erlang to improve the scalability of the reliable ACO by eliminating global commands and avoiding full mesh connectivity between nodes. We show that SD Erlang reduces the network traffic between nodes in an Erlang cluster effectively.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Keywords: Distributed programming, Parallel programming, Fault tolerance, Scalability, Functional programming, Actor frameworks, Erlang, NoSQL DBMSs, Riak, Cassandra, Mnesia, CouchDB
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science; Q Science > QA Mathematics > QA76 Computer software
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Trinder, Professor Phil
• Date of Award: 2015
• Depositing User: Mr Amir Ghaffari
• Unique ID: glathesis:2015-6789
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 28 Oct 2015 10:35
• Last Modified: 28 Oct 2015 10:48
• URI: https://theses.gla.ac.uk/id/eprint/6789

Related URLs

• http://dl.acm.org/citation.cfm?id=250531...
• http://dl.acm.org/citation.cfm?id=263344...
• http://eprints.gla.ac.uk/108603/