There is a trend towards distributed systems made up of rapidly changing ephemeral single function microservices that can be deployed globally using public clouds. Monitoring tools have tended to focus on a small number of slowly changing monolithic applications in a single datacenter at a time, so a new breed of monitoring tools is emerging that collect data every second, are cloud and container aware, and can see the flows of requests across microservices. Testing these tools is a challenge, and this talk will describe how the SimianViz simulator can be used to model interesting architectures and create stress tests and failure scenarios.

About Adrian Cockcroft

Topic: The rise of distributed systems and how to make it stop.

About Nicholas Weaver

The Internet of Things is not only exciting but has huge potential to benefit society. This talk highlights three key opportunities and associated technical challenges. First, indoor-location based services promise to have a huge impact with a variety of novel and surprising services – especially when we can get highly accurate estimates of a person's or object's indoor location. Second, software-defined infrastructure has contributed to great advancements in the cloud, data center, and other complex distributed systems. We describe how it can potentially provide even greater benefits to IoT via simplification, automation, and agility to roll out new applications and services. Third, while today's popular mobile-to-cloud architecture is highly successful, many IoT applications require an enhancement of this architecture where an additional layer (the "Fog") brings some of the cloud's capabilities to the edge of the network – including support for real-time analytics. We will highlight the technical challenges, recent advances toward their solution, and promising future directions of research.

About John Apostolopoulos

Deploy and manage the Basho Data Platform in any environment through Apache Brooklyn blueprints and the Cloudsoft Application Management Platform.

This session will show the suite of Riak and BDP blueprints available in the Apache Brooklyn ecosystem, walking participants through everything from simple deployment to complex management and multi-datacenter replication.

We'll highlight management available at deploy-time through simple blueprint configs and at runtime using sensors and effectors in the UI.

We'll also demo the latest work managing the BDP including bringing your own blueprints for data and analytics tools.

About Mike Zaccardo

About Alex Heneveld

Spark is a popular new platform for interactive high performance analytics, machine learning, and data processing. The trouble is, Spark tends to monopolize whatever Mesos cluster you run it on, so you either create completely separate Spark clusters for each user, or you otherwise limit the resources each user can use.

Cook is an advanced fair-sharing, preemptive scheduling backend for Spark. You can run one instance of Cook on your Mesos cluster, and it will automatically adapt the capacity for every user and team on your cluster, so that interactive jobs run immediately but utilization remains high.

Cook also has a REST API and Java client, and it's written in Clojure with Datomic.

About David Greenberg

We endeavor to build consistency, availability, and fault tolerance into our distributed systems, but how do we build them into our teams? The human factors in devops require as much attention as do our technical implementations.

Collaboration, understanding, trust: we know how important these interactions are in a devops practice, but how do we enable them between disparate team members, especially in a distributed team? My company's in California, and I'm nearly two thousand miles away in flyover country, USA. Being one of those little squares at the bottom of every video call gives me an outsider's perspective on the inside of our organizational optimization.

Drawing in comparisons from theoretical computer science and practical systems implementation, I'll explore how building understanding requires a practical application of great tools in a deliberate pursuit of a constructive culture.

About Bridget Kromhout

TBD

About Don Rippert

Writing distributed systems is hard. Operating them at scale even harder. With Apache Mesos you've got a highly scalable and flexible kernel for a distributed OS at your disposal. By flexible I mean the capability of supporting both stateless and stateful services. A particularly interesting class of stateful services are NoSQL datastores and here in special Riak. With the recent introduction of persistent primitives in Mesos—dynamic reservations and persistent volumes—managing these sorts of workloads became much easier. There are still challenges, though, including transparent data volume migrations and explicit support for external resources, that is, enabling Mesos to manage cluster-wide resources more efficiently. We will discuss both the theoretical underpinnings and developments in this regard as well as pointing out main working areas in Mesos and the wider container community.

About Ben Hindman

Development practices for distributed systems promise the benefits of both flexibility and velocity, but confidence enables those benefits. As these systems evolve, they exceed the ability of any single human to mentally model. The future of software is complex, increasingly opaque even to the engineers building it. We can no longer rely on an architect to reason about these systems. If we can't reason about them, how can we have confidence in them?

An empirical, systems-based approach is needed to build confidence. We learn about the behavior of a distributed system by observing and experimenting on the system as it runs in production. We call this Chaos Engineering.

Chaos Engineering is a powerful practice that is already changing how software is designed and engineered at some of the largest-scale operations in the world. Where other practices address velocity and flexibility, Chaos specifically tackles systemic uncertainty in these distributed systems. The Principles of Chaos provide confidence to innovate quickly at massive scales and give customers the high quality experiences they deserve.

About Casey Rosenthal

In this talk I will present Transformations, a new vision for building principled distributed systems. I will show how searching for the appropriate abstractions leads to a new approach for building and reasoning about distributed systems.

Large-scale distributed systems change and evolve very fast as a result of adding functionality to the original system, deploying new hardware in the datacenter, responding to changes in the workload, and so on. The high rate of change in their design makes distributed systems harder to reason about and implement correctly over time. Consequently, we sought out a way to build and evolve large-scale distributed systems in a principled manner, using what we call Transformations.

Transformations offer a new way to reason about a large-scale distributed system's properties such as correctness, safety, fault-tolerance and availability as the system changes and evolves over time. Transformations introduces an algebra to reason about how distributed systems evolve over time. Moreover, using Transformations developers can transform an existing large-scale system and change its properties automatically while being able to reason about these changes. During my talk I will cover the main contributions of Transformations by first explaining the principles we use in Transformations and then showing how it can solve very challenging problems we face as distributed system developers: Sharding different components of an already replicated system while batching some of the requests and implementing encryption between select components, or taking a system replicated with Paxos and transforming it into a system replicated using Chain Replication.

About DenizAltinbuken

Time Series and Queriability

Riak TS is a new Basho product built on the core architecture of Riak. Gordon Guthrie, the Eng Lead on Riak TS will discuss the how time series data drives unique functionality requirements including data co-location and querying and how they are addressed in Riak TS. He will also discuss IoT and time series data modeling, and show code examples of building tables and queries for time series use cases.

About Gordon

TBA

About DenizAltinbuken

Containerization makes it easier to package and deploy any application using a unified tool chain. As organizations being migrating, many have virtualized workloads that cannot be easily containerized, or application workloads such as static binaries and JVM applications that do not benefit from containerization. To address the growing heterogeneity of workloads, HashiCorp has developed the Nomad scheduler. Nomad is a globally aware, distributed scheduler designed to handle any type of workload on any operating system. Developers specify jobs using a high-level HCL specification, and Nomad manages the placement, scheduling, auto-healing and scaling automatically. In this talk we discuss the Nomad architecture and how it can be used to handle the challenges of scheduling in a modern datacenter.

About Armon Dadgar

TBD

About DavidP alaitis

Since the formalization of CRDTs (Conflict-free Replicated Data Types) several systems have adopted them to design sound approaches to eventual consistency, and now operate, at scale, with low latency over global networks. Instead of focussing on simple read and write operations, CRDTs try to capture the finer semantics of data type specific operations, thus enabling deterministic merging of concurrent data changes.

The initial state-based CRDT designs had a considerable meta-data overhead, since they had to provide a data format that is resilient to out-of-order transmission and message duplication. In this talk, while describing a reference implementation in C++, we will show how meta-data can be greatly reduced by using causality instead of tombstones. The examples will also illustrate how composition can assemble together and re-use basic building blocks, and how data dissemination among replicas can be further optimized by shipping small deltas in place of full replica state.

About Carlos Baquero

Consistency is hard and coordination is expensive. As we move into the world of connected 'Internet of Things' style applications, or large-scale mobile applications, devices have less power, periods of limited connectivity, and operate over unreliable asynchronous networks. This poses a problem with shared state: how do we handle concurrent operations over shared state, while clients are offline, and ensure that values converge to a desirable result without making the system unavailable?

We look at a new programming model, called Lasp. This programming model combines distributed convergent data structures with a dataflow execution model designed for distribution over large-scale applications. This model supports arbitrary placement of processing node: this enables the user to author applications that can be distributed across data centers and pushed to the edge. In this talk, we will focus on the design of the language and show a series of sample applications.

About Christopher Meiklejohn

When a bug is triggered during a distributed system's execution, developers need to understand what events caused their system to arrive at the unsafe state before they can begin to isolate and fix the bug's root cause. This process of troubleshooting can be highly time-consuming, as developers spend hours poring over multigigabyte traces containing thousands of events.

I will present a tool that reduces effort spent on troubleshooting distributed systems, by automatically eliminating events from a given buggy execution that are not causally related to the bug at hand. The developer is left with a ``minimal causal sequence'' of triggering events, each of which is necessary for triggering the bug. We claim that the greatly reduced size of the trace makes it easier for the developer to figure out which code path contains the underlying bug, allowing them to focus their effort on the task of fixing the problematic code itself.

In the talk I formally define the execution minimization problem we set out to solve, the techniques we have developed to solve it, and the system we have built to apply our techniques in practice.

I draw examples from real bugs we have found and minimized in Raft, Spark, and Pastry. Throughout, I outline how developers can find and minimize buggy executions for their own distributed systems.

About Colin Scott

At Booking.com, we have a constant flow of events coming from various applications and internal subsystems (1.5 Billions events per day). This critical data needs to be stored for real-time, medium and long term analysis.

Our events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak.

First, the talk will briefly show real events examples, and how we serialize and aggregate them.

Then, Riak configuration and data modeling will be detailed, including how data are sent to Riak, and read out of it.

Next will be a section that demonstrates Riak flexibility solutions via 2 real examples: how we cut in half the cluster internal network usage, and how we used post-commit hook to perform real-time data crunching on the cluster nodes.

Finally, the talk will present our solution using Yokozuna to build a Time Series Database on top of Riak, for near real-time ad hoc analysis of a portion of our data flow.

About Damien Krotkine

As ShopKeep grows we must empower our developers to build and scale to meet the needs of the business. This talk looks at the genesis of ShopKeep, and how Riak has helped power our growth across the globe. This talk also has some dos and don'ts on building an actual service-based distributed system leveraging some awesome new technologies.

About Duncan Grazier

Panel Discussion with Heather Mckelvey, Matt Digan, Cuyler Jones and James Gorlick about the Basho Data Platform.

About Heather Mckelvey

About Basho

Panel Discussion with CRDT

Discussion with Riot Games

Panel Discussion with Desiree Van Welsum (OECD), Jonathan Murray

Sponsor Talks - reserved for HP

Intel Keynote

Panel discssion with Alex Williams (The New Stack), Dave McCrory (Basho Technologies), Nicholas Weaver (Intel), Mac Devine (IBM), Duncan Johnston Watt (Cloudsoft), Donnie Berkholz (451 Research).

About Alex Williams

Sqor Sports is a globally distributed social network built using Erlang and Riak. Our customers are some of the largest sports teams and biggest athletes on earth. In this session, I talk about some of the challenges we have tackled using Riak at our scale.

Technical Audience Level: Both advanced and beginner welcome

Topics Include:

• Global datacenters and Riak
• Datascience pipeline challenges
• Monitoring and scalability testing

About Noah Gift

In this talk I plan to discuss the design and implementation of geo-replicated data stores that go beyond offering weak consistency semantics, and instead offer causal consistency guarantees (with eventual convergence guarantees, often called Causal+ Consistency) or go beyond offering operations over single objects in the data store to offer transactions that manipulate multiple objects with isolation guarantees. The talk will be divided in three main sections.

In first section I will cover the motivation for providing stronger semantics than weak consistency on data stores illustrating the need for either causal consistency or transactional semantics with some simple examples. The talk will briefly cover the design and implementation of several existing solutions, addressing briefly the differences of solution in the design space. From this quick overview, the talk will then shift focus to two concrete solutions proposed by the academia in which I have participated in the design and implementation.

In the second section of the talk, I will focus on the design and implementation of ChainReaction, a geo-replicated data store that offers causal+ consistency. ChainReaction was implemented on top of the Fast Array of Wimpy Nodes (FAWN) datastore, which features a one-hop DHT (similar to the Riak one). I will discuss the variant of Chain Replication employed at the core ChainReaction, which trades write latency for fast causal consistent reads, and explain how this design is extended to a geo-replicated scenario.

In the final section of the talk I will address the design and implementation of Blotter, a geo-replicated data store that supports arbitrary transactions with an isolation level of non-monotonic snapshot isolation. At the core of Blotter we leverage on a special configuration of Paxos, which allows a single round-trip among the closest datacenters for committing a transaction.

About João Leitão

The journey of how a Sensor manufacturer went from a pc client connected to Postgres, to a online system with Postgres to a online system with riak.

The system we built is an online management and reporting tool for measuring data. There are thousands of online sensors all connecting at irregular times pushing data up and pulling configuration down. Clients and sensors are distributed over the globe requiring an always on system.

How do you use the strengths and coop with the limits of riak, both from a developer stand point as well as an operational one. How we designed the system to take full use of a distributed database as a core part of the system.

How do you break out of your rdbms shell and start think in terms of nosql, things like denormalise, time slice and other deterministic segmentations.

About Johan Sommerfeld

TBD

About Sarah Novotony

Please join me as we explore some of the important characteristics of a reliable and efficient distributed system, such as scalability, availability, performance, latency, fault tolerance and learn how those same attributes can and should be implemented across all areas of your IT infrastructure.

About Kevin Jones

Does it bother you when all your microservice logs show up out of order in your log analysis tool because your system clocks aren't synchronized? Did you ever wonder what time it was when the dotted version vector on a piece of data became 2.4.5? If so, then this is the talk for you!

Time is a notoriously difficult concept in distributed systems. Perfect clock synchronization is impossible, and most logical timestamp schemes (like Lamport clocks) don't bear a relationship to "wall clock time" that is meaningful to human operators.

In this talk, we'll learn about distributed monotonic clocks (DMC) whose timestamps can reflect causality but which have a component that stays close to wall clock time. This scheme builds on previous hybrid logical clock proposals by adding important operational hooks and by building in mechanisms to prevent a single runaway system clock from dragging a whole cluster's logical clocks forward into the future.

DMCs can be implemented as a "piggyback protocol" suitable for transport in existing application messages--for example, as additional HTTP headers. I'll describe a novel coordination protocol in this unusual piggyback style--a style that may also be useful in ad-hoc clusters formed by Internet-attached "things".

About Jon Moore

Modern NoSQL systems have made it significantly easier to build complex, constantly evolving applications. These systems' support for data-models that allow nested data and dynamic schemas allows developers to quickly prototype and deploy new features. While their support for flexible data-models is a boon for developers, several NoSQL systems have eschewed one of the main programmer-friendly abstractions of traditional DBMSs; atomic transactions. Atomicity guarantees that a group of writes to multiple data items are performed in an all-or-nothing fashion. Atomicity frees application developers from writing error-prone corner-case code to deal with scenarios in which only a subset of a transactions' writes succeeds.

The reason that these systems eschew atomic transactions is their seemingly prohibitive performance cost, particularly in distributed settings. In a distributed database system, atomicity requires all the machines involved in a given transaction to coordinate with each other. In particular, if a transaction reads or writes data residing on several machines, then atomicity requires each such machine to coordinate with every other machine involved in the transaction. To circumvent this distributed coordination, several popular NoSQL systems give up on general atomic transactions, and instead restrict their scope to a single key or partition.

This talk takes a step back and asks; does distributed coordination necessarily preclude performant general atomic transactions? The answer is a resounding NO. This talk analyzes the distributed coordination necessary for general atomic transactions, and describes its impact on three key performance and correctness properties; fairness, isolation, and throughput. We will find that there exists a three-way tradeoff between fairness, isolation, and throughput (FIT); a system which supports general atomic transactions can achieve at most two of these three properties simultaneously. Database architects can use the FIT tradeoff to reason about the performance and correctness tradeoffs associated with general atomic transactions in a principled fashion.

About Jose Faleiro

We will start with an empty git repository and cover all the steps, technologies and concepts needed to build a riak_core based backend.

The topics covered will include:

• Using rebar3 and a rebar3 template to setup the initial project structure
• Building, running and creating releases with relx
• Handing configuration with cuttlefish
• Implementing our first commands
• Understanding consistent hashing, commands and vnodes
• Understanding and implementing quorum based reads and writes and coverage calls
• Understand, implement and exercise handoff via cluster nodes joining and leaving
• Expose runtime metrics with exometer
• Expose our protocol as an HTTP and websocket/SSE/COMET API with cowboy and bullet
• Implementing authentication/authorization with riak_core_security and JWT
• Explore riak_core metadata

About Luis Mariano Guerra

About Manav Mishra

Spend a session reflecting on the foundational papers and people behind distributed systems in a fun and quick moving context. We will take a whirlwind tour through the ideas embodied in the classic papers that form the basis for distributed systems. You may be surprised, hopefully you will be delighted and you will leave with a great bibliography for further inquiry and professional biographies of the people behind the papers.

About Mark Allen

We all know the scenario: what was once an experiment soon becomes a desired product. Suddenly it's time to scale the system, and all the time we hear about running "at scale", but what does that really mean? This talk is about the challenges OpenX faces when scaling a small proof-of-concept to a complex global product. It focuses on our strategies for planning, adapting, and growing distributed systems into fully productionized services. Operational best practices factor in strongly; statistics gathering, careful capacity planning, and precise tooling help pull off migrations and upgrades with little impact. A close relationship with Engineering and working in a constant feedback loop with developers is another part of the puzzle, helping the entire team launch successfully. Through these examples we'll attempt to de-mystify distributed systems, move beyond those basic five-node clusters, and find how the approach to scaling something like Riak is equally applicable to many other kinds of highly available clusters.

About Matt Davis

We present Hyperbahn, a service discovery and routing system developed at Uber. Hyperbahn implements some novel features such as end to end rate limiting, circuit breaking, and backup requests with cross server cancellation. This talk will cover Hyperbahn and how we do RPCs at Uber.

About Matt Ranney

Many newer data stores, distributed ones in particular, carry a great deal of promise. However, many bright-eyed engineers find themselves struggling just to keep the darn thing up instead of experiencing the operational nirvana described in the documentation and marketing materials. What gives?

The reality is that even the "simplest" data stores are complex beasts, and the way they fit into YOUR infrastructure and applications makes them even more so. Learning to be truly "at peace" with even a single database technology can take years. Achieving moderate proficiency and competency usually takes less time, but the act of learning itself can be treated as a skill and honed.

In this talk we'll talk about just that - how to get good at getting to know a new system, and how to understand its limitations BEFORE your production workload is running up against them and falling down.

• getting comfortable interacting with a database
• finding the shortest feedback loop
• teasing out the important and relevant claims and guarantees
• quickly validating said claims
• formulating questions and hypotheses relative to your workload
• comparing vastly different data stores in a meaningful way
• moving existing workloads over while minimizing risk

No one wants to spend their whole life just trying out different databases, but the perpetual tire fire that often results from a poorly chosen one is no way to live either. Let's make better decisions, together!

About Mikhail Panchenko

Today's NoSQL databases provide the perfect framework for big data applications with the ability to scale and use more resources for both storage and compute.They also provide WAN replication capabilities and SQL support. It seems that one of the challenges that still exists in NoSQL databases is the ability to perform distributed transactions.

Distributed transactions are an essential need for finance, e-commerce, telco and other industry applications. One use case for distributed transactions is a money transfer between 2 accounts which are stored in two separated partitions. The challenge here would be to lock both partitions for the transactions and still provide high throughput, reliable data and high availability. In this session, we would like to discuss the ways to achieve distributed transactions on Big Data frameworks.

About DeWayne Filppi

The Internet was not designed to support 50 Billion devices fanning-in Brontobytes of mostly low value updates. And yet every major vendor, analyst and grandmother will tell you that the Internet of Things (IoT) is on the cusp of doing just that. Smart organizations are optimizing traffic, delivering insights and enforcing security closer to the devices critical to their business with distributed IoT Platforms. Distributing insight and security in IoT has several unique challenges from identifying node device shadows to maintaining decision context to the resource limitations of many edge devices. We'll examine how to address these challenges with compose-able nodes, context priority and audit-able IoT framework components.

About Sarah Cooper

TBA

About Sarah Novotny

Chain replication, a variation of primary-backup replication, is an increasingly-common technique for maintaining strong consistency semantics within a data store. To date, more research effort has gone into formalizing the data replication technique than into formalizing management of its metadata: i.e., what decides the chain's order, and decides when the chain order may change without violating consistency constraints.

This talk introduces Humming Consensus, a new technique for managing chain replication metadata. This metadata defines cluster membership, server order within the chain, and changes to chain order in response to peer failure. All participants in Humming Consensus are equal peers that operate without external assistance from ZooKeeper or other coordination service. Humming Consensus may also be used to manage chain replication-based eventual consistency data stores. The talk will also briefly introduce Basho's new distributed file store, Machi, which relies upon Humming Consensus to operate either in strongly consistent or eventually consistent environments. Attendees will need no prior knowledge of Erlang: the talk will focus on Machi's design and testing methods rather than its implementation internals.

About Scott Lystig Fritchie

Would you believe that Riak 2.0 has everything you need to build a distributed queue? That was the question a few engineers set out to answer, after a night out of drinks and disagreement.

In this presentation, we'll talk about an in-house distributed queue we built in Golang, called Dynamiq. We'll cover why we chose Riak and Golang, the design behind the system, and most importantly how it works, and how we put it into production. We'll cover the clever ways you can use Riak to do "the hard parts" of a distributed workload, and build a system on top of it that extends it's core functionality in new ways.

About Sean Kelly

In recent years, researchers and practitioners have been trying to ensure applications correctness while providing low-latency and highly available operations in geo-replicated systems. Despite these continuous efforts, it is well known that many types of operations must rely on cross-replica coordination to ensure that application invariants are preserved at all times. This is a fundamental limitation on improving services, as many companies cannot afford the extra latency to ensure consistency across replicas nor the reduced availability during faults, as this directly translates into revenue losses. In this talk, we discuss how to reduce cross-replica coordination while still preserving correctness properties by exploiting applications semantics.

The talk is divided in two parts. In the first part, we show how to preserve global numeric invariants on top of an existing production KV-store, while moving most coordination outside of the critical path of operation execution. We achieve this using a new replicated data-type to maintain the necessary information and show experimental results for a proof-of-concept prototype.

In the second part, we explore the more ambitious vision of providing generic application invariants with virtually no coordination. The fundamental insight is to identify and prevent the execution of concurrent operations, at different sites, that would result in invariant violations when the effects of the operations are merged together. We combine a static analysis tool and an online concurrency control mechanism to avoid unsafe executions.

About Valter Balegas