Hey!

Welcome to Blip, home of the best original web series! When you’re done watching this episode, checkout some of our top shows or learn more about us!

×

Clojure

David McNeil: "Concurrent Stream Processing"

Other Sharing Options

×
Embed
The embed code has been copied to your clipboard
Share
About this episode
The foundation of our query processing engine is a concurrent data stream processor. This processor is characterized by the need to efficiently perf...
The foundation of our query processing engine is a concurrent data stream processor. This processor is characterized by the need to efficiently perform parallel, non-blocking processing of multiple data streams which are too large to fit in memory. Many such executions need to be executed simultaneously and fairly. The ideas in this talk are relevant to those who work with large scale, parallel data processing within the scope of a single process. A central theme of the talk is the creation of layers of abstractions to eventually create a language tailored to the problem. The talk discusses characteristics of the concurrent stream processor including: core data structures to represent processing nodes connected by data streams, processing plans represented as s-expressions, compiling s-expressions into processing nodes and streams, processing plan optimizations via s-expression manipulations, concurrent processing via a fork/join pool, facilities for debugging and cancelling executions and using the data stream processor as the core of a federated query processor. Less
41:18 How To
Discover the best in original web series.© 2012 Blip Networks, Inc. All Rights Reserved.