PyCon 2011: Creating Complex Data Pipelines in the Cloud: The App Engine Pipeline API

Brett Slatkin This talk will cover App Engine's new Pipeline API, which connects together complex, time-consuming work (including MapReduces and human actions) in a distributed system. The API transforms Python into an asynchronous language for describing data dependencies in a novel way. We will discuss how the API works, how it achieves parallelism, and how to reuse its design and code outside of App Engine.

More episodes of PyCon US Videos - 2009, 2010, 2011

Featured episodes in Learning

PyCon US Videos - 2009, 2010, 2011

PyCon is an activity of the Python Software Foundation, a 501c3 non-profit organization. To support future conferences, please donate to the Foundation at www.python.org/psf/donations . Video and audio material from PyCon are licensed under the Creative Commons CC-BY-NC-SA license . This means you can incorporate excerpts or entire recordings in your own non-commercial projects, as long as you credit the speaker and you CC-license the finished project.