Hi, my name is Reza Rokni and I'm a Google Cloud developer advocate for Google Cloud dataflow. Today, as you can see on the agenda, we are going to be looking at Beam notebooks and running them on the dataflow service. We think about the way that we would normally develop a Apache Beam pipeline. The SDK allows us to declaratively describe the pipeline that we would like the service to run. As we can see in this example here with the store sales and the online sales, which comes and describes a directly slightly graphic computation, a DAG. Once we've done this, we actually submit it to the service and the service goes ahead and runs that pipeline for us. This is fantastic for production use cases because it allows the runner to be able to do fancy optimizations like dataflow fusion, where it can collapse stages together and make it very efficient for the processing that's going to happen. However, it's not the best experience when you're first developing new pipeline because we often get into this, write, submit job, read, waiting for logs, waiting for print statements. This is especially true when we're still exploring our data, where we would not only access to the data as we proceed, but access to the data as our transforms or transforming that data. Now, with Apache Beam, we do have the interactive runner that's available, and the interactive runner allows us to be able to get access to the pipeline. Specifically, it gives us access to the intermediate results that are available from all transformations allowing us to do exploration and the next state is about development. Importantly, as you would expect with Apache Beam, the interactive runner also works with batch and stream sources. We no longer have to mock out our objects as we're doing our development. We can actually, when needed, run directly against the real data, even if that source is unbounded and in stream. Going on to how we get to run this, we'll look at the Beam notebooks. On this slide, you actually see the various steps we need to take with the dataflow service UI to be able to set up a notebook environment. We start the data for notebooks that gives us access to creating new instances. Nice thing there is that once a new instance is being created, which is the host for our notebooks, all of the libraries and things that we need for Beam already in place so we can immediately start doing our development. The notebooks environment also comes with some ready-made examples which are great for exploration and learning, but also very useful for code snippets. Specifically, we'll talk a little bit more about one of the examples, the word count example, and we won't reproduce it completely in this session that's for you to hopefully do once you start exploring the notebooks directly. But just to take a few snippets and some of the options that would be available when you do start making use of the notebooks. In this slide, we can actually see some of the transformations in that word count example. First, we need to read from our unbounded source, which we do with read from pops up. This puts it into the words P collection. We then apply a fixed window to those elements a fixed window of 10 seconds. This puts it into the window words P collection and finally we do a count. Now, if we were doing this without the interactive runner, at this point, we'd have the fourth transform, we'd do some logging or convert it to a sink where we can then look at the data. But because we are using the interactive runner, we can access intermediate results. However, we do need a way to tell the system when to stop reading from this unbounded source of information. The way that we do that is with a couple of options from the interactive runner, which is the ib.options. Either recording duration to be said, or recording size limits, the duration gives a fixed amount of time for the interactive runner to record data, and the recording size limit gives us a fixed amount of bytes to read. The latter is very useful when you're working with a real stream of information where you might have a very large volume of data, and you don't want all of that to be put into your notebook as you're doing the experimentation. The other important factor is that, we have the option to actually reuse the stream of information that we've gathered, or to get fresh data. The reuse is useful because then as we are exploring data and manipulating it, we are working with the same dataset. In terms of looking at those P collections and looking at that intermediate results, ib.show allows us to visually see the information as we can see in this slide example here. The auction include Window info will also give us some extra metadata about each element. For example, the event time and the Window that data belongs to. Visualization is obviously very useful, but we also want to be able to use this data and manipulate it directly within our code. For that, we could use ib.collect which allows us to then output this into a pandas DataFrame, which then we can do our manipulation against. If we wanted to actually do further visualization of data with graphs, ec ectra. The notebook comes in built with a feature via ib.show, which is to set visualized data to true. This gives us access to the UI that you're seeing here and you can do various visual exploration of the information. With these core primitives we can now start getting out of that write, submit to service life cycle when we are dealing with our data. But obviously once we complete our development, we then want to be able to submit the job to the service and we want to do that with as little as code as possible. Finally, [inaudible] going from development to production. There's very little we actually need to do with the code. Because at the end of the day we're just writing Beam SDK code for this whole process, we just need to enable running on the service by importing the dataflow runner, by providing options through pipeline options, for example, the project and the staging directory, etc. Then we just do runner.run_pipeline which will submit that job to the service. Hopefully with this, you've got a nice overview of what we'll be available when you start working with the notebooks. Thank you.
