So now, in this last chapter, we're going to talk a little bit about visualizing data. But what we're really doing is we're summing everything up, because we're going to retrieve data from the network, we're going to process the data, we're going to store it in a database, we're going to then write it out and visualize it. So, it's all coming together, and and it turns out that this notion of gathering data, the data gathering using the network it's pretty common thing. It might take a cleaning or processing step where we're- the part of the problem is when you're pulling data off the net, you want to be able to start this process because it'll run, and run, and then your computer will crash or it'll go to sleep or something. So, you don't want to start from the beginning because it might be quite a bit of data and it takes a while to do it or as we've seen in some, you might be talking to an API that's got some rate limit that says, oh, you'd have to stop at 14 of these things, or stop at 200 or whatever. So, this is often a restartable process and it's usually a simple process. It's usually a relatively small amount of code, where you have a queue of things you want to retrieve, you go to the next one and then you stored in database, next one starting database and when you start the process up, you start filling this database up with stuff and then if it blows up and you restart it, the first thing it does is it reads the databases. "Oh, I don't need any of those", and then it starts to get the next one, and the next one, and the next one, and the next one. That is how you make this restartable. Databases are really good at having it, so that your program that's writing to the database can blow up and you don't corrupt your data. You don't have partial data, it's either written or it's not written and so these things can blow up in. Sometimes you just blow them up, because you want to blow them up and start them up, and you start them up again, they scan down, say, ''Where was I? Oh, I'll start here, here, here, here. '' So, this is often a slow and restartable process. It also might be limited but for some reason. So, this runs for awhile and the third thing we'll do in this chapter, it might run for days actually. Then, you have your data and then you start doing stuff inside your computer where you don't really care so much about the network, it might be this is raw data that came in off the APIs and you want to make the data in some new little format, so you might go from one database to another database or a database to a file and produced data that's really ready for visualization. If this might be a little complex or there might be flaws in it, you might write scanners that go like, "Oh, wait a sec, this is inconsistent, sometimes it looks like this and sometimes it looks like that, so I'll clean that stuff up." Then, using some visualization or doing some Python Programs that loop through the data once it's cleaned up and then do some summing or adding or who knows what they are that they're doing, but analyzing or visualizing. What we're going to use is we're going to use things like Google Maps to do our visualization, a lot of JavaScript and a thing called D3.js, which is a JavaScript library. Now, in this class, we're not teaching a JavaScript, we're not going teach you Google Maps. I provided all these things, so that when you run these programs, that stuff is all there. But if you want to learn and see some examples of how to make a little simple JavaScript visualization with a line or a word cloud or a map, we've got it, and you can take a look at those things. Now, this is one form of data mining and its really a data mining for an individual, where you're pulling this data, you're getting at local and then you're working with it there are other much more sophisticated data mining technologies that you might find yourself using. But often, you'll also find Python is part of these or Python helps you manage these or you write a Python Program to scan through these things or to prepare them or to do something. So, there's lots of different data mining technologies, this is just one oversimplified very Python-oriented data mining technology. I'd call this personal data mining. You should take classes. If you really want to become a Data Mining Expert, this is just giving you some of the skills that we've learned in this class and solving some data mining. So, the first application that we're going to data mine is an extension of an application we played with back in the JSON chapter. The idea is it has a Q of Locations. These are not pretty locations, meaning their user typed in your locations, they're actually from data from many years ago. It's anonymized data from the students who actually took one of my very first MOOCs, MOOC on Internet history, but it's reduced in anonymized just play with it. But it's not accurate. We don't have GPS coordinates. But if we use the Google GeoData API, but JSON we can do this, but we need to avoid rate limiting, so we're going to cache this in a database. Meaning, we're only going to retrieve data at once and then we're going to use the Google Maps API to visualize this in a browser. The sample code is right there and that sample code geodata.zip has a read me and it tells you exactly what to do to run this and it shouldn't be very hard for you to run it and produce a nice visual result. Here's a basic process diagram of what's going to happen, there is a list of the things to retrieve called where.data is just a list of the locations, but these are not correct, they don't have GPS, there just a as typed into a text field by a user and Geoload is going to start and start reading this and it checks to see if it's already in the database. This is a restartable process as I mentioned and then it looks to see the first unretrieved data and then it goes out and does a web service, parses that then puts it into the database and then goes to the next one, parses that puts it in a database and this runs for awhile and then maybe you blows up then you fix whatever or you start your computer backup and runs for a while. So, this is a restartable process that in effect is adding stuff to this database, it's an SQLite database and you can use the SQLite Browser to look at this if you like stuff we did in the database chapter. So, you can run that, you can see what you got, run at some more sewage which he got, debug it by using the SQLite Browser. Then at some point you've got all of your data and you want to you've got a couple of things we got this application called geodump.py that read through all of this data and then print some information out, nice summary information. It's really common to want to do this to get some summary information just for sanity checking, so you don't have to use SQLite Browser but this also writes out a little JavaScript file called where.js which then combined with where.html and the Google APIs. This uses JavaScript to put all these little pins on based on whatever data is in this database. So, that's our first end-to-end spider process visualize. First thing. So, up next, we're going to show how we can use this to build a very simple search engine and then run the PageRank algorithm.
