In module two, we will explore some classic computer vision techniques from low level features all the way to face detection. But first in lecture five, what is an image? We're going to be talking about images. What are images? How do they look basically for a computer algorithm? And then we'll start talking a little bit about image features, right? What are things in images that we are interested in detecting? So what is an image? When we look at an image, us as humans, we can notice a lot of things. For example, this is a famous image for computer vision scientists, most often it's referred to as Lena. Just a tidbit of information here. And this is an important image often used in computer vision algorithms as a testing imaging computer vision algorithms because it has a lot of components, a lot of things that were interested in detecting. So for example, there's a human face but it's not exactly frontal, it's a little bit tilted towards the profile, it's a little bit obstructed by the hat. The hat itself, it's not necessarily a smooth surface. There's some texture both here in the ribbon and up at the top. There's a lot more texture here in the feather, right? The woman appears to be more than focused in the background and there's a lot of different colors in the background, but they all seem to be kind of the same shades of tan, browns, yellows. Now we notice all these things as humans looking at an image. But what does an algorithm see when an image is loaded right into the algorithm, into the computer program let's say. So here we have the upper right corner of the image which is basically for the purpose of this presentation. Just a 12 by 12 pixels, Image, right, a sub image of the big image. You can think about the image as a matrix or a grid. And the values in the matrix are going to be intensity values. Now there's actually going to be more than one value most often for each pixel. So for example, this tiny pixel here in the corner, right, this one the last one, has three values, one for the color red, right, one for the color blue, and another one for the color green. So we call this the RGB system, and there's three values for intensity in the red color, intensity in the blue, and intensity in the green. What you see over here are the values for red for this sub image, this corner image is 12 by 12 pixel image. So I'm basically going to have a matrix, so you can think about it as an array of 12 by 12 by 3 values, right? And this will be really important because all these intensities are going to help us figure out features in the image and maybe something more complex, will be like a different some features arranged in a certain order or in a certain relationship with respect one another. That's what computer vision is trying to do. Computer vision is trying to look at these numbers, these intensity values, and try to come up with ways to figure out what is it that we should see in that image. Now another thing that I want you to know is that these values are one bite per value, eight bits, and they basically can represent integers with values between zero and 255, where zero basically means less intensity, right? So zero is going to be darker or black basically, and 255 is going to be lighter, but we're going to think about lighter as being more intensity, right, a higher intensity. But if we would put them all together, for example, the color white has maximum intensity in red, in green, and in blue, summing up maximum intensity in red, green, and blue will give us white. Now this is a three dimensional image. You can also encounter images. There are basically what we call gray scale, right? Gray scale images are not going to be three dimensional, are only going to be like a two dimensional matrix or array. So in this case, if we would take a little sub image here of 12 by 12 pixels, the data will be just 12 by 12, right, it will not be by two, right? So gray scale images, two dimensional color, RGB color images, three dimensional. Now let's look at again at this sub image at 12 by 12 pixels dimensions image. This particular one here is in grayscale. The values here are in grayscale, right? And let's dive a little bit into image features. So what are image features? Image features are some sort of specific information that we could detect and that would then help us detect other things, right? So the first things, like we call them low level features, things that we are trying to detect. Let's say we're going to start with edges. So we're trying to find edges in images. Of course, you can already see in this little image that there is an edge, there it is. This is an edge. How do we characterize an edge? That's one of the first questions that we want to ask ourselves. How do we characterize an edge? So an edge is somewhere basically where there is some sort of border, some sort of area in the image where the intensity changes, right? Over up here, we have high intensity, right? And over here we have low intensity. So an area where we go from high to low, where there's a big change in intensity, we're going to have an edge. And you can see this by looking at the values in this matrix, we have here the values, right? And we can kind of see that there is a demarcation and it's almost along the diagonal. Maybe a little bit more like this, I would say, right, where everything in this area is higher, the values are actually higher than 100 in the intensity of the pixels, while everything in this direction is lower, right? We can see that there are changes, right, for example from 128 to 81, from 131 to 69, right, from 97 to 51, all the way here to 21 right here in the corner, right? We can also visualize this even better if we try to plot this intensity values as a surface. Here is our image as a surface. One of the things to notice, these pictures were obtained in MATLAB programming language and I basically just did a surface plot of this matrix. And one thing that I want you to notice is that the origin of the coordinate system when we plot values in an image and even when we try to address a certain location in the matrix, the origin is over here. So the upper left corner of the image is basically the origin. So this particular pixel here is on the first row, first column, and its intensity is 117. Again, the surface plot helps us figure out that what we're trying to find in an edge is exactly this drop in intensity, in this case it's a drop, right? But depending from which direction we look at, it could be a drop if we look in this direction or it could be a climb if we look in this direction, right? And then the edge we're trying to find is somewhere around here, right? It's not necessarily a straight line, it doesn't have to be a straight line. Notice that here, it's kind of curving, right? So it starts like this and then it's kind of curving. So edges don't have to be straight lines. They could be all sorts of things. And actually, let's see why are we interested in edges. So we're interested in edges because there are a lot of things that we can find out if we discover edges in an image. So here's a kind of a diagram, let's say, of an object and things that could show up as edges in an image. So what causes edges, right? Here is one example over here, the first one, surface normal discontinuities, what does that mean? It means that our object has some angles, right, some object angles. Our object is not necessarily smooth, in this particular case, the bottle has a cap, right? This change over here, right here, could give us an edge in the image, especially if there is also some light issue where maybe this area here will appear lighter and this area here would appear darker. That would generate an edge over here, right? Here's another one, number two, depth discontinuity. Depth discontinuity, you can think about it is basically the edge of an object, the edge, the end of an object. So because this object is in front of something else that is behind in the background, we say that the object is at a different depth from the camera, right? And hence this edge here at the border of the object, right, will cause an edge in the image. So that would be a depth discontinuity. Three, surface color discontinuity. Okay, now this kind of has a funny name. Surface color, what does that mean? That means that our object, the surface of our object whether is a cheetah or whether it is a checkerboard or whether is a car, right? The surface, it's not all the same color. And on purpose, I chose the cheetah, right, in a car that could have some letters printed on it, maybe a bus, something like that, right? So anything where the surface of the object is not mono color, right? Any sort of letters, printing, texture, right, like the cheetah or like the checkerboard, right? All of these would generate edges right here. There's an edge over here and there's another edge over here. There's all of these are going to be detected as edges, right? And then the number four, another situation where edges could appear, illumination discontinuity. This is actually pretty cool. So we have the object, but then we have this whole thing here that has an edge, it's not the object but it has the shape of the object and it's basically the shadow of the object. So sometimes the object, the contour of the object itself could generate edges. Other times the shadow of the object would generate edges too, and we have to be really careful that we do not detect basically two bottles in this image. One and the one that is the shadow, but rather just one bar, right? Now we'll find out that in classic computer vision, we have edge detectors and we're going to learn about them in the next lecture, but no edge detector can discriminate between these cases. So even though edge detectors will give us edge maps, edge maps can be hard to use because just because we detected that there is an edge in the image over here, we don't know for sure if the edge is caused by the contour of the object or by the shadow or by some print texture something in the surface color discontinuity or by some angle in in the object, right? So we will get an edge map, but we won't be able to say what caused every edge on that map. Okay now let's look a tiny bit at types of edges that we're going to try to detect. So the step edge, so we said that we are going to be looking for a change in intensity. So in the step edge, the change in intensity is very sudden, right? Other like this from low to high or like this from high to low, right? This is the step edge, right? Very clear distinction of the sudden everything was dark and then it changed and everything was white, right? The intensity changes abruptly. A ramp edge, a ramp edge, the intensity changes but not as abruptly, right? So it could happen over the distinct period of time. But it started somewhat constant, it changed and then again it will encounter another part of somewhat constant, again from low to high and from high to law. Ridge or line edges, this is where the image intensity changes abruptly, right? But then it somewhat quickly returns, right? This would be, for example, an image that has something like a stripe, right? The intensity changed, this is more like this, the intensity changed, but then it quickly got back up, something with lines or stripes. It could be for self driving cars, lines on the road, right? It's gray, it's white, and it's gradient, right? And then the last one is roof edge, right? Roof edge is pretty similar to the ramp edge, where basically the intensity doesn't change instantaneously, occurs over a finite difference of time. The problem with the roof edge is that it doesn't stay at the high point too long but it comes back over here, right? Really it quickly returns back, right? Sometimes we say it could be generated by the intersection of two surfaces, but you never know. It's basically the intensity ramps up and then it quickly comes down, right, it could be the other way around as well. I have here definitions of all the different types, profiles of image intensity edges, and you can read over it and you can see some examples on the previous page. And the next time, we're going to look at gradients. How we use the idea of gradients to detect edges in the images. We're going to look at pre processing, maybe it's a good idea to smooth the images first before we look for edges. And then we're going to look at linear filters that we can apply to the image via an operation we call convolution to detect features such as edges. And last but not least, we have here on the last slide definitions for different profiles of image intensity edges. We have again the step edge, the ramp edge, the ridge or line edge, and the roof edge for which we had examples on the previous page. This concludes our lecture about image features. Join us next time as we introduce the concept of convolution. Thank you.
