Welcome to this lecture on designing a virtual reality. Are you confused? Well, I just thought what a funny way to kick off this lecture. Being in virtual reality. In fact I was just looking at a virtual reality desk. So the Oculus kind of like area that you see when you start and, [LAUGH] wanted to play a trick on you here. No, this is really, this week is going to be quite cool. You're going to learn about a lot of the basics when it comes to virtual reality design. I'm going to share a few techniques, I'm actually quite excited. So the way I've structured it is like, this is the intro to virtual reality but development. I'll just assume you know about the concept and you know about the technologies. If you feel like you have some refreshing to do, please look at other MOOC. The first one is part of the XM MOOC. The first course really provides this foundation, in depth review of VR and AR both the concepts and the technology. So designing a virtual reality. And, as I said initially in week one, we're going to have two case studies throughout this entire MOOC, contributed by two of my students, Cara and Shweta. And they are obviously great students, but also they contributed really cool case studies for us. This one that we're going to focus on this week is a zoo. It's a virtual reality version of well it is modeled to some extent after the Detroit Zoo, but I have to be very careful here. The Detroit Zoo, this is not affiliated with them. The Detroit Zoo was not a client of ours [LAUGH], there's a different story behind it. I was doing an online office hour on Twitch as part of my residential teaching. And I felt like hey, I should really design a virtual reality and I should probably model it after some reality. Even though I'm not a strong believer that we need to copy the real world, but I wanted to have a zoo. And I started out with the basic layout and the animals. And now, well then later at some stage, I thought, hey, this would be a really cool case study for us for the MOOC. So I asked Cara, Cara Daney, my student who helped me, who was a course design assistant as part of this MOOC in the background. But we are going to interview her as well. She helped us create this really cool case study. In fact, she sat down and implemented all of this. So we actually going to learn about a number of things. We're going to focus a lot on environmental design, just in this lecture, but later we are also going to learn how to travel in virtual reality. You see elements of this already in some of the videos I show. We're going to learn about object manipulation. So traveling in a zoo you can imagine, right? You want to see all the animals so you go to different areas in a zoo, that makes sense. So then we were thinking, what kind of manipulation could we do? And so we have this petting and feeding area. It's actually quite a scary area because we have like really big gorillas there. So you want to be careful, not so much petting and more feeding. And probably more about far manipulation [LAUGH], that's the term that I'll introduce later. Then anything coming too close to them. I want to be also again, very careful here, we're going to look at this, I do say Detroit's Zoo from time to time because again, this is one of the smaller after. It just started in an office hour as I said, this is not approved or somehow affiliated with the Detroit Zoo. In fact, the Detroit Zoo, we talked to them, they have very high values, and a petting and feeding area is actually a no go. Now I thought that is very important for us to know. I totally understand animal welfare. They're making a really good case there. It would be very disruptive to the animals and it would be actually not good. But we can do this in virtual reality, without harming anybody. So here is a screenshot of what you going to see in a second. I'm going to actually walk through different stages of design. In the end it became a Night Safari, but it didn't start out as a Night Safari. It looks really quite sad. This is what I produced as part of my office hour, so this is what I had. And I was busy creating a map, we had a little interaction there so you could use your controllers and you could teleport around, that's relatively easy to do. But then Cara came in and Cara, she turned into something beautiful. You can see the basic elements that I had the stool there, but now we have animals there lots of animals actually. Everything here is based on free publicly available models. So we didn't actually model anything in here. We created a menu so we can actually navigate in this world. And this is the part where we learn more about object selection and travel. And so navigation and menus, and then object selection is here, and manipulation. So we have this [LAUGH] As I say, feeding area, we have monster bananas that you can give to really big monkeys. [SOUND] Is what I say here, I'm going to share this case study with you as well if you want to, I'm going to do a walk or I'm going to take you on a 10 minutes safari. [LAUGH] You can listen to me and learn a little bit more about the ideas behind this project and it's a cool case study, I'm so glad we have it, so I'm just going to say thanks again to Kara, we're going to talk to her later as well. This is a beautiful VR so we have a very nice birds here and really quite cool areas to explore, putting it together with this night safari which is something I really insisted on, [LAUGH] Inspired from a few trips to to zoos, I'm a big fan. And sort of Night Safari is also that idea of the Night Safari, so turning this into a torch light, and also having a laser come out of it, that was actually pretty good for our case study because you learn more about light and and this idea of really giving agency to users and adds to this explorative nature of Virtual Reality. And even though everything in there, you probably know say this Michael, if this is it, if this is your zoo, then that's a little disappointing. Okay, then maybe you have very high standards, maybe you're used to going to the movies and watching Disney movies, so keep in mind that was not the goal here, this is a design for aesthetics, although I do think it looks quite pleasing. It's not high quality high polygon counts, actually all three models, but I suspect that when you get started in this space, this is how you will work, you won't be this Disney artist, maybe if you are, that's cool for you, [LAUGH] I'm jealous. But the majority of us are just throwing things together and maybe you can find we need this model and that model and maybe we work with somebody who can model things for us. So, remember my what doing XR actually involves lecture was talking about our Disney project, so I was just thinking back to that, if you don't know what I'm talking about second course week one, check it out. All right, let's go and talk about what we're going to learn, so Virtual Reality content, okay, so we're going to learn about a few things, what does it actually entail to create a virtual reality? So if you think about it in terms of just content, yeah, 3D models and animations, sure, we're going to put some 3D characters in our world, we're going to have a story that way, [LAUGH] These 3D characters are actually the main ingredients, okay? They can be in the foreground and in the background, in fact, they can actually play a really cool role in the background just for atmosphere. Lights and shadows required for rendering obviously, [LAUGH] Well, I say obviously, but that's something we've touched on in the 3D lecture last week. So, yes, we need light in order to see something, an ambient light, and then we have directional light and then we can have spotlights like we do for the torch light here and, can give visual cues to users very important. The environment, so within that terrain or like whatever you walk on or what you see and this environment like trees, rocks, lots of things, really cool tools in unity, but even A-frame has an environment component that can be configured in many different, interesting ways. So that's usually sufficient to create this kind of like presence in a virtual world, and physics is really important, like if you want to model after the real world where we have physics, at least on most planets that I've been to, which is actually just Earth, make or break the VR experience. So if you don't get the physics right, if there are missing and users are expecting them to be there, not so believable, so that's really something to take into account. Spatial sound, so audio, 3D audio, whatever you may call it, spatialized sound, so that's more than stereo, okay? Really truly spatialized sound would, let's say if there were an object in between us right now, you would not really hear me anymore, and than as that object goes, I was just saying you wouldn't hear me really anymore, that would make a difference, okay? So bringing an object and obstacle in here would manipulate, modulate the sound, and if you like in a big room, you would expect it to echo for example, and if you're in a small constrained room, you can make it feel very tight. So all these things coming together, don't forget about the sound not just the visual, so it's actually key to immersion, your experiences with sound much more believable and actually something is really missing. But initially when you start out and you're designing for able bodied and you yourself are able bodied you probably predominantly design for the visual senses, but don't forget about audio. So menus enhance really they work actually differently in via VR, so heads up the space is something that really just makes sense in Virtual Reality and also in augmented reality, so we're going to learn about this as well. Well as we go through these content this week, so allow users to trigger functions navigate and customize the virtuality experience, especially the menus. So another way to think about it and these are also some things that we will actually learn about is virtual reality and interactions okay? So as we need to distinguish at the highest level between selection and manipulation. So on the selection side, very common. It's is to have kind of like a virtual hand. So your controller can show the controller or an approximation of that physical controller. But we can also give you a hand model. It's very common. And now we actually have hand tracking. So it will almost look like a real hand like a hand and instead of a virtual copy of this and the in between solution has been that you show the controller. Or you then replace it like remember when we were entering virtual reality where I am not sure whether you have seen my first lecture really on VR concepts, but I was going in the Oculus lounge was going through the Oculus tutorial actually. Then, due to the controller design and how the buttons are placed, we know roughly which finger is where we can detect, we can make these sensitive, which is what they did. And so we can detect where the fingers are. So we can actually render a hand model that does interesting pointing thumbs up, or whatever kinds of interactions, even if you don't have hand tracking yet, but hand tracking is becoming more or less the standard. You can have a laser pointer. We're going to talk about that. We're going to talk about different kinds of manipulation or grasping to really change objects in the world. That's really exciting. It's a little bit more challenging, but it's possible. And then navigation or travel, so it can be gay spaced. So what I mean by that is you travel where you look. So we're going to learn about this and half-life Alyx, they actually really did this. Where you're looking is where you're walking. It's very interesting. But what I'm also thinking about is, let's assume you're not rich, we don't have all this stuff here. Which is, I wouldn't have all this stuff if University of Michigan didn't give it to me. I would probably just have this. And many of us will just have, maybe not even the virtual reality headsets that I show from time to time like the Oculus Rift while the rift s or the quest and maybe many of us would just have this, and so gaze based travel, and so even just three degrees of freedom is something we can implement and it's still a form of virtuality, you can still have quite immersive experiences in there, or in the Oculus go, which has been discontinued, just like the day dream. So no more three dogs. Teleporting so use the controller to activate the travel function. That's something you can do. And then obviously you can do menus and we learn about menu designs, you can fix them, so anchor them in the world or attach them to the camera they can follow. So it's a little bit of delay just like you moving your head and then it's coming rather than you moving your head and to attach to your gaze. So it's always there I like this following more. And so it's very common. For example, on the architect is not so much but they actually have a fixed menu, at least in the latest version there are as changes all the time, design partners. On the HoloLens, we have this follow menu. And so some of these ideas and concepts really translate nicely between AR and VR. And then some of the things some of the things we talked about are really just unique to VR or AR. So, we're going to do this case study, we're going to look at it step by step. Okay? So here in many when you're designing a virtual reality, yeah. One of the things I did was actually just put in an environment just to have it a bit of a few people that are actually enabled to travel. So allowing teleportation, and then I was thinking in terms of zones. And so I was just planning out the map and then gradually Kara came in and she made it beautiful. Gave me some animals, played a more with,light. We added the menu, we going to talk about this more in detail in the next lecture focused on menus and navigation. So this menu then allowed us to navigate in this world in addition to teleportation functions, so common travel functions and then we also have a billed as I said into the zoo a feed the animals area. This is completely fictional, nothing that you can do in the Detroit Zoo that this was modeled after and then we put the experience together and it actually looks really beautiful. And so, I mean, from my perspective, some as someone who is not a true 3D artist, and I enjoy working with people who are really good at 3D artists 3D art and 3D design. But I'm personally not I'm more of a developer and engineer. So we also got to learn about object placement. We have actually really important key terms here that we need to learn about. When you place things in the virtual world. We have basically At the highest level, four different ways to do this, okay. Especially when we're using WebExr, okay. So, and this is not just related to virtual reality. This is a little bit more, some of it doesn't make so much sense in much reality. So it's more like a XR based. So anyway, you can place things in the world. So the object actually exists within the 3D world this is called diegetic. They are part of the story okay. So they can be anchored in a fixed position or just like somehow floating in the world and then following you but they are like, essentially in world rather than in screen space. So screen space means there the menus will actually be floating so the object is in 2D over the 3D scene. So could be attached to your heads up display to the virtual camera, and this would be non diegetic. It doesn't participate in the story. It's just visible to you. So in half life when we look at the menu that is really a non diegetic menu. This is really separate from the story it shows there that you bring it up to configure the game and that makes sense. Then we also have obviously options of putting objects in 2d and the web page. And so while you're thinking, Hey, but this is not in virtual reality. Well, on the web, when you're working with virtual reality on the web, you can always exit out of VR and enter VR. You can actually have very interesting transitions between the 2D web and the 3D web and then especially virtual reality. And so any kind of placement\g in the DOM or page means it's actually not visible in virtual reality. So it's actually separate. It's separate from the canvas that renders the 3D scene. So keep that in mind. If you're like working on a Webexr solution and using a frame, put the stuff inside the AC, not outside the AC, and you, this is part of the honest track. So it's an invitation to come there. We're going to hackle and understand this better. And then attached. So you could actually, is a combination of the above, so you could have the object anchored within the user's gaze, controller or hand. So could be head gaze. You could even do eye gaze very soon. I mean it's not that standard yet but it is coming to the to VR devices as well as narrative devices. We really deal with object positioning and sizing a lot. I think this is one of the key things when you're transitioning from 3D, just plain 3D from last week to this week thinking about VR and just porting your scene and seeing it the first time in virtuality is going to be cool, but it's about actually be catastrophic, because that might be just too big. So we will need to think about the position, the rotation and the scale of objects, and really have to understand how these things are defined as a unit of one, and may translate, at least per default depends on how on the resolution of your virtual reality. It can translate one to one to the real world. And this is what we expect at least in augmented reality. But we can usually scale the virtual world and so that scale, and then the world scale determines the resolution. And so that's when things get messed up. But it allows us to overcome portions of that world. So it's interesting. In any case, the location of the object in absolute 3D coordinates, when I say position, this is what I mean. So think meters away from the cameras calibrated startup location. This is especially true in AR and but also in VR and sometimes it's not really clear where the world starts. It has to do with and what you do sometimes when you press the home button on the Oculus, press the home button like put forward press the home button longer. It really centers everything this way, so this will be your starting location and then the scene will be laid out in front of you this way. If I go like this, then everything will be appearing this way. This would then be zero degrees, not like 30 degrees from that previous example. So keep that in mind, rotation the angle of the object. So this is really when we were doing a way of moving objects around the y axis so I'm going to give you this example, moving objects around the blanks would be this way, right. Okay, and x and z is a little bit more complicated to show [LAUGH] I'm not going to do that. So just stick with y for now. But this is something you've learned last week and probably I'm sure, we use a bit more too much time. Think in terms of degrees or radians and radians is a little bit of a harder concept mathematically, it's used a lot internally. So maybe something you'll come across, especially when you move out of A frame into three JS. And so it's essentially the in degrees the difference from the models orientation. And keep that in mind, it's going to be very important. Let's say you have a virtual reality model of Michael. And he put the people point here and then you rotate me around the X or something. And then that would be We, why we rotate. If you put it at the bottom of my feet, then it's more like the whole body is moving, if you see here, okay? Pivot point here, pivot point here. Do you see the difference? So keep that in mind, and if this doesn't make sense right now, the you haven't actually done anything in VR yet. This is really key in 3D in general, but especially in VR. Keep that in mind, especially when you work with lots of different models. Cara can sing a song about that. So then that is really key. Scale finally, nobody really knows what 1.0 scale is with models. Every model comes in their own original scale. So while we have this really nice Zoo here, if I put everything at one zero scale, this is what it would look like. I actually went in there and looked at some of the issues when Kara was struggling. She was really struggling. A little complaining maybe about some of these Google poly models where they are cool and hard to work with. And this is everything normalized, okay? But this doesn't mean that they all have one zero scale. [LAUGH] If we normalize it in 3D software, then yes. But because these were modeled by various different 3D artists in whatever software they were using probably blocks and yeah. Nobody seems to pay attention to coordinate systems and people points and scale. And so this can be very painful. So keep that in mind. All right. I think I want to give you a few tips at the end. So when you design your virtual reality, you should think of it in terms of designing, in terms of layers,okay? So there's a foreground where you put the 3D models, spatial sounds or that's really the stuff that you see really close up. There's a mid ground that is already adding to the atmosphere. So in terms of visual, I would say that's a terrain, any kind of environment, maybe some trees in the background, maybe three years, let's say in the mountains, the rocks, like I grew up in the Alps in Germany. So that would be the background and the background here would be a skybox. So a 360 sphere at 360 sphere is usually the outer bounds of our virtual reality or a cube depending on which software you're using a could also be a cube box, but it's like a skybox in general. And so ambient music I would put, so it's not like really, like if you're putting an animal and that's foreground. If you have some kind of birds in the background or whatever, we have this rain forest, sound, that would be Amede music, it's usually on loop. It doesn't react much. It could still have some kind of additional background sound as well. And so, I would distinguish as I was trying to explain these layers in terms of what that means for the 3d model. For those with the 3D content and then also the sound. These are all the things you put in the foreground. These are the animals you can see some nice giraffes, this is the first thing I always look at, there's zoo plate here and then we have the mid ground, okay? The rain terrain, in this case this terrain comes with trees, they call this dressing, it's part of the AI frame environment component. You can configure that you can style it doesn't have to be green, can be winter, can be whatever it can be more abstract, and but you see the difference. Creating more atmosphere and then bringing all the light and additional actually a skybox. So we're changing the skybox here, and then also we're doing a bit more to be honest, this is the final design. We're also bringing in some of the light here. And it does. It's like, I mean, it doesn't exactly look like these really cool when they show you how to do it, how they made the Marvel movies, but it's close. Isn't it? [LAUGH] Okay. This is when we really put everything together and additional layer to really have some kind of light in the background. And we changing the sky box a little bit more. So this has been, you can then tune forever, but if you also had a deadline, so I'm ready happy with this example.
