This is a public version of this post. If you are a Taylor employee, contact me for the link to the internal private version of this post with strategic commentary.
This summer I decided to work on a Virtual Reality 6DOF (six degrees of freedom) proof-of-concept. This POC took between 30-40 hours, an hour or two in the evenings here and there over the course of a few weeks. Here is a video demonstration.
The key things I was trying to prove were that:
With newer 6DOF room-scale VR systems:
- Can we build a compelling experience based on user photos with a resolution quality that makes it engaging? Up till now, resolution in mobile VR mediocre at best.
- Can we configure a product realtime and make the 3D model accurate enough to match what is produced as well as realistic enough for someone to actually order it all within VR?
- What is the level of runtime texture manipulation available in Unity and other 3D engines since that is really key to configuration and is it fast enough?
- Any limitations with consuming our KLIC Photo API’s (memory, performance) since that is crucial to any solution?
Summary
My initial goal was to just build a Virtual Art gallery from your Lifepics photo albums but this quickly turned into more than that. This app begins by generating a Virtual Reality art gallery based on photos dynamically retrieved from your KLIC Photo Albums via the KLIC Photo (Lifepics) API. You can actually walk through the gallery using positional tracking and there is some background ambient noise from the National Art Gallery in London so it really feels like you are in a gallery/museum. The gallery is randomly generated each time you come in to keep it fresh.
It also has a dynamic product generator that will create a personalized photo product (coffee mug) you can hold, examine, play with and purchase right within VR based on the photos from your photo library! Doing this requires real-time manipulation of the base texture for the mug to inject a photo correctly and get it to match what will be produced. I am not very good at creating 3D models but I found a coffee mug asset that has a photo area with an aspect ratio very close to our Lifepics 11 oz mug. I was able to tweak this so it matched and it absolutely matches production (I verified this in MediaClip). This is very doable as long as the model created follows some guidelines in how the UV map is created (more on that later).
Buying the product is as simple as placing it on the shopping cart pad. It detects a generated product and allows you to check out with your default shipping address in Lifepics. Similar to the Alexa Skill prototype I created a few years ago, if you want to ship to a different address instead you can simply add the configured item to your cart and you can complete checkout on your computer or phone. This is thanks to the fact that KLIC Photo is an omni-channel ecommerce system.
I also started creating the beginning of a game that demonstrates the capabilities of Unity (and other 3D engines) to manipulate textures. Essentially, I randomly pick a photo from your album and make four cubes out of the photo which you have to put together as fast as you can to form the original photo. This needs some additional development before it is complete but the idea was to try doing some additional texture manipulation in creating other objects including crop, scale and slice.
This looks SO much better in the actual headset versus what I can show you in a video or screenshot, but if you have played with a newer 6DOF VR device before you will get the idea. The gallery canvases actually look really great. It is neat seeing your photos enlarged in a “real” space and being able to pick up, play with, closely examine and order an actual product with full positional tracking. I spent a lot of time trying to make the most of the hardware so the canvas art and product look very clear up close. Definitely good enough to decide to buy the product!
History
This picks up from the iOS CVS AR prototype I did a couple of years ago for Lifepics but this is in pure VR. Taylor already does some VR stuff, but appears to have been restricted to 3DOF so far. This is probably due largely to the fact that there hasn’t really been an affordable 6DOF wireless VR solution that did not require a monster PC. That changed last month with the release of the Oculus Quest and it opens up a lot of opportunities (they are selling like hotcakes). There are a few things that differentiate this VR demo from others I have seen here at Taylor:
- Rather than just looking at preconfigured products, you are configuring/creating a product dynamically based on user assets real-time.
- You can actually add the product to your KLIC Photo cart and complete purchase right within VR thanks to API integration.
- This uses complete positional tracking so you can grab, play with, examine, stack and even throw the actual product you built rather than being limited to gaze selection with 3DOF VR. This allows a much more immersive experience where you can be more confident about what you are buying.
3DOF is kind of disappointing to be honest, but with 6DOF you can completely move around as you would in the real world and actually walk around a room, grab and pick up objects, bend down, etc. The art gallery is a little too large to find a space within the limits of room space for this device, but you can traverse some of the gallery simply by walking without resorting to the joysticks. The gallery scene is about 7×7 meters but it does not REQUIRE a large space to use.
Some Technical Notes
I built this using Unity and while it is targeted towards an Android platform (Oculus Quest), it could be retargeted to a different platform like the Vive or the Rift that also support 6DOF. Within Unity, I am using C# code to consume the API and do all the dynamic generation of art/models. For some things like network calls, I recommend using the Unity classes versus .NET. This makes porting the app to other platforms easier (some may not support .NET networking stuff for example). The canvases themselves are adjusted realtime based on the aspect ratio of the artwork and available wall space and are created using QUADS but those could easily be 3D models of canvas prints we sell. That was just too easy, I wanted a product with that was truly three dimensional that offered a challenge. The product model (coffee mug) is dynamically created based on adjusting UV Maps of the surface area where the personalized photo appears. This involves a LOT of runtime image manipulation. This is something that probably hasn’t been done in the past at Taylor, namely creating and adjusting dynamic 3D product models in VR with textures on-the-fly based on user assets via realtime API calls.
I am making heavy use of light maps (baked lighting) for performance reasons on the snapdragon processor (I would be nice to have the latest MacBook Pro to reduce bake times…hint…hint). The trickiest part of this is finding the right line between performance and quality which took trial and error on my part. This includes things like the resolution of the downloaded image from Lifepics (Lifepics can adjust on the fly), proper Anisotropic level on textures, mipmapping, dynamically changing eye resolution for both eyes and proper disposal of Texture objects which rapidly eat memory and are not covered by regular garbage collection. For 6DOF VR to be compelling (and not induce sickness), frame rate really needs to be solid. This requires effort rather than just slapping a scene together.
I am not a 3D modeler so I utilized assets from the Oculus Store and edited them as needed. Oddly enough there are actual VR apps where you can design 3D models in Virtual Reality itself (see Gravity Sketch).
I learned a lot of technical things regarding Unity/Oculus integration that would be helpful to others.
Special thanks to Jeff Wieman for helping me with some API/Product Setup things I uncovered doing this POC! Also thanks to everyone on the KLIC Photo team who volunteered their login credentials so I could test with different photos!
Interested in using KLIC Photo / Lifepics for your business? Please contact us at sales@taylordigital.io! Our platform is an omni-channel photo-centric ecommerce platform that covers a wide range of businesses from grocery and pharmacy stores to niche photo solutions.
