Photogrammetry of Pinecones in Reality Capture
Backyard Nature to 3D Mesh with an iPhone 14
I have another page on my site that talks about Unreal Engine, a workflow I am learning a lot about and looking to integrate in my future work endeavors. During the time I was watching a lot of tutorials on that setup I found some connections back to the practice of Photogrammetry. What is it? It is the practice of taking a real world object, one that would be pretty difficult to model because of it's organic nature, and capture it through a series of photos to import into an app and through those photos, the app has the ability to build 3D mesh from the object and apply the captured photos as a seamless texture map. When I first started this experiment, it was a neat trick I wanted to try, by the end I really couldn't believe how incredible it worked. The craziest part to me is that I was able to capture the objects with a cheap LED light kit, an iPhone 14 and 200-300 pictures. Incredibly simple and incredibly practical. The photos below are a few of the photos captured of a pinecone I found in my back yard.
First Pass Was Half Success and Half Failure
My first pass through the experience resulted in a much better outcome than I had imagined. I had tried a freeware version of the technology with no affect at all, no mesh, no nothing. I was then turned on to Reality Capture, which is owned by Epic who also builds Unreal Engine. I took my stack of early photos and added them to the app and was shocked at the results I got first time through. I was getting some pretty crazy details and very realistic textures. It wasn't quite enough, so I went back to YouTube and Google Search to find my shortcomings.
Adding Detail to Ground Plane and Surround Makes Success a Reality
In my search for understanding, I came across a YouTube video from an engineer who worked on the Reality Capture app. As I watched his tutorial, I found out that the more minimized an environment I had in capturing my photos, the less likely the engine would have in finding tracking points to line up all the angles properly. I was originally hanging the pinecone with a piece of fishing line and trying to block out the background with slabs of white board, which is a logical approach for green-screening or isolating an object for digital manipulation, but in this case it was working against me. The engineer from the tutorial was mentioning using newspaper or magazines to get better results, so I changed my setup and reshot the pinecone and sure enough, much better and cleaner results!
Last Hurdle Overcome With Top and Bottom Photo Collections
The last major issue I was struggling with was the ability to be able to use the object from any direction. My more successful attempts still picked up the geometry left over from the Canon box that the pinecone was sitting on. Again, I went back to YouTube to figure out what I was missing and how I could overcome that last bit of garbage geometry I wanted to get rid of so I had a completely independent 3D mesh. What I was able to find was a tutorial about building a top set of images, then flipping the object over and capturing a bottom set of images. If done right, composited correctly, setup in the app properly, the end result was to be a completely whole, very realistic, 3D mesh of the object I had photographed.
LOD Exports, Textures and Unreal Engine for the Final Look
After about a day of struggling, watching YouTube videos, a little bit of cursing and frustration, I had success! I was able to create an incredibly realistic 3D mesh of an organic object that could have easily taken me at least a week, more likely two, to model and probably much longer to realistically texture. The final step was to export LODs (Level of Detail) and import the final piece into Unreal Engine to see how it would render out. Not only was I able to successfully get the new mesh into Unreal and get the texture setup properly, but I was able to repeat the process for more pinecones to add variety. Once I got the handle on things, it would take me about half a day to create one from beginning to end. The end results are below and I was truly shocked about the quality of the object after being digitized. My brain started racing with ways I could use this in other 3D work I wanted to experiment with!
Environment, Lighting and Multi-Camera Setup to Test Realism
As I created more pinecones, I really couldn't believe how good they were coming out, so I wanted to do a multicam test in Unreal Engine to see how far I could push the realism. As I moved cameras near and around the objects, it was very hard for me to actually pick out the geometry in the pieces. I am not sure I can conclude 100% photoreastic, but it really does feel quite close. The images and edit below are animated and rendered in Unreal Engine with no post-processing done outside the app.
The Devil is in the Details
One last thing I wanted to try, when building this page, was to really push in on the geometry and get some closeup shots to see how far it could go. Although my displacement could be better, the details in the geometry is just staggering. The screenshots below are straight out of Unreal Engine, no editing or sharpening in Photoshop. Pretty incredible. With some more understanding of Unreal Engine texturing, I feel confident I could probably push these even a bit further.
Is there a downside to this all? Yes there is. First of all, it captures the geometry as one big mesh. If I had parts I want to animate, I would need to separate those into their own mesh and, considering how complicated the translation process can be, it could be very time consuming. Second, the mesh is very complicated, so it's far more resource heavy than if it were modeled. These could be used in a Cinematic project, but would cause issues in a game environment where optimization and memory management is necessary. But overall, this is a great way to add some organic realism to a scene in way less time than modeling it from scratch.
