Imagine that you could turn your Minecraft project into something realistic. Turn the house you made, the city you made or the world you made into its realistic counter part. It would be cool to say the least. Making a realistic scene would require a lot of processing power, time and a whole lot of manual input from the user. Services like RebusFarm are a great way to simulate whatever you imagine.
But what if there was a way to just put in pictures of your Minecraft world and get their realistically rendered counter parts in return? All done automatically of course. This new neural rendering framework, GANcraft does just the thing. It is able to generate photo realistic images of large 3D block worlds like the world you would create in Minecraft.
The current method converts a camera trajectory in Minecraft to a sequence of realistic images as if those pictures were taken by a real camera somewhere on Earth. The applications for something like this are endless. It could help artists visualize their concepts more easily. It could help graphics rendering in games, taking a rough blocky sketch of the environment and turning it into something real.
The GANcraft model was trained using unsupervised learning techniques. As there were no real photo and 3D block photo pairs datasets. So the model was trained using pseudo-ground truth and adversarial training. GANcraft allows user control over both scene semantics and output style depending on the 3D blocky image fed to the model. Each block is assigned a semantic label such as dirt, grass, or water.
The world is then represented as a continuous volumetric function and the model is trained to render view-consistent photorealistic images for a user-controlled camera. You can learn more about GANcraft from the video below.
The finer and technical details have been discussed in a research paper submitted by the team behind GANcraft which you can read here. The results shown by the team are pretty impressive. The model is able to basically, from scratch, make a whole real world out of it, coupled with realistic grass, greenery, and even water.
Guess it’s time to how the 4 block I placed in Minecraft years ago look like in real life.