Meanwhile, the research team used Unreal Engine 4 to create what they called the "semantic map" of a scene, which essentially assigned a label to each pixel on the screen. Some pixels are assembled in a "car" bucket, others in the "trees" or "buildings" category – you get it. Those pixel muscles also received clearly defined edges, so Unreal Engine eventually created a kind of "sketch" on a scene feeding on the neuronal model of NVIDIA. Hence, AI applied visual elements to what it knew as a "car" looked like moving to pixels labeled "car" and repeating the same process for each other classified object in the scene. It could sound bored, but the whole thing happened faster than you think – Catanzaro said the simulation of the car lasted 25 frames per second and that AI did everything that was real.
The NVIDIA team also used this new video-to-video synthesis technique to digitally convince a member of the team to dance like PSY. The development of this model used the same work as the simulation of the car, only this time AI has the task of finding the positions of the dancer, turning them into basic figures and displaying the appearance of another person.
So far, the company's results speak for themselves. They are not as graphically rich or detailed as the typical scene shown in the AAA video game, but video samples from NVIDIA offer lighting in digital cities filled with objects that look real. Emphasis on "kind of". Language-in-cheek Gangnam Style changing the body worked a little better.
While NVIDIA opens its core code, it is likely that it will be some time before programmers begin to use these tools to bypass their next VR sets. It's also sincere, as the company quickly underscores the limitations of nerve networks: while virtual vehicles spread around their simulated city-looking looks surprisingly real for life, NVIDIA said its model was not great in rendering vehicles, as it were Turn around because of their label maps missing enough information. More disturbing for VR craftsmen is the fact that certain items, such as those annoying cars, may not always look the same as the scene is going on; In particular, NVIDIA said that these items may change color slightly over time. They are all clear representations of real objects, but they are far from being photorealistic for long periods of time.
Those technical shortcomings are one; unfortunately it is not difficult to see how these techniques can be used for inconvenient purposes. Just look at the deeply busy: it's incredibly hard to tell these artificially generated videos, except the right thing, and as NVIDIA proved with its Gangnam Style test, his nervous model can be used to create unpleasant situations for real people. Perhaps not surprisingly, Catanzaro looks a lot on a bright side.
"People really enjoy virtual experiences," he told Engadget. "Most of the time they are used for good things. We are focused on good applications." Later, he admitted that people used these tools for things they did not approve of was the "nature of technology" and stressed that "Stalin had photographed people from images in the 50s before Photoshop existed."
There is no doubt that NVIDIA's research is a significant breakthrough in digital imaging, and over time it can help change the way we create and communicate with virtual worlds. For commerce, for art, for innovation and more, it's a good thing. Even so, the existence of these tools also means that the line between actual events and fabrications will continue to grow more substantially, and rather quickly we will have to start thinking about what these tools are capable of doing.