NVIDIA’s AI Creates Beautiful Images From Your Sketches


Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. I know for a fact that some of you remember
our first video on image translation, which was approximately 3 years and 250 episodes
ago. This was a technique where we took an input
painting, and a labeling of this image that shows what kind of objects are depicted, and
then, we could start editing this labeling, and out came a pretty neat image that satisfies
these labels. Then came pix2pix, another image translation
technique which in some cases, only required a labeling, a source photo was not required
because these features were learned from a large amount of training samples. And it could perform really cool things, like
translating a landscape into a map, or sketches to photos, and more. Both of these works were absolutely amazing,
and I always say, two more papers down the line, and we are going to have much higher
resolution images. So, this time, here is the paper that is,
in fact, two more papers down the line. So let’s see what it can do! I advise you that you hold on to your papers
for this one. The input is again, a labeling which we can
draw ourselves, and the output is a hopefully photorealistic image that adheres to these
labels. I like how first, only the silhouette of the
rock is drawn, so we have this hollow thing on the right that is not very realistic, and
then, it is now filled in with the bucket tool, and, there you go. It looks amazing. It synthesizes a relatively high-resolution
image and we finally have some detail in there too. But, of course, there are many possible images
that correspond to this input labeling. How do we control the algorithm to follow
our artistic goals? Well, you remember from the first work I’ve
shown you where we could do that by adding an additional image as an input style. Well, look at that! We don’t even need to engage in that, because
here, we can choose from a set of input styles that are built into the algorithm and we can
switch between them almost immediately. I think the results speak for themselves,
but note that not only the visual fidelity, but the alignment with the input labels is
also superior to previous approaches. Of course, to perform this, we need a large
amount of training data where the inputs are labels, and the outputs are the photorealistic
images. So how do we generate such a dataset? Drawing a bunch of labels and asking artists
to fill them in sounds like a crude and expensive idea. Well, of course, we can do it for free by
thinking the other way around! Let’s take a set of photorealistic images,
and use already existing algorithms to create the labeling for them. If we can do that, we’ll have as many training
samples as many images we have, in other words, more than enough to train an amazing neural
network. Also, the main part of the magic in this new
work is using a new kind of layer for normalizing information within this neural network that
adapts better to our input data than the previously used batch normalization layers. This is what makes the outputs more crisp
and does not let semantic information be washed away in these images. If you have a closer look at the paper in
the video description, you will also find a nice evaluation section with plenty of comparisons
to previous algorithms and according to the authors, the source code will be released
soon as well. As soon as it comes out, everyone will be
able to dream up beautiful photorealistic images and get them out almost instantly. What a time to be alive! If you have enjoyed this episode and would
like to support us, please click one of the Amazon affiliate links in the video description
and buy something that you were looking to buy on Amazon anyway. You don’t lose anything, and this way, we
get a small kickback which is a great way to support the series so we can make better
videos for you. Thanks for watching and for your generous
support, and I’ll see you next time!

100 comments

  • cunt

    Wait you're Hungarian?

    Reply
  • DJ L3G3ND

    good GOD thats impressive

    Reply
  • VenoFuj

    What the fuuuco

    Reply
  • Elsa Debroglie

    Looks amazing for concept artists who want to save time during game design.

    Reply
  • Jacob Swanson

    I don't need tensor cores for this, do I?

    Reply
  • Oz Mol

    2minute papers
    4minute video
    hmmmm

    Reply
  • MelodyZE

    where do we get this?

    Reply
  • SPACE droid

    Looks like my time has come .

    Reply
  • acutelychronic

    this could potentially be used to make completely alien landscapes and cityscapes
    or even hellscapes

    Reply
  • Bit Narukami

    someday it'll make furry porn and i'll lose my niche job that i thought i'd never be replaced.

    Reply
  • Space V

    this is amazing but at the same time sad when it comes to what art is, humans keep finding short cuts to everything these days

    Reply
  • Dan Torrino

    Someday I will be able to make 3D models of my own video game characters, and make my own open world game, with AI generated voices and speech xD

    Reply
  • Shaq Blox

    You can try a demo/low quality version here: https://zaidalyafeai.github.io/pix2pix/scene.html
    Please like this so other people can see 🙂

    Reply
  • GDNacho

    shut up where do i get it

    Reply
  • Beats basteln :3

    can i download this programm and play around with it myself? will you release a version where we can feed data ourselves? that would be awesome! i'd pay for this

    Reply
  • Lord Flea

    when a computer is more creative than you

    Reply
  • baki gyorgyi

    Bojler eladó!

    Reply
  • bwedges

    Oh nice now i can draw windows xp's well known grass

    Reply
  • Kyrieru

    The only artists worried about being replaced by technology are bad ones.

    Reply
  • Piotr

    Bob Ross: Lets build a happy little cloud. Lets build some happy little trees.

    Reply
  • Kettlesimulator E

    yay i can finally get an A* in art

    Reply
  • Riveravi

    For the record: This will not cause CGI artists to loose their jobs but will make their jobs easier.

    Reply
  • Axel Savage

    Heck thats cool

    Reply
  • Michael W. Perry

    Sounds like they intend to put photographers and artists out of work.

    Reply
  • C Game Theory

    You could do this in 3D for easily doing amazing Computer Game worlds. Same technology, but you would need 3D lazer scan samples or multiple angle photos instead of 2D Photos. Is it possible to eventually do it with creatures too? If so we could come up with simple sketches for Monsters/Aliens and have the ai create the skin tone, texture, warts, mouth, muscle definition, etc, etc.

    Reply
  • Jia-Ho Jian

    Suddenly everyone can be Bob Ross.

    Reply
  • Xiaohuang Guo

    Anyone installed it and is willing to upload? The instructions on Github are so unprecise…

    Reply
  • roxE-

    Rip human art

    Reply
  • TrueStop [guarda TrueStop16]

    How to install in Windows?

    Reply
  • RecalledLine

    Who are you?
    im you but stronger

    Reply
  • The Scuf

    over 2 minutes

    Reply
  • Arouxayis

    a lot of people are talking this up. Honestly we aren't at a point where it's entirely photo realistic, if you noticed objects had a pretty wide blur outline around them. I'm not saying this is a bad demonstration because i believe you could still do some really cool abstract work on this that could translate over well in designing backgrounds without having to overuse/rely on source material.

    Reply
  • TIBOX

    And I oop-

    Reply
  • Romar Boer

    I am an environment concept artist and this is our next tool.

    Reply
  • the dantdm shrine

    magyar?

    Reply
  • Hype

    AMD left the chat

    Reply
  • juanme555

    Everyone getting excited , this is fucking creepy af im panic'ing right now.

    Reply
  • CreedFN

    Can you download this?

    Reply
  • Tony Woods

    Artist: *going for a morning walk

    Unemployment: " On your left "

    Reply
  • Hard Boiled

    Train an A.I to translate nouns into shapes, verb into motions, etc and input it to this and say goodbye to shitty human-made adaptations

    Reply
  • Giacomo Tombolan

    ARE YOU FUCKING SERIOUS?

    Reply
  • Brock Madigan

    ya, all that creative brain synapse stuff is over rated.. Humans will devolve into globules of advanced mindlessness.

    Reply
  • 1NDecent

    Great tool! I can use that to figure out what my daughter's doodles actually are.

    Reply
  • Baleur

    The sick thing is, this is almost like what your brain does in real life……….
    This is kind of like, how an AI algorithm would visualize the world. And in some ways, if it has enough fidelity, its own "imagination" of how the world works, isnt more valid than ours we generate in our own brains.
    Granted, humans have far higher fidelity, we take "super high res sample data input" in the sense that we get very fine light information via our eyes, but our brains DO interpolate and transform this data into out best "guess" for what it is we're looking at, based on native established parameters in our evolved brain, lots of fine lines of green data is interpreted as a field of grass, for instance. This is what the AI algorithm does too, but with far lower accuracy and input resolution.

    But my point is this, the AI's interpretation, it's "guess" at what the world "looks like" based on the input data, is the same thing we're doing with out eye input, in our brains. And in some sense, who's to say which one is more accurate?
    Perhaps in the future, given enough training data and input resolution, the AI algorithm will generate a more accurate representation of reality, than our own eyes and brains can.

    Reply
  • Gabb

    HOLY COW i wish i had my hands on this program…

    Reply
  • Parker's NBA Videos

    Is the narrator's voice also AI generated?

    Reply
  • Floh

    That's ridiculous

    Reply
  • Sarthak Agrawal

    The future's here already.

    Reply
  • TheEeefs

    Amazing. This should be called the Bob Ross.

    Reply
  • MindTech

    wowowowowowowowowowow

    Reply
  • TeddySpaghetti

    AI can fake faces.
    AI can synthesize Joe Rogan's voice almost perfectly.
    AI can now render entire scenes like this.
    Can AI create any movie with any actors, eventually? Actors and celebrities can upload their entire body profiles as data and create timeless, immortal versions of themselves captured in that exact moment. Maybe with a few updates before death, maybe not. (I'd only hope Lucy Liu could get in on it before she's too old.)

    Reply
  • OHYtheAwesome

    This is some awesome stuff, just imagine pretending to be an amazing photographer when you really are not.

    Reply
  • Colossal7

    Trying to find someone being WOOOOSHED

    Reply
  • Cee G

    scary

    Reply
  • Teckzus Feralupus

    Its kinda scary to see an image that I think is real, but it's actually a computer-generated fake…

    Reply
  • Aaron Wyatt

    The death of the technical aspect of art wasn't really what I had in mind going into the 21st century, but…alright…

    Reply
  • laskieg

    Autotune for artists.

    Reply
  • Harry Tsang

    Can we make extremely low texture video games and use this AI to create realism?

    Or better, use as pairs of original minecraft and raytraced minecraft as input data pairs

    Reply
  • willing4sth

    So finally police can use the drawings made by the victims :O

    Reply
  • Cuti3Littl3Girl

    At least they can't animate right?? ( I hope i'm not giving them any ideas..)

    Reply
  • david abe

    I mean I guess it's cool… It's basically just sticking images over one another, you even tell it what image to use.

    Reply
  • Samuel Davidson

    That's freakin' amazing, well done NVIDIA!!!

    Reply
  • Csigusz Foxoup

    Wait MAGYAR CSINÁLTA

    Reply
  • chromosoze

    can you make it hyperrealistic

    Reply
  • Ugandan Knuckles

    So now we have fake people and fake scenery?
    Welp, we're officially fucked.

    Reply
  • SpacingCake

    Where can you download this?

    Reply
  • MrAPOD

    The technique is interesting, but it doesn't look good, and its nowhere near realistic. The waterfall was really cool though.

    Reply
  • PiscesAverted

    Isn't this cheating…?

    Reply
  • Al Chandeck Chen

    How can I get my hands on this tool/program you’re using?

    Reply
  • Leerill

    As someone who spends a significant amount of time merging photos to create backgrounds for architectural renders, this is exceptionally exciting.

    Reply
  • Valronic Lehre

    Digital artist here….fuck you.

    Reply
  • Zapper Zapped

    It's like a "dream come true" quite literally.

    Reply
  • MrFirehouse22

    Was going to use Fiverr before I saw this video…

    Reply
  • Wiza k

    I am already confused about which job to get maybe something creative machines can't ma-

    …. ok

    Reply
  • Blue Green Algae 2.0

    This is better than flying cars

    Reply
  • OtakuSanel

    this needs to be used to generate textures for game assets!

    Reply
  • Solve Everything

    needs more objects, such as houses, castles, peoples , crowds, etc etc…

    It should have the entire dictionary.

    Reply
  • Ludak021

    yea, cool. Good luck dealing with copyright claims with this one. 😀 Amazing tho. edit: It is using photos someone has made as "brushes" therefore, if those are not free for use, you have to pay for each photo used.

    Reply
  • WhiteChocolateMocha

    Me : burp
    Nvidia: here is a picture of what u had for lunch

    Reply
  • ten giau

    Reality can be whatever i want
    happy Thanos noises

    Reply
  • Mystic Waveform

    HOLY FUCK

    Reply
  • Nikolai4567

    You all underestimate what this can do:
    Imagine you manage to make this fluent, and then have a 3d mesh scene that only describes the labeling.
    Then let the NN generate the graphics on the fly.
    You could create TRUE photorealistic graphics with this – for all content. This means a game like this will only need a little bit of labeling and everything else will generate automatically.

    Reply
  • Asnickel

    This is just the beginning of the process of creating a Matrix style world. Next, will be 3D models generation. So on and so forth until eventually the machine overlords take control over us.

    Reply
  • adrian5b

    this is ridiculous…

    Reply
  • cowboysamuraiカウボーイ侍

    This is how God made the world

    Reply
  • Khong Minh

    Oh… my… god 😱😧😱😲😱

    Reply
  • Humod Aman

    i want this .

    Reply
  • Ayato

    The voice on this channel is generated by AI

    Reply
  • Daniel Chin

    *What a time to be alive*

    Reply
  • Achmad Zulfikar F. N. H.

    I thought it was react native video 🙁

    Reply
  • Marios Bairaktarhs

    I want this so i can try to make the most STUPID thing imaginable like… Clouds + sea + a stickman made out of lake on the side

    Reply
  • IllI lIIl

    How to install and use it ?

    Reply
  • Can i eat this software

    Reply
  • Shahbaz Sheikh

    People who always wanted to create Video games but were restricted due to their lack of art skill can use this now.
    God bless machine learning.

    Reply
  • Lindsley Daibert

    Are there any neural networks that makes detailed 3D models from a few photos of the object?

    Reply
  • Dalek Metaphorical accuracy not needed

    doesn't work 🙁

    Reply
  • kozirou channel

    nice

    Reply
  • Momo Maximilian

    Where can i get this software?

    Reply
  • Sumedh Pradhan

    Can we use that application of
    Nvidia…

    Reply
  • Lrnd

    this is some kind of sorcery…that's all I can say.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *