Interactive Sketching of Urban Procedural Models – SIGGRAPH 2016

We present an interactive multi-stage stroke-based system that combines sketching and procedural modeling. Our system allows untrained users to create realistic buildings using a tablet or a mouse. Using a deep learning approach, we can find the grammar and the parameters in real time. This allows the user to sketch and edit the model interactively, and, since it is a grammar, we can create
a wide variability of 3D models from a single sketch. Normally, in procedural modeling, a grammar and a set of parameters are used to generate a 3D model. It has the advantage of the detail amplification, few lines of code can generate numerous examples. However, the parameters are hard to control and the complete grammars are tedious and complex to write. Our methods tries to hide all this complexity and manual tuning. The user can sketch few strokes and obtain a complete 3D model. We first run an offline training. Then, in
runtime, the user sketches the building interactively Instead of create training examples manually,
we use our procedural engine to generate images using a custom renderer. For each grammar snippet, we create hundreds
of samples varying their parameters. With the generated samples, we train two types
Convolutional Neural Networks: one for recognition of snippets and the second one for parameter
estimation. Then, the user sketches a building and our
system finds the grammar snippet and the parameters within milliseconds. Iteratively, the user
sketches the different components of the building, and our system continuously updates the grammar that generates the 3D model. We present two sketching sessions. Initially, the user wants to create an observatory.
First creates a dome giving details to the walls and cover
Adds an annex building with specific windows, Then sketches the entrance, with a big gate.
And concludes the observatory using copy. Then, the user decides he prefers to sketch
the Capitol. Instead of start from scratch, removes the
side domes and modifies the main entrance Sketching the proper windows and roofs.
Finally, add the main tower with few strokes and copying and resizing operations. As second sketch session, the user wants to
create a building inspired in the Willis Towers. First sketches the main block and sketches
the windows. Then sketches a tower, then the user can copy,
resize adding floors, and move it. This is possible since we are simply manipulating
a grammar. Then the user decides to modify it such to
create the Empire State. Removes the towers
Resize and center And repeats the process Here are some of our results. Since our system is based on grammars, we
can use one sketch to generate a wide range of variations that are inspired in the original
model. This is ideal to design complete cities where buildings have similar styles but present
different configurations. The user sketches a 3D shape and we compare
our system against the well-known MCMC algorithm commonly used in inverse procedural modeling. Our method finds a solution within few milliseconds,
in contrast, MCMC results has less accuracy and need significantly more time. We have a group of 8 participants use our
system as evaluation. We ask them to create two buildings that look similar to two target buildings. They successfully created them in a very short time. We also asked them to create some original buildings. Here some extra sequences The user can change the parameters directly in the grammar The user can modify the object type by drawing it again.


  • kaliphe2008

    Hi great job!!
    please where i can teste it thank you

  • Jim R. Didriksen

    FYI This is awesome.

  • Fra Stra

    Impressive!! will this tool be available? I really hope so

  • 김효연

    cool. what is tool? i really hope so too

  • Sam Hyde


  • Anastasia Dunbar

    Is every previous SIGGRAPH project dead?

  • Samuel Chu

    it is brilliant

  • Constantine

    This is the future of design software for architecture and not only. One day the software will translate sketches into BIM elements, replacing alot of manual 3D modelling so there is more time for research and design. WIth the advance of computers even more conceptual sketching will be used and interpreted. Possible workflow : The architect sketches his vision as a design direction ( or grammar, vocabulary, style?) and then testes multiple morphologies based on that. Amazing, i hope you guys go forward. I will follow you research!

  • Adolfo Roman

    WOW!! this is amazing! are you guys planning to release this tool anytime soon! you will have my money 😀

  • Usagi

    Amazing ! very promising 🙂
    Best of luck ..


    Is it possible to get the outline of a real image of a building, convert it to sketch into your program, and create a 3d model out of it?

  • Tsin W

    amazing! I find this links in Chinese Social Network Service,most people think this idea is great! especially meaningful for our Architektur Student! can't wait for it! may this project better and better!

  • Colonthree Enterprises

    Jajajaja, I have to try this for myself. It would save a lot of time for my kind of work. :3c

  • MooseyFate

    Mama Mia

  • Uniday Studio

    How can I download or buy the program? Very interesting research!

  • the_ryancardinal

    I have read all the Info, but I am wanting to know if this will be coming out at all? I understand that this might be part of a bigger project, but honestly; this piece of software alone is what I and many others would be using. I am wondering if this will be available anytime in the near future? What you guys have done here would help out immensely in the field of animation and comic book illustration. I hope to hear from you soon and I hope that this piece of the project is not dead in the water, because that would be a total loss to the art community and other industries as a whole.

  • Astrid Alaniz

    would love to see this kind of algorithm implemented in a video game

  • Advection

    Why AI for this? It's mostly a few simple patterns that repeat when modelling buildings…
    Just model 1 example of each unique pattern and replicate it appropriately…
    I guess this is for people that don't know how to make 3D models?

  • mehrdad

    holly fart, it's awesome 😍😍😍

  • warp

    Very interesting technique. Do you think it is possible to generate tyre design procedurally?

  • nivalius

    how on earth did you make it? amazing


Leave a Reply

Your email address will not be published. Required fields are marked *