Metro Re-illustrated: Incremental Generation of Stylized Paintings Using Neural Networks

Update: Our work has been featured in a SIGGRAPH Blog, which is the first time that SIGGRAPH Blog publishes both in English and a language of the contributors' choosing. Read the Chinese version here.

Examples of paintings generated by Metro Re-illustrated of Shenzhen and Chongqing.

Metro Re-illustrated is a project that explores incremental generation of stylized paintings of city metro maps using neural networks. It begins with an interactive system for labeling time-series data on city metro railway maps, which is then used to generate reference images. These images are fed into a neural painter that incrementally generates oil painting-like strokes on a virtual canvas. The generated paintings demonstrate blending and layering features of oil paintings while also capturing the progressive nature of urban development.

Video can be accessed here from Mainland China.

Paper

Accepted to the 50th International Conference & Exhibition on Computer Graphics & Interactive Techniques (SIGGRAPH'23) Conference. 🔗

Blog

City Maps, Art, and Neural Networks, ACM SIGGRAPH Blog

Exhibition

Artistic Realm Uncharted: Designs Inspired by Art History, Principles of Art Course Exhibition, Tsinghua University & University Town Library of Shenzhen

Introduction

In this project, we explore incremental stylized paintings generation of city metro maps using neural networks.
Firstly, an interactive system is built for labeling time-series data of city metro railway system on the transit map and generating reference images for the upcoming painting generation with data points on the corresponding satellite map of the city.
Secondly, the data gathered in the labeling process consists of a novel dataset with a uniformed structure for city metro data with time-series tags, providing convenience for future exploration.
Finally, the incremental generation of strokes on the virtual canvas exhibits natural characteristics of oil painting and the dynamic development of the city metro system.

System Framework

The p5.js painter labels and draws data from city metro maps onto the satellite map image of the corresponding city. The resulting generated satellite maps with data points are then used as reference images for the neural painter.
Using the reference images, the neural painter generates stylized oil painting strokes on an empty canvas. The resulting painting becomes the base canvas for the next iteration of the neural painting, and this process is repeated until all the reference images have been painted.

System Framework of Metro Re-illustrated

Data Processing and Reference Image Generation

We built a data processing program on p5.js, providing interactive map retriving and data labeling experience in the browser, shown in Figure 3. On the left side of the system is the map for labeling, while on the right side are control buttons and a display area for showing the current labeling status and line information to the user.
In the labeling process, the user clicks on a subway station on the map, and the program temporarily stores the station's coordinate information. The user can switch between the current annotated lines forwards or backwards until all lines have been annotated. After completing the labeling, the user can visualize the annotated data on the map and export it as a JSON file of the city's subway lines data.

Data labeling tool built on p5.js

Metro lines and station data are classified by city, and their data structure is shown in Figure 4. For each metro line in each city, it includes the name, representative color, start of operation time, operating interval, and the coordinates of each station on the route.
Specifically, many lines are opened in segments during theconstruction of the city, so in the dataset, multiple records with the same name of the line at different time points may appear. The operating interval of each record is generally continuous, which reflects the temporal and progressive characteristics of urban development.

Structure of the labeled city metro data.

After the data labeling is completed, we fuse the lebeled data with the city satellite map to generate reference images for subsequent oil painting generation, as shown in Figure 5. First, we convert the coordinate points in the data to pixel coordinates, then use the corresponding line color to draw randomly sized circles on the satellite map, and finally save the drawing result as a reference image.
During the drawing process, we use a time-based drawing strategy, that is, we first draw the lines that started operation earlier, and then draw the lines that started operation later, and after each line is drawn, we save and download the current image. This approach ensures that the hierarchy of lines and their changes in the reference image are more distinct.

Process of generating reference images, where i is the number of metro lines.

Stroke Generation and Animation Encoding

The stroke generation is completed by a neural painter which applies strokes on an virtual canvas and calculates the loss between the generated image and the reference image to update the parameters. In the stroke generation process shown in Figure 6, we started with a blank canvas and first try to paint the satellite map image, then in the next iterations, we replace the default canvas with the generated image from the previous iteration and try to paint the next reference image. This process is repeated until all the reference images have been painted.
For each reference image, we iteratively generate 110 strokes on the canvas. The thumbnails of the canvas after stroke 1, 10, 80 and 110 are shown in Figure 6. Since we use the generated image from the previous iteration as the base for the next iteration, the painter utilizes the background layer to predict strokes. The resulting strokes exhibit a color blending feature between the background and new reference images, which enhances the authenticity of the generated oil painting.

Process of generating strokes, where i is the number of metro lines.

The generated strokes are then encoded into a sequence of animation frames, which are then used to generate the final video. The gradually accumulated strokes in the animation further mimic the process of human oil painting and exhibit the temporal and progressive characteristics of the city's development.

Summary and Future Work

Currently, the generation of station points in the reference images is randomized. Therefore, additional data about each station, such as passenger flow and number of routes, can be input into the system to present more dimensions of information.
The stroke generation process can be further parameterized and made differentiable, so that gradient descent algorithms can be used to optimize the generated strokes and to regenerate the painting during the entire generation and training process.
The efficiency of the painting generation process can be further improved to combine with the data annotation process, creating a more real-time painting generation experience.

References

[1] Processing Foundation. 2023. home | p5.js. https://p5js.org/copyright.html. Accessed: 2023-04-23.
[2] Dong Lin, Wout Broere, and Jianqiang Cui. 2022. Metro systems and urban development: Impacts and implications. Tunnelling and Underground Space Technology 125 (2022), 104509.
[3] Zou, Zhengxia, et al. "Stylized neural painting.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. 🔗