MatFormer: A Generative Model for Procedural Materials

Procedural material graphs are a compact, parameteric, and resolution independent representation that are a popular choice for material authoring. However, designing procedural materials requires significant expertise and publicly accessible libraries contain only a few thousand such graphs. We present MatFormer, a generative model that can produce a diverse set of high-quality procedural materials with complex spatial patterns and appearance. While procedural materials can be modeled as directed (operation) graphs, they contain arbitrary numbers of heterogeneous nodes with unstructured, often long-range node connections, and functional constraints on node parameters and connections. MatFormer addresses these challenges with a multi-stage transformer-based model that sequentially generates nodes, node parameters, and edges, while ensuring the semantic validity of the graph. In addition to generation, MatFormer can be used for the auto-completion and exploration of partial material graphs.We qualitatively and quantitatively demonstrate that our method outperforms alternative approaches, in both generated graph and material quality.

Procedural material design (e.g., using a product such as Adobe Substance 3D Designer) typically involves creating a directed node graph, referred to as a material graph. Such graphs consist of a set of nodes — representing noise and pattern generators or operations on textures (e.g., filter kernels, transformations) — and edges — representing the flow of information from the output of the nodes to inputs of the subsequent nodes — finally producing image maps (e.g., roughness, normal, diffuse) for an analytic SVBRDF model. The output maps can be controlled by editing the parameters of the individual nodes. With complex material definitions regularly needing 50+ nodes, authoring such graphs is a form of black magic, limited to a select handful of practitioners. Not surprisingly, the largest publicly-available texture dataset has only a few thousand such definitions, and non-expert users mainly select from these limited options. Hence, there is a demand for automatically generating procedural materials, or assisting with their creation.

In this work, we introduce MatFormer, the first autoregressive generative model for material graphs. MatFormer leverages a transformer-based architecture to model a probability distribution over the space of procedural materials, and subsequently allows sample from this distribution. We found the choice of transformers, as opposed to LSTM, GRU, or graph networks, to be particularly suitable in this context, as transformers effectively handle sparse long-distance connections between graph nodes. However, in order to model the specific structure of material graphs, MatFormer does not generate them in a single pass. Instead, it runs in three sequential stages, each modeled with a dedicated transformer to capture dependencies: first, we generate a sequence of nodes; second, we generate parameters for each of the generated nodes; and finally, we generate directed edges connecting the input and output slots of the generated nodes.

@article{guerrero2022matformer,
    author = {Guerrero, Paul and Hasan, Milos and Sunkavalli, Kalyan and Mech, Radomir and Boubekeur, Tamy and Mitra, Niloy},
    title = {MatFormer: A Generative Model for Procedural Materials},
    year = {2022},
    volume = {41},
    number = {4},
    doi = {10.1145/3528223.3530173},
    journal = {ACM Trans. Graph.},
    articleno = {46}
}

We would like to thank Romain Rouffet, Luc Chamerlat, Geoffrey Rosin, and Gaetan Lassagne for their time, suggestions, and helpful feedback.

MatFormer:
A Generative Model for Procedural Materials

Abstract

Method

Paper

Acknowledgements