Engineering:Synthography

Short description: Method of generating media using machine learning

A synthographic image created with a fine-tuned Stable Diffusion model and outpainting

Synthography is a proposed term for the generation of digital photos using a generative adversarial network. City University of Hong Kong photography professor Elke Reinhuber coined the term in her 2021 conference paper "Synthography – An Invitation to Reconsider the Rapidly Changing Toolkit of Digital Image Creation as a New Genre Beyond Photography," writing "As soon as a generative adversarial network (GAN) is trained to create an image which resembles a photograph, I propose to better describe it with the term synthograph." This is distinct from other graphic creation and editing methods in that synthography uses artificial intelligence art and text-to-image models to generate synthetic media. It is commonly achieved by prompt engineering text descriptions as input to create or edit a desired image.

Text-to-image models, algorithms, and software are tools used in synthography that are designed to have technical proficiency in creating the resulting artificial intelligence art output based on human input. Synthography typically uses text-to-image models to synthesize new images as a derivative of the training, validation, and test data sets on which the text-to-image models were trained.

Synthography is the method used, not the output itself. The output created specifically by generative machine learning models (as opposed to the broader category of artificial intelligence art) are referred to as synthographs. Those who practice synthography are referred to as synthographers. A synthographer can harness the ability of linguistic composition to tame a generative model. Other cases also include fine-tuning a model on a dataset to expand its creation possibilities.

Practical uses of synthography include AI-driven product shots, stock photography, and even magazine covers with some making predictions that synthography may be the future of photography.

Etymology

Reinhuber chose the term "synthography" by combining the terms "synthetic" and "photography," proposing its use to describe images that use "the methodology of synthetic production" and that move "beyond the classic understanding of photography."

History

The event that started the broad usage of text-to-image models is the publication of DALL-E by OpenAI in January 2021. While it was not released to the public, CLIP (Contrastive Language-Image Pre-training) was open-sourced, which led to a succession of implementations with other generators such as Generative adversarial networks and Diffusion models. The next big event, which led to a rise in popularity of such technique, was the release of DALL-E 2 in April 2022. After slowly releasing it as a private beta, it became public in July 2022. In August 2022, Stable Diffusion was open-sourced by Stability AI, which fostered a community-led movement.

\

Difference between Synthography and Artificial Intelligence Art

Artificial Intelligence (Artificial Intelligence Art) relation to Generative Models (Synthography), Venn diagram

Synthography is the method used to create synthetic media using generative models. Artificial intelligence art (including music, cooking, and video game level design) is the output created using artificial intelligence which is an overly and increasingly broad category.

When Elke Reinhuber coined the term synthography in her paper, "Synthography – An Invitation to Reconsider the Rapidly Changing Toolkit of Digital Image Creation", she spoke of a "legitimation crisis" as a need for the term. Before generative models were used, artificial intelligence art algorithms already existed in mediums such as graphics editing software (ie: content-aware fill, application of artistic styles, resolution enhancement) which employs a wide range of artificial intelligence tools, and DSLR and smartphone cameras (ie: object recognition, in-camera focus stacking, low-light machine learning algorithms) all of which continue to undergo rapid development.

Artificial intelligence is a superset of Machine learning. Machine learning is a superset of neural networks. Neural networks are a superset of generative models such as GAN's (generative adversarial networks) and diffusion models. The relation between all of these is depicted in the Venn diagram shown here. Synthography specifically uses generative models, as popularized by software such as DALL-E, Midjourney, and Stable Diffusion.