Those Who, Matthew Lutz and Alessia Nigretti

Just before the opening of The Coming World: Ecology as the New Politics 2030–2100, artist Sascha Pohflepp passed away. Matthew Lutz and Alessia Nigretti, who collaborated with him on the installation featured in the exhibition, Those Who, have produced a detailed description of the work.

Sascha Pohflepp in collaboration with Matthew Lutz and Alessia Nigretti
Those Who, 2019
Multimedia installation with dynamic content, created for the exhibition The Coming World: Ecology as the New Politics 2030–2100 at Garage Museum of Contemporary Art

Production team
Sascha Pohflepp: research, concept, and direction
Matthew Lutz: computational biology and ecology advice, research, audio composition
Alessia Nigretti: Unity programming

Alessia Nigretti (b. 1996, Catania, Italy; lives and works in New York) is a game developer with a special interest in artificial intelligence. She currently works remotely as a Game Engineer for Klang Games in Berlin. Her recent work ranges from developing biofeedback-enhanced audio-visual tools for virtual reality mindfulness to experimenting with artificial intelligence applications for video games and virtual environments.

Matthew Lutz (b. 1980, Philadelphia, USA; lives and works in Berlin) is a biologist, designer, and musician whose research focuses on understanding animal and machine behavior in collective and distributed systems. He has a PhD in Ecology and Evolutionary Biology from Princeton University, where he studied self-assembled structures and traffic dynamics in army ants, and a Masters in Advanced Architectural Design from Columbia University. He is affiliated with the Max Planck Institute for Animal Behavior in Konstanz as a Postdoctoral Researcher.

Project description

Natural history museums show the history of the living world: the forms that exist in nature and what led to their emergence. They often exhibited preserved animals in combination with other media to bring the exhibits to life. Moscow’s State Darwin Museum has historically combined taxidermy with paintings that were meant to illustrate how a given life form fitted in to its environment.

Yet it feels as if the museum may have been after something else that is unique to living systems, a manifestation of parameters whose true complexity we cannot entirely capture. The current exhibit ends with our own species, Homo sapiens, in its “natural” habitat, largely comprised of synthetic materials and information technology.

In response to this, Those Who offers a speculative extension of the museum and its dual paradigm of form and behavior into the realm of simulated life. The world consists of artificial life forms that evolve over time, a set of self-propelled resource entities that they consume, and an environment within which both are embedded.

At the beginning of each day, a new world starts.

Agents are born and multiply, sometimes detach and move in search of resources, sometimes cooperate, and eventually die. New agents are constantly being born into the world, at a rate balanced by the decay and death of the transparent shells left behind, to maintain a computationally efficient equilibrium population size in the world.

Using reinforcement learning to train the agents initially, and a genetic algorithm to evolve new forms and behaviors, the underlying framework of the world is intended as a kind of playground for the free exploration of behavior and the emergence of intelligence, of whatever kind. The levels and values of resources could also be linked to real-world inputs, such as the price of rare-earth minerals, to visually explore market dynamics.

Focusing on the intriguing relationship between reinforcement learning and natural selection, the world provides a framework for asking fundamental questions about learning, evolution, and the similarities between the two, as highlighted by computational theorist Leslie Valiant. While it may not answer these questions, it could hopefully provoke others.

What does the world want?

Algorithm description

Every generation, agents (each with their own genotype) are mutated and selected for according to a fitness function. The fitness of the agents is determined by their success in collecting resources, which provide them with the energy necessary to explore the environment and generate new agents.

Agents are trained to collect resources by a neural network using reinforcement learning, and their performance in doing so varies depending on their genotype.

Reinforcement Learning (ML-Agents)

At every time step (here each physics frame), the agent being trained observes the environment and records if there is any desired resource at the 5 angles (-45, -30, 0, 30, 45) in front of it at X axis and Z axis (width and depth) (total of 9 angles, as 0 is shared by X and Z).

This is used to determine what action to take (in which of these 9 directions a new agent is to be spawned). The neural network outputs 2 actions each frame: one is the direction of the spawn (either -45, -30, 0, 30, 45) and one is the side it is going to branch from (top, left, right, back, front).

A reward is provided every time the agent gets closer or collects a resource, which allows the neural network to find appropriate associations between the observations and the actions made. After a certain number of iterations, the reward is high enough to show intelligent behavior in the environment.

Genetic Algorithm

Every generation (10 seconds), the fitness of the current population of agents is measured (by the value of resources collected). Then, each gene of each agent (“root agent” or “brain”) in the population is mutated with a 50% probability.

Parameters, or “genes” that can mutate in this way, include sensing range, ability to detach, collaboration with other agent types, resources needed to survive, size, and appearance. At this point, the fitness of this mutated population is measured and compared against the previous, unmutated population. The population with the highest fitness wins and replaces the current one.

Epilogue

The sudden shock of Sascha’s passing hit us during the final week of project development. After months of intensely collaborative research, brainstorming, and prototyping, and many hours of conversation probing some of the deepest questions in philosophy, computation, evolution, and ecology (for which we are forever grateful), the inchoate form of the project had to crystallize, and we had no doubt that Sascha would want us to see the work through to completion. While it’s impossible to predict exactly the outcome had he still been with us these last days, we have done our best to bring into the world a piece that we think fulfills Sascha’s (and all of our) vision. This has been a truly collaborative effort from the early conceptual stages, as Sascha was a project leader who deeply valued the expertise and input of everyone on the team, and so we feel confident in finishing the piece as a team in his name.

Of course, the events of the past week have been traumatic for us, and inevitably those emotions have seeped into the final work in some ways. We chose to keep the agents monochromatic, hinting at the visual language of games like chess and go, since games were a love of Sascha’s and a constant topic underlying all of our discussions. Earlier prototypes were rendered in more playful colors, but these were always placeholders, and while we had discussed various strategies, we had not yet decided whether or how we would implement color in the world. In terms of sound, we had only just started discussing the audio elements together, and all of the sound design for this iteration of the piece was done after Sascha’s passing. Matt composed the accompanying audio piece as a kind of tribute to Sascha, so an alternate version of the project might have had a much different overall sound. In the end, the aesthetic reflects something of our emotional response to what occurred, and we think Sascha would find this appropriate.

Matthew Lutz and Alessia Nigretti