by 
Lev Manovich, the author of the seminal The Language of New Media, MIT’s Fox Harrell, who recently published Phantasmal Media: An Approach to Imagination, Computation, and Expression, and MIT’s Nick Montfort examine the ways in which computational models can be used in cultural contexts for everything from analyzing media to imagining new ways to represent ourselves.
Moderator Nick Montfort began the evening’s discussion by asking the panelists to briefly present their current work.
Lev Manovich noted that his early training—he variously studied art, architecture, film, video, graphics, and programming—was particularly relevant to the “creative possibilities of computing.” With the rise of Big Data, Manovich said, “computation and culture have become almost synonymous” through social media; there are millions of filmmakers, photographers, and poets contributing through platforms like Youtube, Instagram, and Twitter. In 2005, as analyzing massive amounts of information became possible, Manovich began generating visual art projects with computers. He wondered if computers could be used to “create expressive representations out of data” and thereby examine past and present culture. The goal of his work, Manovich said, is to “represent our time through the contributions of many on social media.”
Manovich then described a few of his projects. He began with Phototrails, an enterprise that compares the style of Instagram photos within and between cities based upon color, brightness, subject, or filter. Another of his projects, SelfieCity, analyzes the style of self-portraits in five cities worldwide. This experience is interactive, Manovich said; users can filter the pictures by age, gender, or mood. His other ventures include an analysis of Instagram photos and tags that describe ordinary and extraordinary events in limited time and space in Kiev, and an analysis of the center of New York through its social media. This project provides highly reliable data, Manovich added, because companies produce it. Works like this, he said, can help provide social analysis and draw important cultural correlations. But the major goal is to allow an expressive experience and allow people to navigate, explore, and interact with information.
Fox Harrell opened by referencing the theme of the Forum. What is stranger, he asked, “than combining culture with computing?” Harrell said his research group at MIT focuses on creating computational systems for creative expression, cultural analysis and social empowerment. In an online world, he said, people play different roles and characters, often invoking gender and ethnic representations and stereotypes. Data structures and algorithms can implement old cultural phenomena; new phenomena, such as using Facebook statuses to sort people into various categories, can also be used. Harrell said his research focuses on understanding cultural phenomena “at the code level” and then building systems that can simulate and analyze existing cultural constructs.
Often, the latter involves the use of a “phantasm,” he said, which is some sort of sign or symbol that hides a worldview within it. One example Harrell used is the women’s restroom sign. He showed the audience that it is different around the world, as it’s based upon the cultural garb worn in each location. In each place, however, the sign is still understood as “woman.” Phantasms, Harrell explained, combine conceptual imagery with cultural values in a way that can be used to analyze computer systems. He used the example of social identity in computer games. In recent versions of the role-playing game Elder Scrolls, varying strength and intelligence is assigned to characters of different African or European descent; another game has three genders in addition to male and female—“both”, “other”, and “none”—but assigns male body types to all three by default. More than 90% of black female characters are props, bystanders, or victims in video games, Harrell said. These are important paradigms to consider, he added, because research has shown that people change their real world behavior based upon their virtual characters.
Harrell also described one of his projects, Chimeria, in which players must make interactive choices to navigate social groups. He described the work as an engine of modeling social identity that could be used in other systems as well. He noted that programs like Chimeria could provide critical social commentary.
Montfort then asked the panelists to speak to the process of developing software and culturally informed systems as part of their research.
Manovich recalled the winding road to his current position, noting that he had been hired to develop the visual art program at UC San Diego before becoming a professor of computer science at the Graduate Center, CUNY. His job had been to create expressive ways to represent or do something, he said; now, he examines software epistemology.
Harrell said that he sought a multidisciplinary grounding in art, logic, and computer science. Much of his job has been writing code and software, both alone and with collaborators. He added that it was important to ask how to make the work accessible; it’s also critical to get other people involved with the products. When an interface is built, there are many different entry points for people to come in and engage. Harrell said he sees several distinct levels of authorship in a program, beginning with the coding (which has some barriers to entry), and ending with the actual performance of a game or software. This provides many opportunities to become involved in the production process.
Montfort briefly described his collaboration with Harrell, a project called “Slant.” Slant is a story generation system that allows different units to contribute to a story and then blends the concepts. The goal, Montfort said, is to model small-scale genres of writing. He noted that developing multiple interfaces incorporates more users, but it can also slow down the development process. He asked the panelists to discuss what they felt the public needed to know about computation and media to understand the modern computer environment.
Harrell said that there are different types of literacy that people can have in this realm. Aside from operational and computational knowledge, it can be helpful to know how to think in a systematic way; individual talents can also be applied to building these new programs. It’s not essential to understand the technical details of these systems, he added, but it’s useful to understand how the systems function as networks as well as their connections to consumerism and culture.
Since the time that software became an engine of information in society, Manovich said, the world has run on systems that few people understand. It makes intelligent public conversations about these systems a challenge. Our society will soon face decisions, he continued, about what is most important: should we be as efficient as possible, building systems whose function we don’t fully comprehend? Or should we limit the complexity of software so everyone can understand these systems fully? Learning to code isn’t enough to grant full software literacy, he added; even those who code some systems don’t really understand them.
In keeping with the theme of the Forum, Montfort asked the panelists if there were particular domains or communities where “making strange” exists.
Manovich said that 21st century art relies upon making unrelated things seem related, and making familiar things seem unfamiliar. He noted that these are inherent processes in analyzing large amounts of social data. Social media data itself is an unfamiliar world, he said, and as such, it is challenging to defamiliarize. But often, analyzing apparently familiar cultural data, such as artwork, reveals that a conventional cultural understanding is flawed. He offered an example from his own analyses. He showed maps of Impressionist paintings, clustered by hue and brightness, and Japanese manga comics, sorted by texture and image complexity. The first map revealed that traditional ideas of what Impressionism looks like might represent only a small percentage of the period’s pieces. The second revealed that image type transcended individual works in manga. Ultimately, these represented a defamiliarization of cultural history. Style itself, said Manovich, could be an artifact of small data sets.
Returning to the question of where “making strange” exists, Harrell said that the worldbuilding in novels such as Invisible Man provided a formal structure to comment on society. Certain literary works and music such as jazz can balance improvisation and formal representation to convey commentary and meaning. Harrell said that he designed his work to look systematically at how values are structured and then make the structure accessible to people.
Manovich said that Big Data has, in a way, returned society to an age of Realism. Harrell’s artistic work, Manovich added, becomes very important in counteracting this. Harrell responded by saying that he isn’t interested in social realism, but rather in evoking cultural imagination.
Montfort then opened the discussion to questions and answers from the audience.
The first audience member recalled that Manovich had said that economic data collected via social media in New York was more accurate than the government’s data, and asked Manovich to clarify.
Manovich explained that with social media such as Instagram and Twitter, the information that comes in has some “stamp” from the machine that produced it. While it’s impossible to know if all the data has been received, the data that is produced through social media is “pretty good,” he said. He contrasted this with data that comes from the census, where geographic regions are broken into broad blocks, response rates are probabilistic and inconsistent, and error rates are high.
Another audience member said that there had been extensive discussion about the narrative potential of the tools Harrell and Manovich had built, but wondered if this could be harnessed in a different way as well. Big Data leads to increased categorization of our lives, she said, and it limits the ability of consumers to check the accuracy and nature of information collected about them. She wondered if Harrell and Manovich’s systems could be used to critique other data-driven narratives that arise from Big Data.
Harrell said that there is potential for abuse of many non-mainstream works. He felt that the lack of understanding and accessibility discussed earlier was the major problem; it helps, he said, to make processes familiar and accessible, and to make projects open source. It’s possible to restrict data collection to publicly available data, he added.
Manovich said that his goal is to build, not critique. Today, he said, we have the ability to create competing sources of information. Instead of engaging in academic critique, he asserted, he sees his duty as making issues visible; helping others understand and construct alternative sources of data is very important to him.
The audience member followed up by asking if Harrell and Manovich were speaking to social science, or promoting a different discipline that happened to be social in nature. Would their systems prompt a re-examination of the same data we’ve seen, or introduce new variables to measure and perceive social reality?
Manovich referred to the work of one of his personal heroes, French philosopher and anthropologist Bruno Latour. Latour believed that that social science should monitor reality, but thought the discipline lacked fine measurement tools, Manovich said. Latour hypothesized that with large amounts of data, a new type of social science could arise—one based on the individual, and without abstract terms like “class” or “status.” Manovich said that he sees the potential for a study of society that combines views from the humanities and the social sciences. We should be progressing, he said, to a place where computers can help humans better consider these concepts.
Harrell remarked that it’s helpful to identify important questions for research, and then determine which disciplines are most useful in answering them. Defining the discipline of a work often has to do with the disciplines of those involved in its discussion. Understanding the different value systems of distinct fields can improve communication, he maintained, but what matters most to him are the core questions that bring disciplines together. It is still important to consider social categories because people do respond to them. He added that the social understandings that come from other fields can help interpret the results of new emergent data. A data-driven approach doesn’t require the disposal of existing social analyses and structures, he said, but instead provides a new way to understand them.
