> open pdf
Project description
With my project, I wanted to address the abstraction of bodies, their presence and activity, within the media ecosystem, to get a critical understanding of this process of computational sensing. I started by learning how sensors work, specifically looking at distance sensors, and through their functioning, at how presence in space is sensed and codified. This allowed me to think about how the ambiguity of natural language and organic matter is abstracted and formalised by technical protocols, in order to render it available to computation. I wanted to address this process of formalisation and abstraction from an aesthetic perspective, to produce a figuring of technological sensing. As organic matter is picked up by the sensor and coded into discrete numerical data, it becomes information, and as such, it can be used to generate any type of automated response through a computer. My project was about making an automated system that transforms the data generated from the sensing of distance, into a sound response.
The Sensing Media Ensemble consists of an interactive system that perceives the presence of bodies through a set of distance sensors, and produces an automated sound response to it, by inputing the data gathered through the sensors into Pure Data, an open-source signal processing software, to electronically transform the information into sound. The project is experienced as an interactive system that produces a real-time materialised sonic response to any bodily presence or movement captured by the sensors. The sensors would ideally be distributed across an environment, opening up the ensemble to any number of human and non-human entities that populate the space. However, they can also be installed on a vest for a single performer to wear, thus centralising the sensing process.
The purpose of this practical experiment is threefold. Firstly, it is a way of building a technical system that re-embodies abstracted organicity, and through each step of the process, understand the transformation and translation from embodied to abstraction, and back again, as a logical-procedural activity. Then, it offers a way of figuring the invisible activity of computational sensing by making it aesthetically experienceable, and through this, thinking about questions of agency and subjectivity within a socio-technical ensemble of humans and machines. Lastly, it is a way of turning inaudible ultrasonic vibration, a fundamental element of contemporary technology - from distance sensors to telecommunication, from medicine to informational systems - into an audible vibration, that can be physically felt by a human body.
The media ecosystem
In general, computational sensing is a key process for the invisible mesh of human-machine interactions that constitutes our media ecosystem. In our everyday life, we stay interrelated with a whole ecology of machines that are constantly operating, sensing, and processing, all around and within us. However, despite the ubiquitousness and pervasiveness of this machine sensing process, we enter these relations and interactions almost always automatically and unconsciously. (Schmidt, 2012) In fact, the pragmatic purpose of much of the design of technological systems, devices, interfaces, and processes, is to produce seamless interaction, to make technology disappear, normalised in the concrete environment and its social practices. As a result, we are used to taking for granted the diffuse ensemble of technological objects, protocols and processes that silently influences our social relations and alters our conduct, as our thinking, behaviour, and bodies adapt, in a constant process of information exchange, to be more in tune with these computational structures. (Harwood, 2019) These invisible, networked, synchronised technical processes create a vast, obfuscated, area of “grey” media, that enables the stabilisation and naturalisation of a socio-technical milieu, made of habits and processes that go largely unnoticed, under a threshold of critical perception (Fuller and Goffey, 2012).
Ultrasonic sound, and vibration more broadly, is a core element of many of these processes of computational sensing, from surveillance infrastructures to environmental sensing devices, from invisible interfaces to the push notifications that articulate the relentless stream of information that demands our attention, and constantly stimulates our nervous system. Wether we hear it or not, we are constantly interrelated with a whole sonic ecosystem of machines. Therefore, we can think of electronic vibration as the material source of both the audible and inaudible sounds of computation. Similarly, if we take into account the pervasive presence of ultrasonic technology in our technological infrastructures, we can understand high frequency waves as a permeating presence, which reconfigures our environment as a high-frequency electromagnetic field.
Working and thinking with the project
While developing the project and learning how sensors function, I carried out a genealogy of ultrasounds, which allowed me to frame ultrasonics as a technology that men, despite being neither equipped with, nor able to directly perceive, observed as already present in nature, and tried to artificially replicate and apply to a wide range of fields. On one hand, the necessity to detect and avoid obstacles and threats beyond the capacities of our senses, pushed the development of prosthetic technologies aimed at augmenting our senses with technologies we observe in other species, to reassert our dominant position in the natural world. At the same time, the development of these technologies also emerges as a practice of understanding nature by mimicking its systems. In fact, the story of ultrasonics starts with observations of animals and ends with discoveries of echo-ranging abilities in undersea mammals that make our scientific advancements look rather humble in perspective. (Graff, 1981)
It might be an interesting coincidence, then, that I ended up testing my project with animals, as I had the sensors pick up the presence and movement of the pigeons and ducks that inhabit Peckham Rye Park. Reflecting back on the experience of opening the Sensing Media Ensemble to animals helps me think about the implications of taking part in a collective performance with an assemblage of sensing agents. In fact, engaging with animal cultures was a way for the project to make a step out of itself to enter a collective sensing and performative process with nature. Fuller’s idea of “art for animals” addresses the perceptual reflexivity and the different capacities for perception of a heterogeneous ecology “at multiple scales, from the social, to the medial, technical and aesthetic, to that of subjectification.” (2010) This interrelation of capacities and materials allows the project to address the question of subjectivation across a human-animal-technical assemblage, which seems to work towards a nomadic notion of the subject, exemplified by Braidotti’s conception of “affirmative interrelations” across the process of becoming of different life forms. (2006) Working with and through different material and sensing capacities, different aesthetic dynamics come into composition across the interrelations between machine logic and different biological forms of cognition and signification. By de-centring the meaning-making activities that shape the aesthetic form of the performance, this engagement of the technical object with the biological world has the effect of curtailing the human maker’s agency, in favour of the unintended. Although the decision to have animals as the main participants and audience for my project came in response to the limitations I was facing during the dramatic situation of the global pandemic, in some way I saw this practice of making art for/with animals as a strategy for sabotaging the imperative of human command, control, and domestication, typical of the traditional anthropocentric character of western thought. After trying to retain some degree of control over a flight of pigeons or a flock of ducks - the latter group showed particular reticency to getting close to the sensors, which suggests they might be more sensitive to ultrasonics - one is inevitably led to accept that the control of the experiment is irreversibly out their hands. In light of this, we can see this experience as an interesting and perverse exercise of "abandoning the human as the sole user of producer of art”. (Fuller, 2010)
Making the project as critical practice
It’s worth reviewing the process of making the project as a way of critically understanding our interaction with media, as well as our interrelation with flows of agency, signification, and logic, across an heterogeneous assemblage of entities.
We can think of critical practice as the process of “attempting to explain the phenomena caught in the lens of a project or a proposition, ultimately reflecting on the process”, which means making things as a way of exploring them, incorporating artistic practice, not necessarily to make art, but to use its ambiguity as “a method of enquiry into a particular sociotechnical milieu”. (Harwood, 2019, p.32) Specifically, a critical computing practice allows us to figure computation as a technical, social, and cultural object, in an attempt to produce a critical understanding of media interaction and its multiple meanings and implications. Although our relationship with technical objects is increasingly more intimate, the way we talk, think about, and interact with them is often restricted and obscured by various cultural and social constructs, which are generally set in place to ensure that technical systems run smoothly, and that their infrastructures are not too visible and overwhelming to the user. (Penn, 2019) Critical making can offer a form of “counter-testimony”, and the objects made can represent a “counter-narrative”, that foregrounds the obscured, the inaudible, the unnoticed, as a matter of concern. (Dada, 2019) By spending time making and engaging with technical objects, this immanent process of material critique allows us to question and understand their physics, histories, relations, and the flows of agency connected to them.
If we consider that “knowledge can be contained in an object and its functioning is its explanation” and that “creating and analysing are not necessarily different processes that need to be separated,” we can think of knowledge as flowing between thinking as abstract, immaterial, making, and making as tangible, material, thinking (Harwood, 2019, p.30; Cole and Perner-Wilson, 2019, p.110). This allows us to combine a culture of the technical with one of the critical, using practice to think about theory and vice versa. In fact, if theoretical abstraction is a strategy for distancing from concrete experience to enable manipulation, referencing, and conceptual synthesis, making is a material and affective activity. (Critical Media Lab, 2019) By making the project, I had to learn how to move back and forth between thinking and making, in an iterative loop of perception and reflection. Each step of the project involved different degrees of abstract thinking and material engagement, functional action and free-form curiosity, not necessarily as discrete or modular processes, but more as a wholistic and reflexive practice shifting on a continuum, and ultimately instantiated in the project.
Lastly, by reflecting on my engagement with technical learning, materials, histories, and other collaborators - from strangers willing to share their knowledge and experience online, to my tutors and colleagues willing to help me, to the pigeons willing to engage with my sensors - I see posthuman theories as a useful tool to think about the whole of these relations as an assemblage of material and discursive actors that configures a form of more-than-human making. By questioning the assumption that the ability to affect or be affected is the exclusive domain of the human, critical posthumanism facilitates instead “a framework for maker culture that explicitly recognises the non-human vitality within materiality” and projects technology outside of the traditional master/slave paradigm. (Foote & Verhoeven, 2019, p.76) Therefore, we can adopt this posthuman perspective to reposition the maker as an equal creative partner in a vital assemblage of organic and inorganic, human and nonhuman, agents, materials, and objects “which exhibit a latent capacity for affecting his own agency and the outcomes of the work.” (Foote and Verhoeven, 2019, p.78) Moving beyond binary dialectics of human and non-human agency, to accept “a position of productive uncertainty, situatedness, and relationality to surroundings and materials” enables us to “greatly expand our understandings of the multiple agencies, dependencies, entanglements, and relations that make up the world.” (Foote & Verhoeven, 2019, p.83; Forlano, 2017) This perspective constructs an idea of subjectivity not bound to the concept of the individual, but continuously forming, as in Simondon’s idea of a technical ensemble, where different agencies interrelate in an emergent collaboration. (2017)
Information and materiality
From a technical standpoint, a lot of the operations involved in making the project, were about putting the sensor, that had to send out and receive back the ultrasonic waves, in communication with the computer, that had to perform the calculations to measure the distance, print the data to a file, send it to another computer, which would then input the data into Pure Data, to turn in into audible electronically-generated vibration. This whole process was an exercise of translation between computational language and the materiality of the electronic circuits that controls the transmitter and the receiver in the sensor. This process forced me to think of what it means to put in relation the abstract logic of computing with its actual materiality. The logical problems of computing are abstract versions of real engineering conditions. In 1937, Claude Shannon translated 19th-century Boolean logic into the electrical circuits that are in every contemporary digital computer, establishing a rigorous equivalence between arithmetic logic and electrical engineering. When logic is instantiated in physical computing operations, a number of material differences come into play, like problems of timing, heat generated, or the need for power supply. We can understand the formal languages of computing as attempts to fill in the gap between logic and circuits. A programmer is then a designer of artificial languages, and using these languages is a way of “doing logic” as a form of translation between abstract logic and electrical circuits. (Sack, 2019)
However, in much of our experience and relationship with computers, there is a tendency to obscure the vibrant materiality of machines, and think of them as primarily informational, without considering the processes and agents that act, communicate, and resist, through material difference. This perceived primacy of information over materiality blinds the user to the actual value, functioning, and nature of information communication. (Hayles, 2000) This illusion of the complete disembodiment of information requires the systematic cancellation of many significant aspects of our interaction with computing and communication in general. In fact, the efficacy and the very existence of information depends on a highly articulated material base. In Shannon’s 1948 mathematical theory of information, no message is ever sent, what is sent is a signal, and through its encoding in this signal, for transmission through a medium, the message assumes material form (1948). Therefore, for information to exist, it needs to be instantiated in a medium, through signal. Abstracting information from this material base is “an imaginary act” that constructs the matter/information dichotomy. Although almost never used by those actually developing the technologies, as they can’t afford to ignore the materiality of the interfaces they make, this dichotomy is imposed it on our general culture and language of the technical. (Hayles, 2000) By enabling us to attend to the materialities of media and technical objects and their implications, a critical technical practice can be a way of producing new cultural understandings of technology and our relation with it.
Time
Another fundamental question to consider, in our engagement with information and materiality, is that of time. First, it’s worth mentioning a few moments, in the historical relation between temporalities of information and matter, when the idea that the usefulness and efficacy of information are time-dependent became evident. In the summer of 1790, with France on the verge of a revolution, Claude Chappe and his brothers developed the semaphore telegraph. The system consisted of a series of towers no more than 20 miles apart, topped by a machinery featuring two movable arms linked by a crossbar that could be arranged in different positions for a total of 196 different symbols, that carried encoded meaning. Messages were transmitted from tower to tower by human operators, who would observe adjacent towers with a telescope and replicate the signals for the next tower to pick up. This technology allowed the establishment of the first large-scale communication network that could carry a message faster than a horse-riding messenger could over a good road system. That speed of transmission had remained essentially the same since the Roman Empire (Norman, 2005). This was the first time information transmission surpassed transportation in speed on a large scale, and in some way, the moment of separation between a material temporality and an informational one. However, the fact that the value and usefulness of information is time-dependent became clear more than ever during World War 2, when rapid technological infrastructures had developed sufficiently for information to come into its own as a commodity that could be as important to military success as weapons and infantry. (Hayles, 2000)
In my project, I found myself dealing with issues related to the synchronicity between Python scripts, information transmission, and the execution of Pure Data patches, which determined the real time character of the interaction I was trying to realise. This made me think about the concepts or time, synchronicity, and real time, in human-machine interaction and responsive computational systems. In the realm of assembly code and machine language, computers “operate within a profoundly non-Cartesian space” where distance is measured by clock cycles, the clock rate of CPU, which essentially determines the rate at which computations occur. (Hayles, 2000, p.90) According to this rate then, the computer performs its functions by fetching, from the memory, the next operation in the process, decoding the operation, executing it, and then moving on to the subsequent operation, in what is known as the fetch-decode-execute - or FDX - cycle. (Sack, 2019) According to Chun, “code is logos: code as source, is code conflated with, and substituting for, action” (2016, p.70). Code, literally, does what it says. If the source code is correct, it will produce the expected execution, by which word becomes action, closing the gap between logic and praxis. Instruction and result coincide, as the text embodies executive power. Rule-making and execution happen simultaneously, and indistinctly.
However, when interacting with the non-digital environment, this timeless temporality and abstract logic of code unfold in relation to the material and spacial dimensions of the actual interaction. For example, in a system such as the one established in my project, we can think of “real time” interactive response as a form of what Chun calls “volitional mobility”; a mechanism of dynamic changes in digital media that seem to respond to the user’s inputs, creating a sense of “liveness on demand”, and transforming the computer into an “alive” machine that reacts to contextual inputs. (Chun, 2016) However, this sense of a-liveness is a computational illusion, as mechanic real time is not alive, but simply reacts to the liveness of human temporality with instantaneous responses. The sense of “liveness” it produces is just the quick response to user activity. Computers are feedback machines based on automated response mechanisms that set in motion the operations of programmed labour. What we experience in such an interactive system is not a real, live decision, but the already planned, a reaction that is “immediately felt, affective, and based on user actions, and yet at the same time programmed.” (Chun, 2016, p.79) Computational “real time” refers to the speed, the timeless time, of computer processing, rather than to the materiality of the user’s time. The fluid interaction we experience is not really in real time, but always deferred and mediated, even if only to an infinitesimal degree of synchronicity. When designing an interactive system by making a contraption, a DIY, non-normalised technical object, it is possible to observe how all the procedural operations that actually produce the interactive response, ultimately add up to a small lag in the flow of the computer’s response. At the level of the user experience of this interaction, even before understanding clock rates, code, or abstract and material temporality, one can feel, and intuitively grasp an immanent understanding of, the “artifactual physics” of computing. (Hayles, 2000)
The experience of the project
The interaction of machine logic with the user experience of computational processes produces a physics of the virtual, where the material instantiations of codes and programming, normally invisible to the non-specialist, are felt and intuitively grasped by users in their interaction with computing. This process of feeling the integration of bodily perceptions and computer architectures causes a sense of change in subjectivity, and an instinctive, bodily understanding of the interrelation of material objects and informational patterns that constitutes the reality of cyberspace, our “condition of virtuality” (Hayles, 2000). It’s worth reflecting on the project as a way of producing this bodily understanding of the entanglement of virtual and physical ways of being, and thorough the aesthetic experience of it, getting a sense of change in subjectivity, beyond the human/nature binary and the illusion of disembodied information.
The dynamic interaction produced by the project relies on the formalisation of organic activity into quantified data. The idea of information as a coded message that can be abstracted and re-encoded into any other form is something specific to electronic and digital media. As Manovich puts it, “previously, physical and mechanical media tools were used to create content which was directly accessible to human senses. The ‘message’ was not encoded in any way; it was created, stored, and accessed in its native form,” while electronic media technologies like the telegraph, the TV, and then digital computers, started functioning by coding the message, which “makes possible the idea of information – a disembodied, abstract and universal dimension of any message separate from its content.” (Manovich, 2011) The translation of organic activity into formalised, discrete numerical units requires a form of computational abstraction, that allows the replica of a set of objects or functions in a different language. As the bodies present in the environment are captured by sensors, they become available, usable, in a computable form, through a process of formalisation and quantification. "Human activity is thus effectively treated as a kind of language itself.” (Agre, 2003, p.743) By moving in front of the sensor, and feeling an aesthetic response to the abstraction of our material presence into numbers, we can think of how our bodies are inscribed into computational systems. This inscription coincides with a process of individuation of the subject, what Stiegler would define as an “exteriorisation” of the organic into the technical through which user subjects come alive, and their consciousness manifest, in the technical system (1998). This process of individuation-inscription means that, to technical systems, the subject exists only in the way it manifests itself through computational sensing; as a flow - or an intersection of multiple flows - of data. In computation, to exist is to be sensed, to emit signals, so that the production of data becomes the evidence of one’s sustained existence (Chun, 2016). The subject is replaced by an agent, or an actor, as “the subject's narrative is replaced by the capturing of actions.” (Andersen and Pold, 2018, p.33)
Making the project was a way for me to engage with a process of material critique of media and information, following a non-anthropocentric drive that answers Hayles’ call to “put back into the picture the flesh that continues to be erased in contemporary discussion about cybernetic subjects.” (Hayles, 1999, p.5) The experience of the project, the feeling of a computational re-embodiment of abstracted biology, aims to function as a way of “pushing past interactivity and towards a more existential pondering of connection and forms of communication.” (Burrough and Solomon, 2019, p.68)
References
Agre, P. E. (2003). Surveillance and capture: Two models of privacy. In N. Wardrip–fruin, The New Media Reader (Har/Cdr edition). MIT Press.
Andersen, C. U., & Pold, S. B. (2018). Interface Criticism: Why a Theory of the Interface? In The Metainterface: The Art of Platforms, Cities, and Clouds (pp. 15–38). MITP.
Braidotti, D. R. (2006). Transpositions: On Nomadic Ethics. Polity.
Burrough, X., & Solomon, L. H. (2019). Drawing on Theory to Build/Unbuild Media, or Lalalalalalalalala. In L. Bogers & L. Chiappini (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Chun, W. H. K. (2016). Crisis, Crisis, Crisis, or The Temporality of Networks. In Updating to Remain the Same: Habitual New Media (pp. 69–91). MITP.
Cole, D., & Perner-Wilson, H. (2019). Getting Lost and Unlearning Cerainty: Material Encounters in an Electronic Craft Practice. In L. Bogers & L. Chiappini (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Critical Media Lab. (2019). Unmaking: Against General Applicability. In L. Chiappini & L. Bogers (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Dada, M. (2019). The Counter-Testimony of the Maker. In L. Chiappini & L. Bogers (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Foote, G., & Verhoeven, E. (2019). Tactics for a More-Than-Human Maker Culture. In L. Bogers & L. Chiappini (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Forlano, L. (2017). Posthumanism and Design. She Ji: The Journal of Design, Economics, and Innovation, 3(1), 16–29.
Fuller, M. (2010). Art for animals. Journal of Visual Art Practice, 9(1), 17–33.
Fuller, M., & Goffey, A. (2012). Introduction. In Evil Media. MIT Press.
Graff, K. F. (1981). A History of Ultrasonics. In W. P. Mason & R. N. Thurston (Eds.), Physical Acoustics (Vol. 15, pp. 1–97). Academic Press.
Harwood, G. (2019). Teaching Critical Technical Practice. In L. Bogers & L. Chiappini (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Hayles, K. (2000). The Condition of Virtuality. In P. Lunenfeld (Ed.), The Digital Dialectic: New Essays on New Media (New Ed Edition). MIT Press.
Hayles, N. K. (1999). How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics (74th ed. edition). University of Chicago Press.
Manovich, L. (2011). Inside Photoshop. Computational Culture, 1.
Norman, J. M. (2005). From Gutenberg to the Internet: A Sourcebook on the History of Information Technology. Norman Publishing.
Penn, S. (2019). Learning the Washing Machine. In L. Bogers & L. Chiappini (Eds.), The Critical Makers Reader: (Un)learning Technology. Institute of Network Cultures.
Sack, W. (2019). Logic. In The Software Arts (pp. 107–145). The MIT Press.
Schmidt, U. (2012). Ambience and Ubiquity. In Throughout: Art and Culture Emerging with Ubiquitous Computing.
Shannon, C. (1948). A Mathematical Theory of Communication. The Bell System Technical Journal, 27.
Simondon, G. (2017, May). The Genesis of Technicity. E-Flux, 82.
Stiegler, B. (1998). Technics and Time: The Fault of Epimetheus No. 1 (R. Beardsworth & G. Collins, Trans.; First English Language Edition edition). Stanford University Press.