Fusing Interfaces

In my first period as an artist, my interest quickly developed from the 'static' art work, an object to be more or less passively contemplated and interpreted, to more dynamic forms whereby the audience, in a somewhat primitive way, could 'participate' in the art work, was stimulated to touch it, set something in motion, physically play with it; nowadays we would say 'interact' with it. The techniques used were ranging from inflatables and expanded cinema to street drawings and modified toy instruments.

Most of the projects involved collaboration with other artists and technicians from different disciplines. My favourite stage at that time was not the serious atmosphere of art gallery and museum but an attempt to go out into public space: to construct playful situations in the streets.

One of the main tasks in my second period, as a curator, has been the collection of all kinds of documents produced by the wide spectrum of modern alternative movements. It was a time that saw the spectacular advance of information production and reproduction tools, from the all available off-set presses and photo-copying machines to home computers and camcorders.

Tools, more easily available than ever before, that not only augmented the amount of information produced by these social movements, but also changed the quality of the information itself. The earlier essential difference between possibilities of professional and home equipment also started to vanish, at an ever quicker rate. There was a clear development from the production of predominantly text based to a prevalence of image based information. Over the last two decades I collected a lot of such pictorial and audio visual documents.

The limitations of the traditional way of making these materials accessible by textual description of images and sound, first card file and later a speeded up version of that in the form of computer databases, frustrated me more and more and I started to look for new methods of pictorial and auditory representation of information. In the mid eighties such ideas were not happily embraced by the academic 'main frame' computer world and I had to buy my 'personal computer' to be able to start research into what was just before coined as interactive visualisation.¹

It was at that time that I began to combine my earlier experiences with interactivity and visualisation in the realm of art with the new area of 'human computer interfaces' for information retrieval and dissemination. Several projects resulted. All of these involve group work of artists and technicians; all of these also try to go beyond the traditional cultural platforms combining artistic and educational elements and elements of play. Two examples with special interfaces are The Imaginary Museum of Revolution (1988 - 1989) and Orbis Pictus Revised (1991 - today).

A project in collaboration with Jeffrey Shaw, first conceived for the bicentennial commemoration of the French Revolution of 1789. The original design involved a rectangular space of fifty by fifty metres on which two hundred equally sized replicas of statues from throughout the world, representing two hundred revolutionary moments from the eighteenth to the twentieth century, were displayed.

These sculptures could be touched and would then 'play' short versions of related revolutionary songs. In the middle of this space, on a triangular platform, nine interactive videodisc installations were to be housed in a circular transparent construction with three entrances, flanked with columns with vending machines with miniature versions of the monuments, on sale for the public.

Videos run from a laserdisc player

These miniature monuments once placed on a sensing device attached to each of the video disc installations, gave the public the possibility to travel through time, space and ideology with the specific chosen revolution (represented by the small statue from the vending machine) as a starting point. In the text of the original proposal this was summarised as follows: The dramatised presentation of revolutionary moments from three centuries of world history with instant access by means of compact information carriers; a conjunction of library, museum and amusement arcade.²

This mega project has not yet been fully realised, but a simple first version has been functioning for two weeks in May 1989 in the Bruckner Haus in Linz, Austria. As the budget was limited, and, at that time, the technical possibilities for interactive handling of audio visual materials were still restricted, we decided to use a configuration consisting of a big hard disc for image and data material, and a glass video disc for the sound. The number of statues was limited to hundred and represented on four back projection screens as a continuous parade.

Vending Machines

The mini statues were sold from two automatic vending machines. These statues were no real sculptural objects, but postcards with an image of a statue and a bar code. The public had only one interactive installation at its disposal, made up of a pedestal with an built in bar code scanner and a touch screen. A large video projection system made it possible for a bigger group to see the individual interaction of a user of the system.

The vending machine

The whole set up was in a large darkened space in the Bruckner Haus. For a few shillings the public could buy a postcard with a statue representing a specific revolutionary moment. After inserting the card through the scanner-slit, a short audio visual sequence introducing that particular historical moment was shown. This was followed by a visual description of that revolution using artistic fusions of images from a wide variety of sources.

The sequence ended with a graphic summary of the event: a schematic map showing country and continent for space, years for time and a set of three specially designed pictograms (actors, actions and motifs) to summarise the ideology. On the basis of a large database with over two thousand revolutionary moments, linkages in time, space and ideology with the original chosen historical moment could be browsed. Additional audio visual information on personalities involved, iconic symbols and songs used in the chosen revolution were also available. The public consisted for a great part of school classes from local middle schools.

Orbis Pictus Revised is an interactive installation developed in co-operation with the Czech artist Milos Vojtechovsky. The first research for the project has been supported by the Amsterdam Historical Museum and the Dutch Ministry of Culture. The Media Museum of the Zentrum für Kunst und Medientechnologie in Karlsruhe has now given a commission for the realisation of a pilot version of the total project that will be ready in May 1995 (a full report is available from the author).³

Objects linked to a selection of the 150 tableaus of the Orbis pictus

The project is based on the book Orbis Sensualium Pictus by Comenius, first published in 1658. It was a playful and visual learning book for Latin and German. Later over two hundred different editions for many other languages were published. Comenius was one of the first who systematically developed a system for what we would now call 'sensorial learning': Everything should, as far as possible, be placed before the senses. Everything visible should be brought before the organ of sight, everything audible before that of hearing. Odours should be placed before the sense of smell, and things that are tastable and tangible before the sense of taste and of touch respectively.⁴

Tableaus

Innovative in the Orbis Pictus is the didactic use of the 'tableau method', whereby word and image are presented in a contextual way. Depicted words appear as parts of still lives and landscapes, tableaus. Each tableau has a title (nomenclature). The words on the accompanying page correspond with numbers in the tableau. Each word is set in a short phrase. It is a system whereby singular things are shown in their 'natural' context. This system gives the opportunity to order and unify in one place, according to classes and groups, that what in reality is separated in time and space.

It shows the singular in relation to the whole. An approach that is distinctive from the 'lexica system' whereby the - arbitrary - alphabetical order is the primary ordering principle. An incoherent mass of singular pieces makes it difficult to get a picture of the world as a whole. It is no coincidence that the Oxford-Duden pictorial English dictionary by Dieter Solf and John Pheby of 1979 is still constructed according to the same principles.

These 17th century tableaus offer an ideal 'spatial metaphor' for touch screen applications and it is from there that our ideas developed. We designed three ways of interaction: 'looking and pointing' (touch screen), 'speaking and listening' (a telephone horn linked to a voice recognition system) and 'touching and feeling' (sensing of manipulation of loose objects on a table).

Six Levels

For the touch screen interface the picture material is based on a study of the development of the process of making the world understood by means of duplicated imagery. Several similar books, published over the last four centuries, will be shown. Except for the numbers in the tableaus there is no written text. Words are spoken in three different languages (English, German, Neo-Latin) when 'hot spots' are touched. We are now developing a system to travel through time, and show both continuity and change in the representation of the world.

This is done by showing the available information on six levels: cosmograms representing the different centuries; a virtual bookshelf to get an idea of the limits of the amount of information available in the system; a division table with stacks of tiles representing the relative amount of information in the subsections (chapters) of each book, each stack symbolised by an abstract sign (characteristica universalis); a movable 'band' with modern pictograms (concrete signs) linked to a particular tableau as part of a chosen subsection; and finally, isolated elements from a tableau (singular word level).

The touch screen will be fitted horizontally in a table with an additional sensing device at the bottom of the screen. Touching inside the screen will activate the virtual objects at the six different levels, while touching the sensor outside of the screen will move the user back and forwards through time. The speech interface is still in an early stage of development and will be very much dependent on the availability of suitable speaker independent voice recognition systems, a market which is very dynamically developing at the moment.

Here the starting point is what Comenius called 'a playful alphabet': letters represented by pictures and associated with onomatopoeic sounds. Although such 'sound symbolism' can be called a misconception, seen from the perspective of nowadays linguistics (individual letter sounds do not have a meaning), it can not be denied that onomatopoeia have always played an essential role in the process by which children learn to speak. The different ways by which, in each language, 'real sounds' are imitated, will be another element used to interact with the visual materials displayed on a monitor.

Braille

The touching and feeling interface will have a large table (colour grey) with approximate hundred hand size objects (toys, playthings and other arty crafty objects made of quite different materials, painted grey also). These objects link to a selection of the 150 tableaus of the Orbis Pictus. The table will have three different sensing areas, one for each of the three languages. A large monitor is placed behind the table. When an object is picked up from the table, the screen will show the corresponding tableau and the position of the related object therein immediately.

There always is a second object from that same tableau on the table; this object will be highlighted on the monitor. To be able to proceed, the user has to find it in between the hundred other objects on the table. Each time an object is picked up, the system speaks its name in the active language of that moment. Languages can be switched by moving the object through one of the 'sensing areas'. These areas are indicated by light bundles and three dimensional icons (with captions in Braille) in the surface of the table. This part of the installation can also be used by blind people.

Tjebbe van Tijen, 1994

Notes

1. Henri Hudrisier, L'Iconotheque (Paris, La Documentation Francaise, 1982) p. 140-144.

2. 'The Imaginary Museum of Revolution', in: Mediamatic vol. 2 #4 (1988) p. 192-208.

3. Milos Vojtechovsky and Tjebbe van Tijen, Orbis Pictus Revised, an interactive installation (Amsterdam 1993).
Send a request to my email address for a copy.

4. Comenius, Didactica Magna (Amsterdam 1657), quoted in Jan Amos Comenius on Education (New York, Teachers College Press,1957) p. 95.

Go to: other texts by Tjebbe van Tijen | homepage IISH