Computer Games for Partially Sighted and Blind Children
Contents
Today, computer games are an integrated part of children's play activities as well as school education. Most contemporary computer games feature sophisticated 3D animated graphics, accompanied by sound effects that provide extra feedback to the player. For a partially sighted or blind child those two components make it extremely hard to play computer games on the same conditions as their sighted friends. It can be very frustrating for them to have the constant noise of a soundtrack playing while trying to figure out the interface and several details on the screen. Sometimes the sound carpets are so disturbing that they prefer to play the games with the sound turned off. The visual content can also often be problematic as different characters move around constantly, which makes them hard to "grasp" visually.[1]
The background to the computer game project, which will be further discussed below, is that the Swedish Library of Talking Books and Braille (TPB) were to give partially sighted children and blind an introduction to computer games. The intention was/is to offer joy and pleasure to the same extent as for full sighted children.
It is easy to understand that it could be very complicated for a blind person to play an ordinary computer game, since most of them are built on visual output. In such games, the objective for the player is to solve different tasks via visual information, and to find strategies that make it possible to win the game. These strategies are manly built on visual information, although they can also be communicated with text.
To solve a problem or to find a certain strategy it is often very important that one has an overview of the situation. The player of a computer game obtains this overview by looking at the screen, where the interface layout often helps the player to perceive the overall picture. For a person with limited sight, it is often a big problem to get this overview due to the impairment. One of the most common pieces of equipment for partially sighted computer users is software that enlarges what is displayed on the screen.[2] With this software, the users can chose what size they want the text depending on what they find the most comfortable to read. A consequence of this is that a much smaller part of the text is shown on the screen, compared to the original text layout. This is convenient for text reading, but it is not suitable for pictures and especially not for animated graphics.
Access to illustrations
Our knowledge about the perception of children that are partially sighted is based on experiences from the production of tactile picture books.[3] Even though the tactile picture books are primarily made for touch, they are created to make it possible to see different elements and details even if one has a very limited sight. The designs of the pictures in the tactile picture books are based on theories that are influenced by the Gestalt psychology with regard to the ideas about the importance of shape.[4] We recognize an object from its outline shape. To recognize an object by its shape, it has to be depicted from an angel that makes it recognizable, that is as close as possible to a 90-degree angel. Within psychology one makes a difference between perceptual forms and physical forms. The perceptual form is dependent on the angle of view, while the physical form refers to the actual shape of the object. The further from the physical form a representation of an object is, e.g. a very unusually angle for the depiction, the harder it is to interpret the actual object. For tactile interpretation, and for partially sighted to perceive a representation of an object, it is very important that the perceptual and physical shapes are very close.[5] That does not mean that a congenitally blind person could not interpret picture elements depicted in perspective, which has been proved by John M. Kennedy.[6] An interpretation via touch will definitely be more complicated if complex image with many details a created in perspective.
The use of colours
When we talk about large contrast between colours, we usually mean contrast colours, referring to the colour you find in the opposite place in the colour circle (complementary colours). An example of that is orange and what is called Klein blue. The complementary colour always has the same degree of saturation, and that is a problem for a person with colour blindness. For a person who perceives the environment in a grey scale, there is no contrast between colours with the same degree of saturation, due to the impairment. It is not uncommon that a person is colour blind as an effect of a visual impairment. Since the contrast in complementary colours not work for persons with partial sight, the computer games are made in colour contrast that is based on saturation.
To avoid that details in the computer games to appear as silhouettes, the background is made in a dark colour and the different picture elements, such as objects and characters, are in brighter and lighter colours. As a consequence of this, the player will perceive that the background settles while the objects will move forward. This is ideal for a person with visual impairment, and especially for those who lack colour vision.
To illustrate depth and volume
Since the Italian Renaissance we are used to create and interpret images with perspectives. A common way of illustrating depth on a surface is to make objects that appear further away smaller than the closer objects in the foreground. In addition to the differences in size of the objects, colours of different nuances, light and shadow are often used. For a person with visual impairment the colour nuances and shadows can cause problems, since the parts with shadows or another nuance easily can be interpreted as separate objects.
In the computer games created for TPB's website, the designers have used neither colour nuances nor shadows. The pictures in the computer games are designed to look flat. Maria Beskow and Annica Norberg have used the same style for the composition of the site as a whole and in the single details, as was used in the tactile picture books: clear, simplified objects and characters in light colours on a darker background.
Today, it is popular with fluidised colours, and a person with no sight problems usually finds them very colourful. The problem with the fluidised colours is their luminary quality, since the contours of the object will appear less distinct. The quality of the contour is crucial for the outline of a shape.
The TPB computer games
The computer games produced by TBP are divided into two parts, called Sound-Based Games (Spel med bilder) and Image-Based Games (Spel med ljud).[7] The first category demands that the player has some sight, since the interface is based on pictures. The games of the second category all have sound interfaces and are designed primarily for children that have no sight. The games with pictures are all played by using the mouse. However, the sound-based games are played with the computer keyboard, since the mouse is avoided by many visually impaired computer users. The games have intentionally been designed for standard home computer equipment only, so one needs computer with a standard soundcard, a pair of headphones or loudspeakers to play the games. Since the games are web-based, the computer can be either a Macintosh or PC, as long as it is equipped with a web browser with a Flash plug-in. The games could also be distributed on CD-ROM to be played in a web browser without an Internet connection.
Computer games for partially sighted
A computer game suited for partially sighted has to be very well designed. By well designed we mean with clear and simple illustrations and with no extra details, i.e. details that are unimportant for the actual game. If we look at traditional computer games, one finds that they very often show both exteriors and interiors with a lot of details that might appeal to children, such as toys, animals, cars and so on. These kinds of details are not always necessary parts of the game, rather they are often included to make the games more attractive for the children who play and in many cases they are attempts to create an authentic milieu. Those elements could distract a child with visually impairment; they very easy lose their focus and have it very hard finding the main character in the games.
When we started to produce the computer games for the TPB website, our first goal was to create very clear pictures with few details. The definition of a clear picture is not obvious; it depends on how it is used and on the needs of the target groups. In the TPB games, a simple and clear illustration is typically a depiction of an object showing its distinguished features. It does not necessarily look naturalistic in the same way as photography does.
We also decided to involve slow movements in the games, which made it possible for the children to follow the motions and also stimulate them to really look. However, many partially sighted are not very interested in using their sight because of the big efforts and the small rewards. In general, visual representations contain pictorial elements that easily could be misinterpreted or hardly perceived at all by a person with visual impairment. As a result, many people who have limited sight do not expect any useful information from pictures.
The computer games intended for partially sighted consist of three categories of games. The first game can be categorised as a play environment, since the main objective is exploring. Next there are the puzzle games: four different memory games and three jigsaw puzzles. The two other games, the Beetle game and the Balloon game, can be classified as action games since they require fast and skilled input from the played.
In the discovery game, the child visits the house of a small troll. By clicking on objects found in the house, the child can activate various events such as animations, sound effects and voice recordings. The small troll accompanies the visitor through the house, where each room offers new experiences. This game is based on a subject interface, which means that the player has a character that he/she can recognise and follow. Research has shown that the presence of a familiar character stimulates the player to continue exploring. The small troll does not appear in every room, although she is represented by her voice, telling the player what to do. In one of the rooms, the child meets the grandfather of the troll. In another room the child can look around in the kitchen. The player can also dress a baby troll in different closes that can be combined in several ways, which one young boy, 6 years old, enjoyed very much during a test session.
Figure 1. The kitchen of the Troll game
There are three different kinds of jigsaw puzzles, with three variations of motives: a horse, a snake and a castle. The horse puzzle starts by briefly showing the player the final picture, before he/she can start putting the pieces together by dragging them into a frame with the mouse. The other two do not start by showing the motives, so the player has to find out the motives by putting the details together step by step. At the start of the game, the pieces are randomly ordered, varying from time to time. The design of the illustrations of two of the jigsaw puzzles is based on simple, distinct shapes that are characteristic for the objects; the horse and the snake. It demands a great talent in drawing to manage to find the simple outline of a horse and transfer it in to a very simplified drawing. Maria Beskow has made the drawing of the horse in profile, except for the head that is shown enface. When the jigsaw puzzle is put together, the horse moves its head and neighs. The second jigsaw puzzle depicts a snake. The image does not show a snake of any specific species, but one can easily recognize the animal both by its outline and by its motion in the animation that occurs when the puzzle is completed. The content of the third puzzle is a castle, or rather the symbol for a castle. It is a building with two towers, which are the significant features that one generally associates with castles. However for children who lack the knowledge of how to interpret these symbols, the motive of the jigsaw puzzle is not understandable.[8] A clear layout of a picture does not necessary implement an easy interpretation. Even a picture that appears as very simple is often quite arbitrary.
Figure 2. The Snake jigsaw puzzle
The four memory games have different motives and vary in layout. Two of them feature cards with different symbols, such as very simplified stars and moons, semi-circles, circles, triangles and squares. One game has objects with light colours on dark backgrounds, and another has inverted images, with dark figures on light backgrounds. The third memory game consists of abstract moving objects and the fourth of different farm animals. This variation in motives poses different challenges on the player, since the pictures are of many themes more or less abstract and static.
Figure 3. One of the Memory games
In the Beetle game, the player shall try to stop beetles from eating pieces from a cake. If the child does not manage to stop a beetle, it will take a bite from the cake. The angle from which each beetle turns up is random. The main challenge for the player is to synchronise the movement of the mouse with the movement of the beetle across the screen, which requires simultaneous awareness of two moving objects. In the end of the game the number of beetles defeated is displayed.
Figure 4. The Beetle game
The objective of the Balloon game is to smash balloons by shooting arrows at them. This is done by clicking the mouse when the arrow turns up in the lower part of the screen. In this game, the arrow can only be moved in one dimension, from left to right, although the main challenge is more analytical as the player has to predict how far sideways the balloon will move before the arrow reaches it. It is possible to play five different levels in the game. As one progresses from the first level too the fifth, the balloons move faster. At the end of each level, a staple indicates how many balloons have been smashed but the player is not told the exact amount.
Figure 5. The Balloon game
Sound in computer games
While most common computer games today feature high quality sound effects and soundtracks to accompany the animated graphics, the sound content of most games is generally added only as an embellishment. A common feature in games is a decorative soundtrack that plays constantly in the background. For many partially sighted children, such soundtracks are experienced like "fog for the ears", something that disturbs the focus on the pictures and thus destroys the concentration. Neither do soundtracks enable blind children to play games. Many games do have sound feedback that accompanies events and objects, but it seldom contains enough information for blind players to understand what happens in the game. Furthermore, the audio cannot be clearly connected to a specific position at the screen, which means that one cannot get the necessary information about the game space by sound only. In nearly all common computer games, one needs to receive visual information to understand from where one got the sound feedback.
This means that mainstream computer games generally are inaccessible to blind children. Therefore TPB started investigate how to develop computer games that rely entirely on sound output. This is a little investigated field, however, in the last few years, there have been other attempts in several countries to create commercial computer games for visually impaired gamers. At the start of the TPB audio game project in 2001, we knew of very few sound-based games, but today there are some games commercially available, especially American titles. (See Appendix for a list of producers.)
Even though our sound-based computer games feature the same type of animated graphics as the other TPB games, they are primarily designed so that they can be played without using the computer display. The mouse is not used for input, so the games can be controlled with the arrow keys of the keyboard. The four audio games published, Towers of Hanoi, Memory, Tag and Skybells are possibly the first web-based games designed for blind players. The combination of a complete auditory and visual interface enables visually impaired children to play together with sighted friends. However, while the animations are supplementary, they can in fact make the games quite a lot easier for a person with limited sight.
Below, the TPB sound-based games will be described from an entirely auditory perspective to emphasize the unique issues involved when designing the audio interface. The games have quite different content and pose different challenges on the player. Together, they demonstrate a variety of challenges faced when designing sound-based interactive media. As the following discourse benefits from a good knowledge of the games, the reader is recommended to test them in parallel to this text.[9]
Conveying space with sounds
In the sound-based games, the player obtains a mental image of the objects in the game space by listening to the sounds that they are associated with. As we guessed that this process can take some time, all the sound-based games were designed to stage few objects with simple spatial relationships. The first two games published, Towers of Hanoi and Memory, are both puzzle games that can be explored at a pace controlled by the player. These games depend on the player to check different positions in the game field to find out what is there. This way, the sounds always appear in sequence, which relieves the player of having to take in too much information simultaneously. If one instead uses continuous sounds to represent different objects, the sounds blend into a more complex code that can be difficult to decipher.[10]
Figure 6. The Towers of Hanoi
In all the audio games, the sounds are positioned in stereo, which allows them to be separated spatially on a left to right axis. This spatial information is crucial to the gameplay, which means that the player needs to set up speakers correctly for the games to make sense. However, as stereo only represents one dimension, other spatial dimensions have to be simulated. Height is represented in three of the games, using two different strategies. In Towers of Hanoi, the convention is that when checking a pole, the objects on it are presented from the bottom and up, whereas in Memory the pitches of the sounds that represent card positions change as one moves upwards or downwards.
Figure 7. Memory game
Skybells also uses both strategies to indicate how close the ground the falling objects are; the pitch of the bells change as they fall and the sounds are separated in time. In Tag, height is not of any interest to the player, as this game moves in depth. The Tag game also has a different approach to space than the other games since all sounds are heard from a first person perspective.
Figure 8. Skybells
Since stereo sound is limited to one dimension, it is considerably harder to convey spatial relationships with sound than it is when using visual displays. Graphical displays feature two dimensions, where the third, depth, usually is simulated by graphical trickery. While there are ways to simulate spatial sounds and model acoustics quite realistically, this requires very advanced algorithms that to some extent are included in leading 3D game engines such as the Half Life and Quake engines. However, sound reproduction is generally more unreliable than graphics, as the acoustics of the room where the game is played alter the sounds emitted from the speakers. The sounds in the TPB games are more consistent when using headphones instead of speakers.
Timing sounds
Many mainstream computer games feature fast-paced action that requires speed of reaction rather than the problem solving skills emphasised in puzzle games. Action-oriented games are very popular amongst sighted computer game players, and they can also be designed as audio games. Our experience from the project is that using sounds, it is easier to indicate action and temporary events than it is to convey spatial overviews.
Figure 9. Tag
In the two action-oriented games produced by TPB, Tag and Skybells, the game rules are not so difficult to learn as they are in the puzzle games. Instead, mastering the input system is the challenge in Tag and both action games require fast reactions. (As the pace of events readily can be varied in action games, different game levels are easily created, which increases the lifetimes of the games.) However, the main challenge for the game designer is to plan the timing of events in a functional and aesthetic way.
When several different events occur simultaneously and in sequence, as they do in Tag and Skybells, the emerging sounds will generate rhythms. In the Tag game, the musical quality of this rhythm is emphasised as the events are timed so all sounds fit into the musical flow without interrupting it. This approach is intended to give the player a feeling of influencing a soundtrack similar to the music of films or cartoons. A drawback of enhancing the musicality of a game this way is that it often can decrease the player's immediate control over the events.
Categorising audio information
In sound-based applications, all information is communicated aurally by an auditory interface. Different game concepts rely on different ways to convey information. The following system is a suggested way to categorise different types of information that sounds can convey in computer games.
The puzzle games rely mainly on event sounds. These sounds refer to a brief event, generated by player activity, such as checking positions and moving objects.
Some event sounds are indirect effects of player activity, such as winning or losing and can be called effect sounds. These sounds are results presented by computer, the sounds of cards being turned back, or reward sounds when the player wins.
Next, there is action sounds, which are sounds initiated by the computer. Such sounds are central to the action games, where the player has to react quickly to events that occur somewhat randomly.
Object sounds indicate the presence of objects. In the action games, these sounds are mostly loops, such as the sounds of the running person in Tag, the sound of the truck in Skybells. In the puzzle games, objects are presented only when checking certain position in the game space, which is why they belong to another category. Ornamental sounds are not necessary information for the actual gameplay, although they enrich the atmosphere of the game. In the sound-based games such sounds are only used as background music in the game information menu, since they might distract the player if they occur during a game. In the image-based games, ornamental sounds are used in the Beetle Game during gameplay, but in the other games they occur very scarcely, except for as introduction melodies.
Another way in which sound is used is as instructions. In the TPB games most instructions are speech recordings found in the menu or between game levels. However, some instructions occur during the gameplay in Tag, where the person tells the player to press Enter to start the game.
Sound sources
In audio games many different types of information need to be presented. To distinguish between different types of messages, auditory interfaces can rely on speech, music or "sound effects" to convey information.[11] The TPB audio games use several different kinds of sounds to indicate events, effects, actions and continuous processes within the game environment.
In all the TPB games, recorded speech is only used for communicating very precise information, such as the menu posts and game instructions. However, inside the actual games, the use of speech is limited. The main reason for this is that speech generally is too slow to communicate events that occurring in a high tempo. Furthermore, spoken messages tend to grow tedious if repeated too often. Neither is speech suitable if more than one message is conveyed simultaneously.
In most cases it seems better to convey audio information with recorded sound effects and music. If possible, is convenient to communicate the game content and events with sounds that the player has some previous experience of. Therefore one often tries to use authentic sound recordings, which means that the sounds actually relate to the objects represented in the game. In the Towers of Hanoi game, we have used sounds that are correlate with the objects or events they are to illustrate, so the stone discs make "clink" sounds, and the wooden poles sound somewhat like poles being stuck into the ground. When moving a disc sideways, one hears a "swish" that gives the impression of an object being moved, even though stone discs rarely make such sounds.
However, some objects and events do not relate to sounds in a straightforward way in real life, so authentic sounds are not always available. In other cases authentic sounds can be difficult for a listener to recognise. It can then be necessary to represent objects and events by completely abstract, musical sounds. While abstract sounds might take longer for the player to learn, they can generate very pleasant, musical interfaces since the sounds can be chosen considering their aesthetic qualities. In the Memory game, we have used pitched percussion instruments to indicate different positions in a grid of cards. These sounds were chosen because the action of checking positions is not clearly associated with sounds in the real world. Other examples of events that are musically illustrated in Memory are the rewarding sounds played when finding a pair of cards or winning the game, and the error alert sound heard when attempting an impossible move.
The reception of the computer games
The computer game project has involved pilot studies on two different occasions. We met five children, between 6 and 12 years old at a first study in 2001, and four children between 9 and 12 years old in a second test in 2003. In the first test we focused on the games for partially sighted; how and if the games work for a child with limited sight. To be able to analyse how the children perceive the screens and the single elements, we made an eye tracking study. All the children in the group have heavy nystagmus (constantly uncontrolled eye movement). Nystagmus is very common among people that are partially sighted. Scholars have discussed if the consequences of that could lead to limitations in the ability to interpret visual information, especially represented with images. What we found though, was that the children had no problem playing the computer games and that they had a sufficient coordination between what they looked at on the screen and the hand movement controlling the mouse. This meant that they could play the games without problems. This was distinct in the Beetle Game, in which it is important to synchronize the movement of the mouse and ones sight to be able to smash a beetle before it takes a piece from the biscuit.
Surprisingly, the children found the horse jigsaw puzzle harder to play, because the time that the motif of the puzzle was displayed before the play starts was to short. The game starts when the child takes one of the twelve square pieces that, via rollover, should be placed in the right position. To solve this task consciously, and not by chance, one has to remember the outline of the horse, and be able to associate the different details on the pieces with a specific part of the total image.
When we two years later, in 2003, had finished the sound-based games and tested them we found that Memory was hard for the blind children to play. Three out of four children could not manage to figure out the organisation of the cards on the screen. The one who managed had played the game earlier together with her father and, together with him, created a mental map of the surface. The girl and her father counted four rows with three vertical numbers in each. The structure could be compared with two Braille cells, although the second cell continues with 7 and do not start over again with 1. Only one child in the second group was legally blind. So, as a second trial we asked the children to play by using their sight. One of the girls, 12 years old, classified as blind with an almost non-existing sight, made a better result with the memory by help from the graphics.
Skybells, like the memory, is space related in the sense that the player here has to understand that the bells and/or the stones fall in four different positions sideways. The problem for the children was two folded; to get the positions and to understand how to avoid the stones and to catch the bells. The children found the game much easer to play when using headphones instead of loudspeakers.
One child with some limited sight, who after a careful explanation could play Skybells relatively systematically, had a big problem when asked to use his sight. Though his sight is very limited and he first learned the game via the hearing, he gets confused when he gets both visual and auditative information. In this specific case the picture and the sound did not complement one another for the child.
The biggest challenge in the Tag game was to handle the arrow tangents/keys. The player has to be fast, but not to fast, just in time to catch the figure Bip. If the player starts too early, Bip will comment on that, which the children find very funny. They liked the interactive part and chose to start too early to get feedback from Bip. One of the children that we met in our pilot study had motoric problem that leads to some difficulties in managing the tangents. Since he found the game very stimulating it will work as a platform for practising and challenging his motoric limitations.
A great challenge for the children is the Tower of Hanoi, especially for the children who do not have access to the visual representation of the objects and have to rely on sound only. To be able to solve the problem the player has to understand the spatial relationship between the objects before it is possible to start moving the bricks. The player's mental image has to fit in with the actual representation of the game space in order to make it possible to grasp how the moves should be carried out. Tower of Hanoi has not been solved by children younger than 9 - 10 years old and they need careful instructions before they start the task.
Conclusions
The game project by TPB has shown that it is possible to create fun computer games for visually impaired players. The published games are of two kinds, sound-based and picture-based. It is in many ways a great challenge to create games based on auditory interfaces. To a large extent, this is due to the spatial limitations that characterise auditory interfaces. Pictures are exceptional since they visualize things such as appearance as well as size and spatial relationships. Sounds, and also written texts, are generally very limited in these areas. A visual interface offers the player of a computer game an overview that makes it possible to start investigating a game at random. In our pilot study we found that for a child with serious visual impairment, it was much more complicated to play Memory than Tag because of the puzzle game's lack of overview. Therefore, it is important to provide clear and coherent instructions for games that require a spatial understanding to be solved. For younger children, the instructions have to be introduced by a person, it is not enough with a pre-recorded verbal instruction that accompanies the game.
The game elements developed in the TPB project can be used to develop larger and more advanced computer games for visually impaired players. In our work, we have reached the limits of the web audio features of both Flash 4 and Flash 5. To create more advanced web-based audio games, on could use software such as Director or Wild Tangent. However, with high downloading speeds of broadband Internet connections, many computer users can today download very large games from the web and then save them onto their home computers. With the additional possibilities to distribute games on CD-ROM or DVD-ROM, there is little need to restrict ideas for future games to the limited audio features of web browser plug-ins. Then more advanced audio features, such as surround sound output, real-time effects and more realistic acoustic modelling could be used in the game design. Another interesting possibility is to use advanced 3D sound modelling algorithms (binaural sound engines) that simulate surround sound output from normal headphones.[12] Tools like these open up endless possibilities for creating complex and realistic audio games that avoid many of the problems of sound positioning described above.
Still, regardless of style and software used, several of the issues discussed in this document will frequently arise when attempting to design sound-based games. A major obstacle when communication relies on non-speech sound is the lack of conventions that one can base an auditory interface on. While Western culture has a rich tradition of visual iconography, there is no well-established auditory counterpart. The lack of an established canon of auditory icons means that players of audio games need very elaborate instructions before they can start playing. However, one can look for design inspiration in the musical languages of film and cartoons, radio plays and programme music, learning from their use of expressive sound effects and music.
Appendix. List of available audio games
It has been very interesting to follow the development of other sound-based games during the two years we have been working on the TPB audio games. Since 2001, there has been an impressive growth in the number of available games, game developers and academic interest in audio games. Below is a list of selected links for further research (all accessed in December 2003).
Web resources for information about accessible games:
AudioGames.net http://www.audiogames.net
Audyssey Magazine http://www.audysseymagazine.org/
Game developers:
Bavisoft http://www.bavisoft.com/
BSC Games http://www.bscgames.com/
ESP Softworks http://www.espsoftworks.com/
GMA Games http://www.gmagames.com/
Kitchen's Inc http://www.kitchensinc.net
LWorks http://www.l-works.net/
PCS Games http://www.pcsgames.net/
Pin Interactive http://www.pininteractive.com/
SIT, Resurscenter Syn http://www.sit.se/net/
Sonokids.com http://www.sonokids.com/
Soundsupport.net http://www.soundsupport.net/
The Blind Eye http://www.TheBlindEye.com