Written by David Tebbutt, Mensa item some time in 1991 - scanned
Apple Computer broke completely new ground with the launch of its Macintosh computer. It was the first successful mass market computer that was easy to use. By standardising on the appearance and behaviour of all programmes, suddenly users found they could migrate from one application to another with the minimum of effort.
Although the original ideas for the user interface were inspired by work done at the Xerox Palo Alto Research Centre, Apple was the company that brought them to a wide and appreciative audience. Now, all the major computer companies are adopting similar interfaces. The others are all grafted on top of existing operating systems, whereas the Macintosh was built from scratch with the graphical user interface in mind.
New, demanding applications are stretching the ability of today's interfaces to cope. As the companies try to catch up, Apple continues to research new and better ways of interacting with the machine. Much of this research takes place deep inside Apple, in the Human Interface Group, which is itself part of the Advanced Technology Group.
The group, headed by S Joy Mountford, includes psychologists, mathematicians, artists, computer scientists and designers. This group is split into a number of teams, each of which has members of varying and complementary skills. One team is researching hand-held computer interfaces, another looking at the potential of speech, a third at the incorporation of video in the computer, and a fourth at the high end computers of the future, beyond Apple's present range.
We've already seen the huge increase in popularity of notebook computers. These small A4 pad-size machines pack the power and storage capacity of yesterday's huge lumpy desk-top machines. Further reductions in component size and the abandonment of the keyboard for some users will result in even smaller machines in the future. We will need new ways of interacting with such machines, so you won't be surprised to learn that the Human Interface Group (HIG) is already working on them.
Rather than using a mouse, or a touch tablet, with which to place an on-screen pointer, we will do this directly by touching the computer screen with a stylus or with our fingers. We will be able to treat the computer like an infinitely large notebook with simulated pages which can be turned with the slide of a finger. We will be able to write directly on the screen with the stylus. And we'll have a selection of icons with which we can mark individual documents for later retrieval.
A research prototype at Apple was of a still video camera. Now this might not seem to have much to do with computers, but it actually highlighted many of the techniques which will be needed on keyboard-less computers. The camera stores its images internally and displays them on a screen on the back of the device. The user wants to be able to browse through the pictures, select only pictures which meet certain criteria for showing other people, and to annotate them with electronic ink and sound.
These requirements aren't that different to computers without keyboards. The camera allows the user to draw on an imaginary transparent overlay which covers each picture. It allows the user to record a sound to accompany each picture. It might be a 'voice-over', or the ambient sounds at the time of taking the picture. Picture browsing is simply a case of sliding the finger across each image, as if the picture is being flicked to one side. A set of icons to indicate people, dances, buildings, views and so on can be called to the screen. Each picture can have an appropriate collection of icons associated with it. A group of friends in front of St Paul's Cathedral could have 'People' and 'Places' icons. When the time comes to flick through the snaps, you can just choose 'Places', for example.
On a computer, you might want to scribble notes and associate icons with them - 'Call Tebbo on 01895 677845' might have a 'Telephone' icon associated with it. If I wanted to gather all the telephone numbers, I could use the telephone icon and get the number listed, together with the adjacent note. If I wanted to review an article I'd written on my main computer, I could squirt it across to the notebook computer and make my handwritten corrections on the train journey home.
Of course, the team faces challenges with this work. How can the computer tell the difference between a deliberate gesture with the finger and an accidental one - as you point something out to a colleague, for example, by jabbing your finger at the screen. Nevertheless, work continues apace, and it won't be long before the stylus operated computer becomes commonplace.
One of the Apple researchers, Michael Chen, has come up with a truly original way of rotating three dimensional objects displayed on a two-dimensional computer screen. Rotating things in two dimensions is relatively easy. Although the programming is difficult, for the user it is simply a case of grabbing a point on the image and then swinging it round to a new position. Three dimensionality brings the problem of depth.
Traditional methods include three horizontal slider bars - one for the x axis, one for the y axis and a third for the z axis. This method was improved when someone (I think it was Chen) had the bright idea of overlapping the sliders so that the x and y sliders formed a cross. The x axis was an imaginary ring which ran around the overlapping central area.
This was the start point for Chen's inspiration. He realised that if the operator could imagine the screen object encapsulated in a transparent sphere, movement in all three dimensions could be achieved by imagining the screen pointer to be on the surface of the sphere. He put a reference circle around the object to show where the three dimensional sphere intersected with the two dimensional screen. Both formal research and my own subjective experience suggests that the 'virtual sphere' is the best all-round method for this kind of manipulation.
Another key area of research is into the recognition of speech. Different companies researching speech have different motivations. Some want to mimic a typewriter, some want to recognise key words from any speaker, some want to recognise continuous speech from a single speaker.
Some companies have already demonstrated speech as an alternative method of controlling computers. This is fine when the user is asking the computer to carry out a complex activity but, as a replacement for cursor keys, it is a complex waste of time. No-one wants to sit at the computer saying 'up, up, up' or 'down a bit'. It is easier to use the mouse or cursor keys.
On the other hand 'call the wife' is a very good use of speech because it triggers a sequence of events. First, the computer understands its user saying 'call' and 'the wife'. It translates 'call' into an action to initialise the telephone or modem. 'The wife' gets the computer to look in its files for a number corresponding to 'the wife'. It might also look at the time and day and decide whether 'the wife' might be at home or in the office.
This is the area of research that HIG is engaged in. Once the Macintosh can respond to the spoken command, the machine becomes many times more useful. If it can be operated from across the room, there's no reason why it shouldn't be operated from across the world. As long as its user can make a telephone call to the speech recognition components of the machine, the computer can be triggered to make telephone calls, send faxes, prepare reports, tell the caller what messages have been received on the electronic mail and so on.
The graphical user interface was the last major jump in computer functionality from the user's point of view. The next major jump may come when the technology advances enough to support speech in the ways described above.
Another area of great potential for computers is the creation of short movies and their inclusion in documents such as reports and electronic mail. Imagine receiving a newspaper, turning to the sports pages and, instead of seeing a football in the corner of the net, touching the picture and watching the sequence leading up to the goal, just as if it were on television.
At a more personal level, this is the kind of technology we can expect to see in personal computers. You could go to an estate agent and, instead of a couple of sterile photographs of a house, you could actually tour it by watching the video. Before we can all use this kind of technology, it needs to be made easy to use. None of us can afford to become film directors. Once again, Apple's HIG is at the forefront of research in this area. It has created a number of tools to make moviemaking as easy as word processing. Apple's HIG has produced a targettable controller which lets you control an external video device from the Macintosh and display the results in a window on the computer screen. It also has a movie grabber which lets you pick the start and end points of movie fragments and bolt them together into a continuous sequence. This controller lets you put the end before the start, to give reverse video. It also lets you choose a smaller area of the original start or end screens, thus creating 'zoom-in' and 'zoom-out' effects.
Of course, no self-respecting movie would be without its fades and wipes to smooth out changes of scene. Once again, HIG is working on this. It has developed a tool called the transition factory, which enables you to select from a series of transition effects. With tools like these, it won't be long before we are all capable of producing very professional video sequences and sending them to our friends and colleagues.
The team is looking at the next stage in computing, using technologies that will emerge in the next couple of years. Computers have increased in power and fallen in price continuously since they were first invented, so we can assume this trend will continue. Today's inaffordable office computer will be in everyone's briefcase in five year's time...
Apple's HIG is looking carefully at how users can handle the power and complexity of these machines and the masses of data they will be able to access both locally and over the telephone networks. It is likely that the computers will use some of their power to become easier to use. For example, rather than have to dive around from one computer programme to another, it is likely that the tools for the various tasks will be available at all times. They will live in tool drawers which can be slid out from the side of the screen. As users buy new electronic tools - paint brushes, connections to remote databases, pencils, typefaces and so on - they simply drop them into compartments in their tool drawers, where they appear as small descriptive icons.
Documents will be layered so that annotation may be stored separately to the underlying text, or sketches can be built up, layer by layer, into a finished drawing. This would be particularly useful for designers and architects who presently have plenty of support for creating finished drawings but little for the creative process. As we become overwhelmed by information, both in our own computers and in remote systems, we need to find ways of using the machine to help. HIG has devised a number of strategies which centre around the user providing 'clues' about the sort of documents they are seeking.
This article has only been able to give a glimpse of the work going on at Apple's Human Interface Group. It's nice to know, though, that somewhere in the computer industry there's a group of people who have our interests at heart.