| A BBC Domesday System |
|
Click on the emulated BBC Domesday system screenshots in this article to view them at full size in a new browser window
On Monday, 2nd December 2002 I was privileged to be able to attend the Emulation and BBC Domesday Conference at Leeds University. This two-hour event, which started at 4 o'clock in the afternoon, represented essentially the culmination of the CAMiLEON Project's work on retrieving the BBC's Domesday Project from the mid-1980s in order to allow it to be used on current hardware and to preserve it for posterity. The event began with presentations from three speakers and then continued with demonstrations of the BBC Domesday system running both on original hardware and on modern equipment in the form of emulation. In attendance were several individuals involved with the original project.
Readers whose computing memories don't go back as far as the mid-80s, or who were unaware of the Domesday Project at the time, may not understand the magnitude or significance of what it represented, so let's begin with a summary of what the BBC's project was all about.
| A BBC Domesday System |
|
The Domesday Project hardware comprised an enhanced Acorn BBC Master-series microcomputer with a trackerball input device, a Philips LaserVision LV-ROM player and just two double-sided video discs containing all the amassed data in a mixture of analogue and digital formats. The analogue data comprised both static and moving images whilst the digital data included text, statistical and map data, plus application software.
The map-based Community Disc was navigated via a map of the UK which the user could pan around and zoom, linking into information about local communities in photographic and textual forms. The information on this disc was contributed by over a million school children from all around the UK, together with community groups and other researchers.
The topic-based National Disc comprised mainly statistical information, sets of photographs of UK life and culture and a great deal of video footage. This material was gathered from a wide variety of researchers, photographers and scholars.
|
The title page of the National Disc Note that the colourful 'figure of 8' shape shows the layout of the virtual art gallery |
|
Whereas the National Disc provided an overview of Britain, the Community Disc represented the country as seen by its own inhabitants; its content and subject matter was left largely to its contributors, who were organised by a contributions co-ordinator elected by the Local Education Authority. In order to gain comprehensive coverage of the country as a whole, the UK was divided up into blocks of 4×3 kilometers (E/W×N/S), with one contribution coming from each block. There were around 24,000 such blocks in total, and each contributing group was asked for three photographs and twenty pages of 40-column computer-screen text. Although the text was checked for legal purposes, it was not edited in any way, so that the peculiarities of grammar and spelling of the contributors of the day would be preserved as an accurate picture of people in Britain.
|
|
|
Zooming in over the South of England to Southampton, to find information about the International Ferry Terminal |
|
|
|
A national photographic competition was held in order to obtain thousands of pictures of life in 1980s Britain, and the National Disc contained many of these photographs along with other sets of professional photos. Linked sets of pictures were compiled to create nine individual 'surrogate walks' in which the user could explore a node-based environment interactively. One well-known example was the virtual art gallery in which the user could 'walk' between rooms and examine the pictures on the walls. The National Disc also contained text from newspapers and magazines, video of news and sports events, and statistical data, some of which could be displayed in the form of graphs or map overlays.
|
|
| The virtual art gallery | |
The BBC was the main force behind the project as a whole, while Acorn produced the primary computer hardware (its computers were already used throughout the country's schools), Philips (as the only European producer of videodisc players) was responsible for the player technology, and Logica wrote the application software. The computer hardware was based on a BBC Master which was upgraded with a SCSI card, a Turbo co-processor card, videodisc filing system (VFS) and a trackerball input device. The Domesday application software was written in BCPL (a short-lived precursor to C) in an attempt to achieve some level of cross-platform compatibility, and a version for Research Machines was produced after the initial BBC Master-based release. However, the cross-platform approach eventually accounted for little because various patches were needed in order to run the software on an RM Nimbus system, and the BCPL language never became widely established in any case. An Amiga-based version was mooted but was never initiated.
| To the user, the Domesday system presented a very innovative and intuitive interface to the enormous quantities of data on the discs, combining photos, text and video with overlaid navigation controls. This impressive interface combined with extremely powerful search capabilities to produce a resource which was both easy to use and enormously comprehensive. Following their release on 30th November 1986, BBC Domesday systems appeared in schools, universities and libraries across the country. Unfortunately, for a variety of reasons, the cost of the Domesday systems ended up being very much higher than was originally envisaged: the complete package cost £3990 ex. VAT (over £4500 inclusive), which was four times the original estimate. Despite concessions (£2995 ex. VAT to education customers, and one system at £1495 ex. VAT per LEA), the high price put the systems beyond the means of many institutions which would otherwise have purchased them, and the project was a marketing failure. |
|
The short life of the computer-based Domesday Project is an excellent example of a wider issue: the problem of preserving digital material for the future. This problem concerns not just the degradation of media over time, and the rapid turn-around of storage types in terms of obsolete hardware. It also concerns the ability to continue to access material on future systems when the original systems, on which the accessing applications were originally intended to be run, have become obsolete. In essence, the problem is one of preserving access to information regardless of the media on which it was originally stored and the technological mechanism by which it was originally intended to be retrieved. Technology changes very rapidly, and in a way which may often render static information inaccessible.
So the BBC Domesday Project presents a perfect example of the problems of digital obsolescence. It relied on a combination of hardware and software which not only dates from 1986 but which was proprietary at the time because it implemented capabilities which were beyond those of the normal systems of the day. Some functional Domesday systems do still exist, but they are approaching the end of their working mechanical lifetimes. These systems cannot be replaced because the computers themselves are no longer produced, the discs are of a proprietary format, and the LV-ROM players were designed specifically for the project and are similarly no longer available. The hardware is all obsolete and the discs are of an unusual, non-standard format. Finally, the software which allowed the user to interface with the data on the discs was devised for the proprietary Domesday hardware, and without it the content of the discs is largely meaningless even if it could be retrieved.
It is with these kinds of considerations in mind that the CAMiLEON project was set up. Its aim was to research technical strategies for the preservation of digital materials, using the Domesday Project as a test case. The Domesday Project was chosen for various reasons; notably, its scale, complexity, interactivity and reliance on proprietary hardware make it very difficult to move to other systems, and its historical importance also brought a level of urgency to its preservation. However, the Domesday Project is but one example of a much wider problem, and a problem on which other issues impinge, such as matters of copyright and intellectual property rights.
In attendance at the conference were several members of the original Domesday team, including the primary speaker, Peter Armstrong, who had headed the original project, and Andrew Finney, who was a Domesday Producer. David Gilbert, one of the original authors of the BeebEm BBC Micro emulator which forms the basis of the CAMiLEON project, was present, as were other technical experts. Several important figures from the digital preservation community were in attendance, including representatives from the British Library and the Public Records Office. Members of the local and national press also attended, though the only representative of the Acorn/RISC OS press other than myself was John Cartmell, the new owner of Acorn Publisher magazine. It was interesting to discover that some visitors were former schoolchildren who had contributed material to the project but had never had the chance to see it before. During the demonstration session after the talks, one man in his 30s was able to locate his Domesday contribution on CAMiLEON's emulator and see it in its published form for the very first time.
CAMiLEON's primary focus has been on emulation as a strategy to allow obsolete systems to be run on future hardware. This strategy has the advantage of retaining the original interface devised for the preserved object as well as its informational content. The project was begun on 1st October 1999 and the Leeds-based component ended on 31st December 2002; the Michigan-based component reaches completion on 30th September 2003.
The project's objectives were threefold:
The project was also involved in cost/benefit analysis and researching the necessary level of emulation and appropriate techniques necessary to achieve the preservation of particular resources.
CAMiLEON decided on BBC Domesday as its 'digital object of choice' because it was a landmark multimedia resource of great historical importance, representing a nationwide undertaking on a scale which has not been seen since. Moreover, it was based around a combination of hardware and software which was extremely complex for its era. One aim of the CAMiLEON project would be to find a cost-effective means of preservation of digital objects, and the work of preserving BBC Domesday could act as an exemplar for further work.
The project started work by gaining access to a partially functional Domesday system from Leeds University's School of Geography. The laserdisc player was connected via its SCSI interface to a Linux PC, which was used to read the text and databases; the images were transferred by using a video frame-grabber at maximum resolution and stored uncompressed in order to avoid the loss of quality associated with compression formats such as JPEG. 70Gb of data was retrieved per side of laserdisc; 280Gb in total for all four sides.
Having retrieved the data, the next major step was to develop software that could emulate the Domesday hardware and thus allow the data to be retrieved in an accurate reproduction of the original user interface. The starting point was the open-source BBC Micro emulator for Windows, BeebEm (currently maintained by Richard Gellman), which was enhanced with support for the additional hardware required for the Domesday Project. This included emulation of the Acorn Tube co-processor, SCSI support and the wide-ranging functionality of the laserdisc player itself, complete with video genlock.
After explaining the fundamentals of the project, Jan Wilkinson introduced the three speakers, each of whom gave a presentation lasting for approximately a quarter of an hour.
Peter Armstrong was for twenty years a film-maker and Head of Department at the BBC. It was he who, in 1983, conceived the idea of creating a modern-day Domesday project using the large base of microcomputers in UK schools. He later set up the BBC's Interactive Unit to create educational multimeda and, when the BBC decided that there was no future in multimedia, bought out the department to set up the MMC (MultiMedia Corporation). Following the arrival of the Web he set up OneWorld.net, a network of over 1500 partners organisations around the world, working for social justice.
Peter gave a very interesting and entertaining talk which was full of nostalgic anecdotes about the history of the Domesday Project.
It was decided that the first copy of the completed Domesday resource should be given to Prince William as the 'poetic successor to William the Conqueror'; the original Domesday Book had been commissioned by King William I in 1086 to provide a record of the land he had conquered twenty years previously, in that most famous date, 1066.
As it was soon to be the 900th anniversary of the Domesday Book, it seemed fitting to recreate the project in a modern format using the most suitable technology of the day, the BBC Micro, which also happened to be the computer most prevalent in schools at the time. It would be an extremely ambitious multimedia project of a kind never seen before, involving video discs. It was originally projected that the cost of the final system, including computer, laserdisc player and discs, would be in the region of £1100. Funding for the project, to the tune of £2 million, was raised with relative ease thanks to the innovative and historical nature of the project.
During the summer of 1984 the BBC assembled around 60 members of staff to work on the Domesday Project, and a network of data-gathering resources was set up, comprising teams of data collectors from more than 14,000 schools. This was more than half the schools in the country, and the project involved contributions from over a million children. The entire UK was divided into 4×3 kilometer blocks of land, known as 'units', around half of which were adopted by individual teams of data collectors. (The size of the units intentionally reflected the 4:3 ratio of a television screen.) The information required included a record of land use, numbers of doctors, dentists, post offices, halls and so on; three photographs were allowed per block. The BBC Micro itself was the data-input device: teams would input their articles on their BBC computers, save the data on floppy discs and send them to the Domesday Project at the BBC. As stated previously, the text was not edited in any way (spelling and grammatical errors were preserved) except where prompted by lawyers: some teams had a propensity to write inflammatory remarks about local 'characters'!
|
|
| Tastes of local life from the Domesday Project | |
|
|
In the end, coverage was achieved for over half the country, with information gathered from over 23,000 blocks: in addition to the data collected by the school-based teams, text and images from other professional sources helped to fill in further areas and close the gaps in the coverage. Most of England was included, but not the more remote parts of Wales and Scotland, as it was hard to find sufficient schools in the most remote areas. The unpaid school volunteers were enormously dedicated to the task, and the entire collection process alone took around two years, not counting the development of the system itself.
The hardware development took place in parallel with the data-collection process, and turned out to be a great deal of work. The Philips LaserVision 415 LV-ROM system was a completely new solution for data storage, and had to be coordinated closely with Acorn, which was involved in writing new operating system software such as the VFS videodisc filing system required to interface with it. The player was a SCSI-based device; the first version of the SCSI interface standard had only recently been confirmed at the time. The player also contained genlock video-mixing hardware to combine the computer-generated video output with the images from the discs. Meanwhile, the BBC and Logica were together involved in writing lots of new application software with which users would access the material. The software was complex, involving over 70,000 lines of code, and had many problems to iron out; at ten weeks before the final deadline, the team members were apparently wishing they'd never started on the project! However, it was completed on time and within the established budget.
All told, the Domesday resource involved use of 24,000 original paper maps from Ordnance Survey which were cut, pasted and captured digitally. Photos were also digitised, and each map and photo had to be put into the correct video frames for the disc, along with captions. All copyrights had to be cleared. In all, the process involved over 8000 statistical data sets, 200,000 photos, one million screens and thirty million words.
The volume of material supplied on the Domesday Project was such that it would take a single person seven years, working 40-hour weeks, to view all of it.
The Domesday Project won the British Computer Society's Premier Award in 1986, along with various other accolades, and the very first system was sent to Geoffrey Martin, Keeper of Public Records at the Public Records Office.
Unfortunately, thanks to last-minute price hikes by manufacturers involved in the process, the initial launch price for Domesday systems jumped up from the projected £1100 to £4000. Acorn and Philips did their best to reduce the price, and ESPRIT funding was obtained from the European Commission, but the situation was not helped by the fact that the British Government had changed its policy from subsidising hardware in schools to subsidising software. Since schools formed one of the main markets for the Domesday system, the greatly increased cost led to a significant loss of sales, and the project, for all its innovation, was a commercial flop.
Peter Armstrong rounded off his talk by explaining that the Domesday Project was the grandfather of all future multimedia projects. On a personal level, it led directly to his subsequent involvement with multimedia developments at the MMC, which sold lots of CDs (for example, 3D World Atlas sold over a million copies) but still lost money, and later his Web-based OneWorld.net project.
He concluded by expressing a wish that the BBC would think about making all the old Domesday material available online.
Dr Thomas Graham is University Librarian at the University of Newcastle upon Tyne, and is also Chairman of the Consortium of University Research Libraries (CURL), which represents the large university and national research libraries in the UK. CURL has been active in supporting digital preservation activities for several years.
Tom's presentation centred on the duty of future generations to preserve data, whether or not it was originally analogue or digital. He referred to the CEDARS project: CURL Exemplars in Digital ARchiveS, a collaborative venture under the overall direction of the CURL libraries consortium. The CEDARS project was also based at Leeds University (with contributions from Oxford and Cambridge Universities), and most of its team went on to work on CAMiLEON.
CEDARS encompasses issues such as the collection, management, intellectual property rights, cost analyses and prototypes of digital archiving. It has a high international profile, and the CAMiLEON Project follows on from, and overlaps with, CEDARS but focuses on emulation as the primary means of data retrieval.
The Domesday Project was very challenging in terms of preservation because of its size and the range of its material, but it was also very important for both historical and technical reasons.
A recent initiative, established in 2001, is the Digital Preservation Coalition, a programme which was set up in part as a result of CAMiLEON's activities, and which stresses international collaboration as well as being important on a national level. Such collaboration was of course demonstrated by CAMiLEON itself, which was a shared project between the Universities of Leeds and Michigan. It is hoped that the Coalition will mark the start of a continuing and productive international collaboration, with funding coming from both the United States and the European Union. CAMiLEON has been just one of many such projects, but more research is needed on costs and business models for preservation. The Digital Preservation Coalition's handbook entitled Preservation Management of Digital Materials is available online and makes useful reading for anyone interested in the subject.
The CAMiLEON preservation strategy is, in combination with emulation of original hardware, to preserve the data itself in an original, unmodified format in terms of being an abstract bytestream, though not on the original storage medium. Software can then be written for future hardware to make use of this bytestream. This could take the form of an emulation of the original system, as with CAMiLEON's current software; it could be a tool to transform the data into a modern format; or it could be software to present an entirely new interface to the existing data.
In order to avoid the emulation software itself becoming obsolete, it is crucial to ensure that the software is not wedded too closely to any particular current platform. The BeebEm software used by CAMiLEON only runs on the Windows platform; it was already in existence before the CAMiLEON project came about, and hence was not written with portability or longevity in mind. However, the aspects which have been added for the CAMiLEON Project have intentionally been made much more portable, so that they could be adapted into another BBC emulator, of which there are already several for various platforms. Indeed, the possibility of a RISC OS version of the Domesday emulator has already been discussed. However, it is important to make the point that copyright issues currently preclude the distribution of the Domesday data, and the CAMiLEON team is regrettably not able to provide Domesday access to outsiders.
Dr David Holdsworth is a Consultant in Information Systems at Leeds University's Information Systems Services, and has watched the evolution of digital storage technology since the 1960s. He was the architect of the demonstrator system produced by the CEDARS project.
David's presentation concentrated on the CAMiLEON Project in terms of the survival of access to digital data rather than the survival of the medium itself. Like Dr Graham, he referred to the project in relation to its rôle within the larger CEDARS initiative.
With digital data, the medium itself may survive, but formats change and become inaccessible over time. The work of the CAMiLEON Project will last, though, because of the following factors:
David expressed a hope that the preserved Domesday Project would still be operational in 2086, the 1000th anniversary of the original Domesday Book.
But could the Domesday Project again be made available to all? It would be easy to make perfect copies of the retrieved data, as it is now fully digital, but the question of copyright may be a problem. At least the British Library would be able to have a better copy, though!
Dr David Holdsworth rounded off by explaining that safe stores were needed for digital objects, such as national libraries. The whole issue is a global one which is now receiving international collaboration.
Some time was spent discussing the matter of copyright. Unfortunately it is unlikely that the Domesday data will again become accessible to the public, at least until there is a change in the law, because of copyright problems which are complicated in large part by the size of the project and the number of contributors. Parts of the data are owned by the BBC, parts by Ordnance Survey, and much is likely to be technically the property of Local Education Authorities and schools, not to mention the other professional photos, newspaper and magazine articles and video. It is conceivable that the preserved data and emulation software could be made available to owners of original Domesday systems, but a release to members of the general public, or perhaps through the Web, currently seems unlikely.
The CAMiLEON team has been documenting the process of working on the Domesday project, including a full account of all aspects of the original system.
|
||
| Paul Wheatley, CAMiLEON UK Project Manager, reflects on the National Disc | ||
| Photograph copyright © University of Leeds Media Services |
Thanks to Paul Wheatley for assistance in the preparation of this article, and to Phil Mellor for supplying the screenshots of the emulated Domesday system.
