1. Introduction

For different purposes, computer hardware and software is often tested on certain benchmarks. Although being sometimes controversially discussed, such benchmarks set (at least) well defined environments to compare otherwise incomparable technologies, algorithms, and implementations.

Benchmark suites for symbolic computations are not as well established as for other areas of computer science. This is probably due to the fact that there are not yet well agreed upon aims and technologies of such a benchmarking. However, during the last years efforts towards systematic benchmark collections for symbolic computations were intensified.

Following the trend of the development of Computer Algebra software, we can classify these efforts roughly into two categories:

1. General benchmarks which cover almost all areas of symbolic computation and whose main intend is to compare general-purpose Computer Algebra systems (CAS). The famous Wester suite [13, ch.3], is a typical example of such an effort.
2. Special benchmarks which concentrate only on a particular problem and whose main intend is to compare algorithms and implementations solving this problem. There are numerous special benchmarks for many particular problems scattered through the literature. See, e.g., [1,2,4,8,11,12] for benchmarks of polynomial systems solving or [10,14] for the polynomial factorization challenge.

For further qualification of these efforts it would be of great benefit to unify the different benchmark approaches and to systematically collect the existing special and general benchmark data such that they are electronically available in a more or less uniform way. This would provide the community with an electronic repository of certified inputs and results that could be addressed and extended during further development. The SYMBOLICDATA project is set out to realize this.

However, the aims mentioned above do not reach far enough: symbolic computations often lead to voluminous data as input, output or intermediate results. Therefore, such a project has not only to collect benchmark data but also to develop tools to generate, store, manipulate, present and maintain it.

Consequently, the SYMBOLICDATA project has the following three goals:

1. To systematically collect existing symbolic computation benchmark data and to produce tools with which this data collection can conveniently be extended and maintained.
2. To design and implement concepts which facilitate trusted benchmarks computations on the collected data.
3. To provide tools that allow data access/selection using different technologies (ASCII parser, SQL, WWW, etc) and data conversions into commonly used formats, e.g., HTML, SQL data bases, ASCII, LaTeX, etc.

In the first development stage of the project we concentrated on the general design principles of the tools and the data collection, thereby trying to achieve a balance between the necessary flexibility/extensibility on the one hand, and simplicity/practicability on the other.

A first application of our tools and concepts was realized on collections of data from two areas of Computer Algebra: Polynomial System Solving and Geometry Theorem Proving.

Further applications of our tools and concepts to collect data from other areas of symbolic computation are intended. For this, we seek the cooperation of persons and groups that have related data collections at their disposal and are willing to spend some effort to enter these data into the SYMBOLICDATA data base and provide the respective add-ons to already existing tools.

The SYMBOLICDATA project grew out of the special session on benchmarking at the 1998 ISSAC conference in Rostock which was organized by H. Kredel. Since then, the project has steadily developed from ideas to implementations and data collections and back. At the beginning of 1999, the authors joint forces with the symbolic computation groups of the University of Paris VI (J. C. Faugere, D. Lazard), of Ecole Polytechnique (J. Marchand, M. Giusti), and of the University of Saarbrücken (M. Dengel, W. Decker). Furthermore, the project was incooperated into the benchmarking activities of the Fachgruppe Computeralgebra of the Deutsche Mathematiker Vereinigung.

In this paper we report about the current state of the SYMBOLICDATA project. Based on the general design of SYMBOLICDATA which is outlined in section 2, we describe in section 3 how the above mentioned goals were realized. These concepts are illustrated in section 4 by two examples of data collections from different areas of Computer Algebra. Section 5 gives an overview of what deliverables the SYMBOLICDATA project has produced so far which is finally followed by some concluding remarks in section 6.