c-t Magazine Logo      Original Article in German

Michael Kunze, Hajo Schulz
translated and re-published, with the permission of the authors, by Firebird Foundation members Heiko Tappe & Helen Borrie

Good Neighborliness

c't calls for a database contest

Which databases are suitable for enterprise use? The German IT magazine "c't" is mustering a contest between popular commercial and open source databases.

 Image borrowed from c-t Magazine

The Westphalian Treaty ended the Thirty-Year War in 1648. While the treaty didn't resolve the question of which faith was right, it did ensure a lengthy period during which differences of religious opinion were not proclaimed with pike and musket.

The propagation of different faiths has not dwindled over the years, as anyone can tell by observing some busy IT forums. The zeal of some of those Evangelists of 'true faiths' -- Windows/Linux, PHP/Java or MySQL/PostgreSQL--might well bring to mind some famous Cardinal.

But, these days, if you want to attract new disciples you have to convince them with facts. To discover how these facts shape up for popular database systems, c't wants to kick off an open developer contest where you set out to convince us and the c't readers that your favourite database system is the best!

This competition is more than just the usual benchmarks. Besides providing information about the behaviour of the database system in lifelike conditions, this competition should indicate something about its usability with middleware and server platforms.

In deference to peace, this test should not lay tributes at the feet of the goddess 'Featurama' - nor should it profess any 'one and only' IT dogma. Rather, we would like it to be a commentary on the travails and hardships a pilgrim encounters along his way. So we are interested in clever shortcuts and perhaps the kind of deeds that merit heavenly Indulgences!

In contrast to the more abstract performance benchmarks of the TPC [1] we want to be able to appreciate the power of a solution in comparison to the amount of effort needed to achieve it.

Shop and pray

This contest is therefore modelled solely on a reference implementation of a typical e-commerce application. On the basis of this blueprint, interested developers can create their own solution and send it to the editors for a crtical inspection. All candidates are tested on the same hardware platform on Windows or Linux. In the spirit of open competition, all cheats and tricks are allowed for the implementation as long as they are documented.

The same goes for the software version: as long as it runs stably in the tests and the results are just fine, you can use beta versions or whatever. Read the exact conditions in the box 'In the laboratory'.

To choose an e-commerce test application, the editors checked out several open source projects. The applications considered ranged from pure laboratory applications like Rubis [2] to complex auction applications like PHPBB-Auction [3]. Finally we decided to use 'DVD-Store', a project that the Linux department of the Dell computer company developed in recent years as a test for its internal server laboratory. It is called DS2 [4] and it is free.

The aim of developing DS2 was to create a kind of 'TPC light'. Dell's original purpose for the test was for studies intended to prove how easily enterprise-critical database applications could be migrated from expensive Sun machines to standard 'in-house' PC hardware. This underlying concept makes it almost ideal for the c't database contest.

Its intended use as a laboratory application explains the simple user interface. But the application comes with several features that make it very suitable for a database contest. On the one hand, the DS2 shop uses features of modern database systems like stored procedures, transactions and referential integrity. In in DS2s big brother, TPC-W, such features are called OLTP (On-Line Transaction Processing) and form part of the essential requirements of the TPC-suite. On the other hand, DS2 comes with ready-to-use backend implementations for MySQL, Oracle and MS SQL which gives an interesting insight into the differences and commonalities of the database systems.

Not just the SQL part but also the middleware has multiple variants. Each database solution has three middleware implementations, namely ASP.NET, JSP and PHP. Such variety has its price, which is already addressed by the simple user interface: the authors simply created a very thin program layer and used just the basic functions of each platform.


All the charm of google: the web interface of the DVD shop is quite simple

For instance, the implementations don't use any session management, but carry the session parameters in hidden form fields, the way it used to be done. Doing without any syntactic sugar exposes some interesting similarities between the three concepts. Absent the need of the 'Popes' Balmer and McNeally to guard their power base, one could dream of an ecumenical community of web programming languages.


Performance: For a convenient graphical evaluation of the results the test client sends the most important characteristic numbers to the system monitor as performance indicators.

For the automatic performance test DS2 comes with its own client simulator, a .NET application implemented in C#. Thanks to Mono, it runs not only on Windows but also on Linux without any problems. As in other, similar projects, this application creates a definable number of clients at start-up, which fire hard-coded requests at the database backend.

There are two different versions of the simulator. The first one behaves like a web client and calls the HTTP stack of webserver, middleware and database. The second is suitable for testing database system performance directly, because it acts like a classic SQL client and also permits direct access. Both versions give detailed statistics of the results after execution.

Crank it up?

There are two possible ways to take part in the c't database contest. You can take one of the reference implementations and try to tune it by changing values of different database parameters or by tuning the application source code. Those whose favourite database system or preferred programming language isn't represented can develop a completely new solution. Just keep in mind that the editors have to run the web client simulator on each solution, so your solution has to behave like the reference implementation when run by the simulator.

These DIYs most probably need to get DS2 and run it. From the download site you definitely need the database- independent part which is in the 2.3 MB file ds2.tar.gz. Additionally, you need at least one of the database-specific archives ds2_mysql.tar.gz, ds2_sqlserver.tar.gz or ds2_oracle.tar.gz.

All archives need to be extracted into one directory. Besides other files this directory will then contain a subdirectory data_files with a small amount of import data for customers, orders and products.

For performance checking DS2 uses three different amounts of data: 10 MB, 1 GB and 100 GB. But we will consider only the first two levels for solutions submitted to the editors.

To create the medium-sized test data, Linux users just have to run the three shell scripts ds2_create_*_med.sh over the subdirectories cust, orders and prod, respectively. The generators exist as Linux binaries and C source code, but Windows users will have to compile the C code with an appropriate C compiler first. (A free C++ compiler is included in Borland's 'Free command line tools', Soft-Link 0520156). The parameters for the executables can be taken from the .sh-files referred to above.

The subdirectory, drivers, in the DS2 directory, contains the test client: the two .exe-files simulate a web client and run on top of .NET or Mono. If you want to recreate them you have to compile the two files ds2xdriver.cs and ds2webfns.cs. You can create a client that directly accesses the database if you compile ds2xdriver.cs together with one of the files ds2*fns.cs of the matching database specific subdirectory.

Inside the directories for each specific database are subdirectories built with SQL scripts to set up the database and load the test data with scripts that import the test data and web pieces. In the latter are more subdirectories with everything you need to run DS2 as a PHP, JSP or ASP.NET application on a webserver. In almost every directory a README file explains further details and, in some cases, provides very detailed directions for navigating the sources.

Let's go!

The c't database contest is not expressly pitched at hobbyists and open source developers. We are also interested in solutions from database developers and professional web application providers. Closing date is November 13th, 2005.


Database structure: The database structure of DS2 contains all that's needed to run a small or medium web shop

As soon as the solutions are tested and evaluated we are going to present the best solutions in detail. There are prizes in several categories: first of all there is the winner for overall best DVD shop performance. Apart from that we want to find out which entries achieved a well-perfoming and robust solution with minimal effort. You can also collect winner points if you have smart ideas to work around any limitations of a specific platform, or if you found tricky ways to boost performance or if you can win our hearts with the elegance of your solution.

We created a forum for the combatants at ctmagazin.de/dbcontest. There you can also find a form to submit your solution. The editors are really looking forward to getting a large number of entries.

In the laboratory

The editors will test each solution of the DVD shop with the same hardware. Two different test scenarios will be used: In the first, one webserver, middleware and database will run on the same machine. This scenario is what most big web hosters offer for professionals and small companies.

The second scenario moves the database system to a separate but similar server. This setup is usually used by companies that have higher requirements and administer their own network structure or use special network providers. With 'high-load' websites, this setup would be likely to use clusters of several machines in both the front-end and back-end infrastructure. However, a database clustering feature should not be considered a criterion for the contest.

The hardware itself consists of server PCs with 3 GHz CPU, 1GB RAM and a 150 GB SATA hard drive. We will use Windows 2003 Server and SuSE-Linux 9.3 as Operating Systems. For the webservers we will use the current releases of Apache2 and IIS for Windows.

For middleware and additional programs there are no restrictions. However, those who want to use commercial products are responsible for ensuring that the editors are supplied with the required software at no cost and have the necessary permission to use it. Components must be installable without the need for a development environment. This means they have to be available in the form of a precompiled installer, appropriate to the prescribed platforms (.exe, .msi, .rpm).

Any solution sent to us needs at least enough documentation so that it's possible to install it without problems. If you use esoteric database systems and middleware or unusual solutions, we'd like to have more specific documentation. And we are especially interested in the amount of time needed to develop that application. If you use external libraries or software whose origin is not obvious, please let us know.

Finally, every participant in the contest has to agree that his or her solution may be published in printed form or online by the publisher Heise-Verlag.

Sources and Links

[1] Transaction Processing Performance Council, www.tpc.org

[2] RUBiS: Rice University Bidding System, http://rubis.objectweb.org

[3] PHPBB-Auction, www.phpbb-auction.com

[4] Dell DVD-Store Referenz, http://linux.dell.com/dvdstore

[5] c't Datenbank-Contest