Benchmarking kaffe (Was: Re: SPECjvm98)

Dalibor Topic robilad at
Tue Mar 26 19:01:47 PST 2002

On Tuesday 26 March 2002 14:07, Jukka Santala wrote:
> On Mon, 25 Mar 2002, Dalibor Topic wrote:
> > Sticking to a standard environment would just limit the number of
> > people able to contribute results.
> That's one of the things I'm afraid of. The last thing we want is people
> upgrading their compiler/libraries on the run, and forgetting to mention
> it in the benchmarks, leading to everybody think they've broken something
> terribly, or found a new optimization.

O.K. Writing a script or a java program that compiles --version information 
for tools and libraries used should be possible. I'm in favor of automating 
the process as much as possible: 'make benchmark' and you get a bench.txt 
file at the end with all relevant configuration information and the results. 
Put a benchmark toolchain definition for that release somewhere where it gets 
parsed by the benchmark script. Let the script flag "non-standard" entries.

> > What kind of contribution process would be suitable for such an
> > effort? Emails to a specific mailing list? Web forms?
> Well, I was most initially thinking of having both a gnuplot graph of the
> development of the benchmark performance over time, as well as a textual
> log of the specific results. In the most simplest case, this would only
> require an e-mail notification of the location of the graphs to this list,
> and the URL could then be added to the official web-page if deemed
> useful/reliable enough. If enough data is provided, it might be worth it
> just to write a script on the web-site machine, that would gather the
> benchmark logs and collate combined graphs from them.

sounds right.

> But, as implied, if we're aiming for just "any benchmark", for posterity
> and some pretend-comparisions between system perfomances, then all bets
> are off, and we should probably have some sort of web-form for users to
> input in that "Herez the rezultz I gotz from running my own
> number-calculation benchmark, calculating how many numbers there are from
> 1 to 1000 while playing Doom in another Window. This is OBIVIOUSLY what
> everybody else will be doing with the VM's, so I think this counts. I'm
> not sure I even have a compiler." ;)

Uh, no, thanks :)

But that raises an interesting question: which benchmarks would matter? For 
example, I assume that benchmarking kaffe's as a platform for's 
java projects might be interesting, since a couple of responses to the "most 
popular applications" thread had those mentioned. What could Ashes cover?

dalibore topic

Do You Yahoo!?
Get your free address at

More information about the kaffe mailing list