[kaffe] Re: Regression/build testing using Xenofarm
robilad at kaffe.org
Mon Jul 28 05:24:02 PDT 2003
Jim Pick wrote:
> Ultimately, I think we'll end up with a variety of regression testing
> systems. I'd like to get results for Gump, Mauve, SpecJVM, and others
> up on the website.
> Here's what I've been thinking:
> 1) Let's not have one "official" regression testing framework. Let's
> encourage people to develop their own regression testing systems in
> a distributed manner.
> 2) Let's define a standard reporting format, as an XML format. People
> format their results in this format, and upload them to the kaffe.org
> 3) I'll write some scripts to suck in the various submitted XML files,
> and produce useful reports. We could even use a database (for
> example, to track benchmark performance over time).
> 4) Install something like CVSps so we can more easily cross reference
> individual checkins.
That sounds like the better plan in the long run. Though it also sounds
like a nice project in itself, "designing and implementing a
regression-test meta-framework". ;)
I'm not quite sure about 2) though. What should go into the reporting
format? A lot of build tools just output (not very formatted) text to
stdout and stderr. Or do you have filters in mind, that XMLize the
output of 'make', for example?
More information about the kaffe