[kaffe] Improving regression testing
robilad at yahoo.com
Wed Feb 26 13:31:01 PST 2003
I've been thinking recently about adding a
configuration flag, to aid those developers who are
or have to cope with kjc crashing its way through the
suite without a viable alternative.
This is the current regression test system:
* the tests are in test/regression.
* kaffe uses automake's support for test scripts.
* every test is first compiled and then run
* you can pick a subset of tests to run
* you can turn on verbose test logging
* each test is a java file, with some extra comments
at the end
* these comments are used by the TestScript to
determine the expected
output, whether to sort test results and what files
the drawbacks of this system are:
a) it is hard to run the tests on an installed version
b) especially after cross compiling
c) it depends on kjc working well enough to compile
d) it is hard to process failure information
I'll explain the points in more detail:
a) kaffe doesn't install the regression test suite,
optionally. Neither does it install the test script.
So if you want
to run the regression test suite with an installed
version of kaffe,
you have to put a bit of effort into it.
I'd like to have kaffe optionally install a test
script and a
b) When you're cross-compiling, the version of kaffe
most probably can't run on the build system. That
means 'make check'
is not very useful.
If kaffe optionally installed the test script and the
test jar. you
could run the tests on your target. That *should* help
developers to find & fix kaffe's bugs easier.
c) Currently, if the compiler has a problem compiling
a test, the
test fails. If you're using kjc to compile the tests
it means that a rather complex piece of software fails
compiled version of kaffe.
I believe that developers would be able to fix kaffe's
faster if they could focus on getting simple tests to
run, instead of
getting the complicated java compiler to work. I
assume that change
would make the life of porters to new platforms
(One such platform, where kjc breaks apart on kaffe
regression test suite is Cygwin, btw.)
d) In the current setup, when a test fails it produces
test-name.fail file, which can be diffed against the
test-name.out file. It requires manual intervention,
knowledge of the
test setup and diff to log failed tests. I assume
that's the reason
why we rarely see failed regression tests posted to
I think the test script should log failed tests and
automatically, proposing where to send them for
inspection and how to
go about debugging them yourself.
All of this could be achieved in the current
framework, I assume. If someone wants to go ahead and
I'd like to hear from you.
I've done some research to see if someone else has
already built a
better wheel (i.e. test suite script).
The current most popular java testing solution, junit,
*not* suited for the job. Testing kaffe using a test
in Java has some 'hack value', but it would be quite
you're porting kaffe and you've barely got HelloWorld
would lead to the same problems that compiling the
tests using kjc
has: it's nice when it works, it's very hard to debug
when it doesn't.
That brings us to DejaGNU, the other well known (among
solution. It looks very nice, but using DejaGNU means
that one has to
get DejaGNU to work on a system first. DejaGNU in turn
needs tcl and
expect to be installed. In my opinion 'make check'
lightweight and perform its work with as little
as possible. that's why I don't think DejaGNU is
suited for the job.
The latest releases of autoconf come with a tool that
suited for the job, though: GNU Autotest. Yet another
barely documented in the autoconf manual. There is
documentation available in the CVS tree of
http://gnuprog2.sf.net . It
is apperently good enough for use as the test suite
script in GNU
autoconf, GNU m4 and GNU bison. According to the
documentation in the
Programming with GNU tools 2.ed book, It should
support a) - d) well.
So I'd like to hear your feedback on these ideas, as
if you are cross-compiling, hacking the build system,
regression tests or have some experience with
different test suite
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
More information about the kaffe