[kaffe] Bug report (java.io.StreamTokenizer)

[Ito Kazumitsu]

Hi,

In message "Re: [kaffe] Bug report (java.io.StreamTokenizer)"
    on 03/06/30, Hermanni Hyytiälä <hemppah at cc.jyu.fi> writes:

> According to the JLS (first edition), the nextToken-method of
> java.io.StreamTokenizer class has the following lexical order:
> 
> whitespace
> numeric character
> alphabetic character
> comment character
> string quote character
> comment //
> comment /*

I see.  But this rule seems to have been ignored even by
Sun's implementation.

(1) Kaffe's java.io.StreamTokenizer.java has this comment:

        /* Contrary to the description in JLS 1.ed,
           C & C++ comments seem to be checked
           before other comments. That actually
           make sense, since the default comment
           character is '/'.
        */

(2) Comment characters must be checked before alphabetic characters
    and Sun's java.io.StreamTokenizer seems to do so.  Otherwise,
    NBIO you mentioned cannot run properly.

> Yet, however, I haven't tested if this parsing order changes the
> behaviour of Kaffe's java.io.StringTokenizer's behaviour (could someone
> test it ?-)).

I wrote a simple program to test with Sun's JDK and kaffe with and
without my patch.

import java.io.*;
public class StreamTokenizerTest {
  public static void main(String[] args) throws Exception {
    StreamTokenizer tok = new StreamTokenizer(System.in);
    if (args[0].equals("NBIO")) {
       tok = new StreamTokenizer(System.in);
       tok.resetSyntax();
       tok.wordChars((char)0, (char)255);
       tok.whitespaceChars('\u0000', '\u0020');
       tok.commentChar('#');
       tok.eolIsSignificant(true);
    }
    System.out.println("TT_WORD = " + StreamTokenizer.TT_WORD);
    System.out.println("TT_NUMBER = " + StreamTokenizer.TT_NUMBER);
    System.out.println("TT_EOF = " + StreamTokenizer.TT_EOL);
    System.out.println("TT_EOF = " + StreamTokenizer.TT_EOF);
    while (true) {
      int t = tok.nextToken();
      System.out.println(tok.sval + ": " + t);
      if (t == StreamTokenizer.TT_EOF) break;
    }
  }
}


Sun's JDK:
bash$ echo "# comment" |java StreamTokenizerTest NBIO
TT_WORD = -3
TT_NUMBER = -2
TT_EOF = 10
TT_EOF = -1
null: 10
null: -1

Kaffe without my patch:
$ echo "# comment" |java StreamTokenizerTest NBIO
TT_WORD = -3
TT_NUMBER = -2
TT_EOF = 10
TT_EOF = -1
#: -3
comment: -3
comment: 10
null: -1

Kaffe with my patch gives the same result as Sun's.
$ echo "# comment" |java StreamTokenizerTest NBIO
TT_WORD = -3
TT_NUMBER = -2
TT_EOF = 10
TT_EOF = -1
null: 10
null: -1