Last Updated: 9 Nov 1999 |
Another more distressing problem with image loading under Java is that of memory usage. It seems that to load and display an image, multiple copies of that image will be made and stored in memory. This becomes a serious problem when an application uses large images, as is the case with ours. An example of this memory bloat follows.
Under JDK1.2, loading and displaying an image 4096x4096 using 16bpp should consume 32megs. However in reality, approximately 158megs are consumed. This becomes an obvious problem for machines with little memory.
Therefore we will develop our image decoding library in-house so we can quickly fix bugs and add new image formats. It was decided to implement the library using JNI and native methods, rather than implement the image decoders in 100% Java. Reasons for this follow:
It appears that the Image Producer/Consumer design used by Java inherently makes multiple copies of image data. However, with JDK1.2, a BufferedImage is introduced which does not have to use the producer/consumer design to get and set image data. This means that it is possible to produce image loaders which will not make extra copies of image data, thereby keeping memory usage down. Our image loaders will make use of the BufferedImage provided with JDK1.2, but should provide facility to work with JDK1.1 albeit with increased memory consumption.
Our image loader must therefore be designed to accept the image data (e.g.
contents of the file picture.jpg) as a stream, and not as a file. This
means that the image loaders will have to work with sequential data, not
random access data. This is illustrated in Figure 2.
The ImageDecoder is a class that takes an InputStream containing data from
an image file, and returns an Image (a BufferedImage if using Java2D (which
is provided with JDK1.2)). Figure 3 shows the
internals of this class. From this figure we can see that the decoding of
image data into pixel data is done natively. The ImageDecoder passes the
image data to the native code, which returns pixel data row at a time. The
ImageDecoder converts this pixel data into an Image object and returns it.
The basic algorithm that the ImageDecoder follows is thus:
The first thing to note from this is that the image type is
specifically given to the native code. This is not inferred from the image
data input stream. This simplifies the native code, in that it doesn't have
to read and buffer header information in determining the image type. It
does, however, introduce a problem if the incorrect image type is passed.
For example, a GIF file may have an extension of .bmp. This means that the
image data will not be recognised as a GIF image and will return an error to
that effect. This is desired behaviour as the file extension is a well
defined method of identifying the file contents and should match correctly.
In step 2 of the algorithm, we send all the image data to the native code.
In step 3 we read in the header to determine the image properties. If the
header is at the beginning of the image data, it is not necessary to
transfer the entire image contents before reading the header. These steps
can operate somewhat in parallel. The native code can read in and process
the header, while the rest of the image data is still arriving. In our
implementation this is achieved by running step 2 inside a separate thread.
So Figure 3 can be implemented as shown in
Figure 4. Here step 2 runs in a separate thread in
parallel with steps 3, 4 and 5. There is a case where step 2 is not run in
a separate thread and the algorithm is run sequentially in a single thread.
This is discussed in the next section.
The image that is created from the pixel data is of a DirectColorModel,
using 32 bits per pixel (as in the default Java ColorModel). This uses 8
bits for each of the alpha, red, green and blue colour components. In the
future, the image will be created using 16 bits per pixel (1bit for alpha,
and 5 bits for each of red, green and blue). Due to a Java bug, this can
not be achieved at the moment, but when this is implemented, the memory
usage for each image will halve.
The first (and simplest) method, is to create a temporary file and write all
the image data received from the Java side to this file. When all data has
been received, we then open this file and pass this newly opened file
descriptor to the decoding routines. This requires a sequential approach to
the algorithm described on the Java side. The
reason for this, is that when reading from a file which has another
process/thread writing to it, there is no way to know when all writing has
been finished. So to ensure safety, all writing must be finished and the
file closed, before reading from this temporary file can commence.
There are disadvantages with this method. Firstly, the sequential approach
means that we can't interleave steps of the algorithm which will increase the
time taken to decode an image. But a more pronounced increase in time is
incurred by the use of a temporary file. Having to first write the data to
disk, and then read it back incurs memory to disk, and disk to memory
traffic. This introduces a significant slowdown to the decoding process.
The second method is slightly more complex. It involves setting up a pipe.
The Java side will send data to the writing end of the pipe, and the file
descriptor of the reading end is passed to the image decoding routines. For
this to be achieved, the Java side must send the data in a
separate thread. The reason for this is that when the pipe fills, the
writing thread will block, waiting for the data to be read from the pipe.
If the algorithm proceeds sequentially, the thread will block when the pipe
fills, and will never unblock as there is nothing to empty to pipe.
This method is more efficient in terms of speed, as we skip the overhead of
writing all the data to disk, and then reading it back again. The
programmer must be aware that the file descriptor passed to the image
decoding routines is only capable of sequential access, not random access.
This is essentially not a problem as most image formats can be decoded
sequentially (see the discussion of Tiff for
an exception).
However there is an implementation problem with this method and green
threads under Unix. If the reading end of the pipe tries to read and the
pipe is empty, it will block until data is available (or the writing end is
closed). This is correct behaviour. However with green threads, if one
thread blocks on a call to read(2), then the entire process will block.
This introduces a race condition. Everything will work correctly as long as
the reader does not try to read from an empty pipe. There is unfortunately
no way to guarantee this. This means that under these conditions we must
use the first (and slower) method.
The metrics are for loading in a particular image of dimensions 4096x4096
and of 32 bits colour depth. The memory that such an image should consume
is 64 megabytes. The memory usage figures are approximate as they were
taken using the MEM Usage facility of the NT Task Manager.
Figure 1. Typical Content Handler
Figure 2. Content Handler invoking ImageDecoder
Architecture
Java side.
The image loader is based around the ImageDecoder Java class (shown in
Figure 2). The content handler is merely the
mechanism that invokes the ImageDecoder. It has no bearing on the way the
image handling works, so it will no longer be discussed.
Figure 3. ImageDecoder communicating with Native Code.
Figure 4. Threaded ImageDecoder
Native Code.
A commonality with the image libraries that we have, is that they all accept
their image data from an open file descriptor. They do not require a
filename and perform the opening themselves. This has led us to develop two
methods of passing the image data to the decoding routines.
Issues with Tiff
Tiff (Tagged Image Format) is an image format that cannot decode an image by
reading sequentially from the data stream. This format stores a directory
in the image file which contains information about the images contained
within the file. When decoding, we need to seek to the directory, read and
process the directory, then seek to where the offset where the image is
stored in the file. If we are using the second method as described above,
then the decoding will not work because we cannot seek. Our solution to
this was to firstly read all of the image data into memory buffer, and then
write our own read, and seek routines to access the data from this buffer,
rather than using the file descriptor.
Performance
Table 1 shows timings and memory usage comparisons
loading and displaying various image formats using jdk1.1.5, and jdk1.2rc1.
These measurements were taken on a PC running windows NT. Measurements were
all taken under approximately the same cpu and network load. The timings
are not to be taken as absolute, they are given to show the relative
difference between the different methods. Measurements given are the
average of four runs, under near identical conditions.
Time to decode and display (seconds) |
Peak memory usage (megabytes) |
Final memory usage (megabytes) |
|
---|---|---|---|
Image loading under JDK1.2rc1 | |||
MS Bitmap | 2.8 | 133 | 67 |
Jpeg | 5.4 | 67 | 67 |
Portable Pixmap/Graymap | 4.5* | 67 | 67 |
Targa | 3.8 | 133 | 67 |
Jpeg (using Java) | 10.8 | 158 | 158 |
GIF (using Java) | 2.8 | 48 | 48 |
Image loading under JDK1.1.5 | |||
MS Bitmap | 5.0 | 139 | 123 |
Jpeg | 5.5 | 123 | 123 |
Portable Pixmap/Graymap | 4.5* | 123 | 123 |
Targa | 3.8 | 155 | 125 |
Jpeg (using Java) | 8.2 | 123 | 123 |
GIF (using Java) | 3.0 | 33 | 33 |
* Due to the fact that Portable Pixmap/Graymap files are very large, most of the decoding time is spent reading the image data from disk. For this example the time taken was approximately 20 seconds. The figure displayed in the table was arrived at by buffering the image data in memory to remove disk access time in the timings.
The table does not contain results for Tiff images as the decoder for Tiff is still being worked on at the time of writing of this document.
From the table we can see that our image loader is approximately twice as fast as the Jpeg loader provided with Java. This in itself would make it worthwhile to use this method.
The real advantage can be seen with the memory usage. Under JDK1.2, the Java implementation of the Jpeg loader consumes 158Megs, while our method consumes 67Megs. Recall that 64Megs is required to hold the image data. The memory usage comparison here is obvious.
Some of the decoding methods have a peak memory usage higher than their final. This is due to the fact that some image formats store data bottom-up, instead of top down. We need to buffer the entire image in these cases, which accounts for the peak being approximately double the amount of memory to store the image data.
Under JDK1.1, we cannot use our BufferedImage, and must rely on Java's image producer/consumer architecture. This accounts for the increase in memory usage. Even so, our custom method is still faster than Java's internal Jpeg loader. The memory usage is comparable with that of Java, so there is no real disadvantage of our method.
Image Format | Mime Type |
---|---|
Jpeg | image/jpeg |
GIF | image/gif |
Targa | image/targa* |
Portable Pixmap | image/x-portable-pixmap |
Portable Graymap | image/x-portable-graymap |
Tiff | image/tiff* |
PNG | image/png* |
MS & OS/2 Bitmap | image/bmp* |
*Although the MIME types for these image formats have not been specified by the IANA, the MIME types are not preceeded by the usual "x-" which is recommended to preceed user defined MIME types. This is because it is expected that the IANA will adopt these MIME types in the near future. Indeed, web servers (e.g. apache) are providing the MIME types for these image files as we have shown in the above table.