Eiffel

XML Parser performance test

The following figures and comments are all prone to change. They might be incorrect in certain cases or be taken under different circumstances. Not all optimizations might have been set correctly. And currently not every test did use exactly Gobo 3.2 as stated.

This page describes my attempts to determine, once and for all, what the speed difference is between the Expat parser and the native Eiffel parser in Gobo. I also wrote C code to test the Expat parser in its native environment. I have done the following tests:

  1. Parse the XML file in order to determine if it is well-formed. This tests the highest speed at which the parser can go through an XML file.
  2. Count the elements in the XML. This tests the overhead callbacks introduces.
  3. Save content in the XML file to some buffer. This more closely follows processing an XML file and doing something. It tries to determine how significant the overhead is that is introduced by the parser versus the work you want to do.
  4. Build a tree. This tests how significant the choice of the parser is, is you just want to build an in-memory structure of an XML document.

These four tests are done for two different situations:

  1. Parsing a single, but large 10MB document.
  2. Parsing many small documents.

Note that for a single small XML document it doesn't really matter what parser you use.

The documents are encoded in UTF-8 as we assume that to be the native and fastest implementation. This assumption is not currently True for the native Eiffel parser. For all Eiffel parsers the set_string_mode_unicode switch was set, so native Unicode strings were returned by the parser.

And all these tests are run with two different parsers: the Gobo Eiffel native XML parser and the Expat parser. The Expat parser itself is used in four different incarnations.

  1. The Expat parser in pure C code. This should be the base line to see how Eiffel performs relative to the most simple C code one can write against the fastest XML parser available. No attempt was made to model Eiffel's features or error checking in C. We just assume everything works.
  2. Raw calls using the Gobo Eiffel Expat API. This should give an indication how much overhead the Gobo Expat binding adds.
  3. The Gobo Eiffel Expat parser binding.
  4. The Gobo Eiffel native XML parser.

The Eiffel code is run without garbage collector if this could be done without causing excessive memory usage. For these tests this implied that all cases of the native Eiffel parser with small files have the garbage collector enabled.

Checking a single, large XML document for well-formedness

This test checks if a large, 10MB document is well-formed XML. The given time is the time to parse this document just once. The lower the time, the faster the file was be processed. All reported times are the minimum times from several runs.

Checking if a large XML document is well-formed
ParserTime (in seconds)Comment
Expat0.760Pure C.
Gobo Eiffel Expat API calls
Gobo Eiffel Expat parser1.355Parsing with all callbacks disabled.
Gobo Eiffel Expat parser4.888Parsing with default callbacks enabled.
Gobo Eiffel native parser7.529Compiled with SmartEiffel 1.1.
Gobo Eiffel native parser55.892Compiled with VisualEiffel 4.1.
Gobo Eiffel native parser16.219Compiled with ISE Eiffel 5.2.

The clear winner is the raw C code of course. It is almost 10 times as fast as the pure Eiffel code. The Eiffel Expat parser, with callbacks disabled, is 5.5 times as fast as the pure Eiffel code. The difference between the Eiffel Expat parser and the native parser is smaller, but the Expat parser is still 50% faster.

The difference between the same application compiled with VisualEiffel and SmartEiffel is shocking. SmartEiffel is more than 8 times faster for the same code. As far as the author is aware, all known optimization settings for VisualEiffel have been enabled.

To be fair it must be noted that SmartEiffel 1.1 has severe reliability issues. It compiles incorrect code in certain cases. VisualEiffel and ISE Eiffel have no publicly known compilation problems. Also the difference in compile times is shocking, but in the other direction. VisualEiffel takes about 1.5 minutes to compile the application, while SmartEiffel with the -O3 optimization takes more than one hour.

Checking many small XML documents for well-formedness

This test checks if 735 small (around 1KB) documents contain well-formed XML. The lower the time, the faster the file was be processed. All reported times are the minimum times from several runs.

Checking many files if they contain well-formed XML
ParserTime (in seconds)Comment
Expat0.089Pure C.
Gobo Eiffel Expat API calls
Gobo Eiffel Expat parser0.249Parsing with all callbacks disabled.
Gobo Eiffel Expat parser0.521Parsing with default callbacks enabled.
Gobo Eiffel native parser0.835

The C code is more than 9 times faster as the pure Eiffel parser. The Eiffel Expat parser, with callbacks disabled, is almost 3.5 times faster than the pure Eiffel code.

Counting tags in a large document

This test counts how often the <file> tag occurs in the large (10MB) XML document. Name space resolution is not used in this experiment. It needs callbacks, but we do very little per callback. By doing something we prevent a compiler that can analyze the entire system state of optimizing away the callbacks. The lower the time, the faster the file was be processed. All reported times are the minimum times from several runs.

Counting how often a certain tag occurs in a large XML document
ParserTime (in seconds)Comment
SAXCount3.106Utility provided with Xerces 2.2.
Expat0.929Pure C.
Gobo Eiffel Expat API calls
Gobo Eiffel Expat parser5.084
Gobo Eiffel native parser7.880Compiled with SmartEiffel 1.1.
Gobo Eiffel native parser56.353Compiled with VisualEiffel 4.1.
Gobo Eiffel native parser17.459Compiled with ISE Eiffel 5.2.

The C code is more than 5 times faster than the Eiffel Expat binding. That's a lot of overhead for a wrapper.

Counting tags in many small XML documents

This test counts the occurrences of a certain tag in 735 small (around 1KB) documents. The tag is in a certain name space, so name space resolving must be enabled. This test needs callbacks, but we do very little per callback. By doing something we prevent a compiler that can analyze the entire system state of optimizing away the callbacks. The lower the time, the faster the file was be processed. All reported times are the minimum times from several runs.

Counting how often a certain tag occurs in many files
ParserTime (in seconds)Comment
Expat0.115Pure C.
Gobo Eiffel Expat API calls
Gobo Eiffel Expat parser0.629
Gobo Eiffel native parser1.076Compiled with SmartEiffel 1.1.
Gobo Eiffel native parser5.854Compiled with VisualEiffel 4.1.
Gobo Eiffel native parser2.315Compiled with ISE Eiffel 5.2.

It's again amazing how fast the C version actually is. The written code isn't code that has production quality, but it is clear that there's a lot of overhead with the fancy callback features of Eiffel. Almost 5.5 times to be precise. And that seems to be a price that is too high.

The Eiffel Expat parser is about 1.7 times faster than the native Eiffel parser.

Saving attribute content in a large document

This test saves the contents of two attributes to a variable. The first attribute is the file name, the second is the size of the file. In the on-end-tag we increment a total size with the size of the file. The lower the time, the faster the file was be processed. All reported times are the minimum times from several runs.

More typical XML processing for a large XML document
ParserTime (in seconds)Comment
ExpatPure C.
Gobo Eiffel Expat API calls
Gobo Eiffel Expat parser5.727
Gobo Eiffel native parser8.473Compiled with SmartEiffel 1.1.
Gobo Eiffel native parser57.020Compiled with VisualEiffel 4.1.

It is amazing that a different Eiffel compiler and different parser can see a speed increase of 1000%! The difference between the Eiffel and the Expat version is about 1.5 when compiled with the same compiler.

Conclusion

Doing this test has highlighted the differences between Eiffel compilers. The fastest Eiffel compiler that gives you optimized code is VisualEiffel, by far. With an empty repository it took about one and a half minute to compile the native Eiffel parser. The next fastest compiler was ISE. This compiler took around ten minutes to compile the native Eiffel parser. SmartEiffel was the slowest. That wasn't due to its C generation, but to the the C compilation process. With the best optimization I could give, i.e. passing -O3 to the C compiler, it took more than one hour to compile a single test. With the -O2 optimization it took about 13 minutes to compile a test, comparable to the ISE Eiffel compiler. As the code is slightly slower, I've used the -O3 optimization.

The speed of compilation has a reverse relationship to the speed of the generated code. The VisualEiffel code is on average 3 times slower than the ISE Eiffel code. And that code is about 3 times slower than the SmartEiffel code.

And finally, the difference between the Eiffel and Expat parser is about 1.5 in this test, when compiled with SmartEiffel. Sometimes it's a bit more. In my opinion a speed difference of 50% is still enough to keep the Expat parser.

Even more worrying is the difference between the C and the Eiffel version. That difference is not easily explained by the desirable features Eiffel adds such as chained callbacks. But perhaps that is all there is to it: the C version fits in the L2 cache, while the Eiffel version doesn't? Or the branch prediction for the C code is better than the Eiffel version? It seems worthwhile to make an attempt to improve the native Eiffel parser by at least a factor 2.

Source code

The source code is available in tar.gz format. To compile the tests yourself do:

tar xvzf gobo-xml-test-0.2.tar.gz
cd gobo-xml-test-0.2
./configure
make

Making might take an extremely long time due to the -O3 optimization flag passed to the C compiler (a whole night on my machine).

Here some examples of passing flags configure to compile for a different compiler or with debug information enabled:.

./configure --with-compiler=ve
./configure --with-compiler=ise
./configure --with-compiler=se
./configure --with-debug=true

My software

  • Linux 2.4.21.
  • glibc 2.2.93-5 (PLD i686 version).
  • gcc 3.2.3 (PLD i686 version).
  • Expat version 1.95.1.
  • SmartEiffel 1.1.
  • Gobo 3.2.

My hardware

  • Dell 7500 Inspiron.
  • Mobile Pentium III.
  • 384MB of RAM.
  • 30GB hard-drive.