<document>
  <header>
    <product>resin</product>
    <resin-2.0>$article/jsp_benchmark.xtp</resin-2.0>
    <title>JSP Benchmark</title>
    <authors>
      <person>Scott Ferguson</person>
    </authors>
  </header>

  <body>
    <summary/>

<section title='Introduction'>

<p/>The benchmark measures the overhead of several popular servlet
and JSP engines.  It consists of a simple "Hello, World" servlet and
three JSP files: "Hello, world", session, and post.  In each case the
generated page is identical except for the background color.

<p/>The methodology section explains why this benchmark is a valid
measurement of servlet engines.  The benchmark source is available as
<a href='benchmark.jar'>benchmark.jar</a>

</section>

<section title='Data'>

The following JSP engines were tested:
<ul>
<li>Resin 1.0 b8 (own web server)
<li>Resin 1.0 b8/Apache 1.3.9
<li>JRun 2.3.3/Apache 1.3.9
<li>ServletExec 2.2b1/Apache 1.3.9
<li>JServ 1.0 final/gnujsp 0.9.8/Apache 1.3.9
<li>JWSDK 1.0 (August release) (own web server)
</ul>


<ct:img src='jsp_perf.gif'/>
<ct:img src='jsp_log.gif'/>

<deftable>
<tr><th>JSP Engine<th>File<th>Servlet<th>Hello<th>Session<th>Post
<tr align=right><td align=left>Resin<td>235<td>231<td>253<td>218<td>194
<tr align=right><td align=left>Resin/Apache<td>493<td>172<td>181<td>150<td>142
<tr align=right><td align=left>JRun/Apache<td>476<td>51<td>32<td>31<td>30
<tr align=right><td align=left>ServletExec/Apache<td>498<td>31<td>31<td>30<td>30
<tr align=right><td align=left>Jserv/Apache<td>498<td>41<td>30<td>30<td>29
<tr align=right><td align=left>JWSDK<td>42<td>41<td>12<td>11<td>12
</deftable>

</section>

<section title='Methodology'>

Skeptical of benchmarks?  Good.  However, this benchmark is indeed
valid for several reasons.  First, it primarily measures overhead, the
best possible performance for a JSP engine.  Second, except for the
overhead, JSP applications have identical performance.  Third, the
benchmark is not cacheable.

<p/>Overhead measurements are always valid and often the most useful
measurement.  It tells you the best you can possibly do, i.e. the
speed limit.  Because it doesn't exercise any functionality except the
overhead, it's impossible to defeat the measurement.  RPC mechanisms
like CORBA, NFS, or message passing kernel OS often quote the
'null-RPC' measurement.

<p/>The bulk of a servlet's work is identical no matter which JSP
engine is chosen.  The Java code runs at the same speed and the
database drivers are identical.  So creating a "more realistic"
benchmark would only add a constant time to each request.

<p/>The second point does become important when comparing JSP/Java to
JSP/JavaScript or Perl or PHP or ASP.  This benchmark would be less
effective in comparing technologies with different languages and
drivers.  It still would give a sureful overhead result, but would not
give a good idea of how real applications compare.  In addition,
because the JDK is the same, the JSP engines can't do the equivalent
of optimizing out loops.

<p/>Third, the benchmark isn't cacheable without heroic measures.
When comparing computers or static web servers, the performance when
the results come from a cache are better and unrepresentitive of real
results.  That's one reason the sieve and n-queens benchmark are bad,
and why SpecWeb goes to great lengths to defeat web server caching.
(Of course, in the last case, many sites can fit the most popular
pages into a memory cache, so the memory results may actually be more
indicative of the real performance.)

<p/>Finally, tweaking configurations typically only gets at most 20%
improvement.  Resin is three times faster.  No amount of tweaking will
change that significantly.  For example, JDK 1.1.7 was used because
it's the most stable on the Linux platform and because all the engines
ran successfully with it.  Internally, I did spot checks with JDK 1.2
and the numbers were similar.  The full results for JDK 1.2 will be
available shortly; time pressures limited their inclusion.

</section>

<section title='Conclusion'>

This benchmark shows that JSP can approach static page performance.  The
comparison between different JSP engines is really less important,
although Resin's performance is certainly personally gratifying.   A
perception still exists that Java is a great programming environment
but hobbled by performance.  The perception is no longer valid.

<p/>The near-static performance means that many sites, who never
considered JSP or servlets because of performance, can now take
advantage of the Java platform's reliability and ease of programming
benefits.

</section>  </body>
</document>
