Benchmarks

"High-Performance" is meaningless without Comparative Benchmarks

When, for the last time, have you seen a Web server's author publishing open-sourced comparative tests with 27 competitors?

1. Apache (IBM)
2. Apache Traffic Server (Yahoo!)
3. IIS (Microsoft)
4. GlassFish (Oracle)
5. Tuxedo (Oracle)
6. TntNet (Deutsche Boerse)
7. Rock (Accoria)
8. Tomcat (Apache Foundation)
9. JBoss (Red Hat)
10. Jetty
11. Caucho Resin
12. PlayFramework
13. Lighttpd
14. Nginx
15. Varnish (the Web 'accelerator')
16. OpenTraker
17. Mongoose
18. Cherokee
19. Monkey
20. Libevent
21. Libev
22. ULib
23. Poco
24. ACE
25. Boost
26. Snorkel
27. AppWeb

Let's have a closer look at Nginx and Lighttpd, the raising stars: (Varnish, "the Web accelerator", is included in this SMP weighttp test because an AB test made by Nicolas Bonvin was criticized for not using Varnish's "heavily threaded" architecture)

G-WAN served 3 to 7x more requests in 3 to 6x less time than the other servers:

 Server   Avg RPS      RAM        CPU            Time for test
 ------   -------   ---------   ---------   -------------------------
 Nginx    167,977    11.93 MB   2,910,713   01:53:43 =  6,823 seconds
 Lighty   218,974    20.12 MB   2,772,122   01:09:08 =  4,748 seconds
 Varnish  103,996   223.86 MB   4,242,802   03:00:17 = 10,817 seconds
 G-WAN   729,307    5.03 MB    911,338  00:28:40 =  1,720 seconds

And despite being the fastest, G-WAN used 1 to 4x less CPU as well as 2 to 45x less RAM resources (note for Varnish: that's the allocated memory amounts measured here, not the reserved virtual memory address space which does not deprive the system from any hardware resource).

When merely used as a static-content Web server, G-WAN (an application server) is 9 to 42x faster and uses much less CPU/RAM resources than all other servers ("Web server accelerators" like Varnish or mere Web servers like Nginx).

Guess how much wider the gap will be with 64-Core CPUs.

Dynamic Contents: Scripts vs. Compiled code

But G-WAN was written to have an application server, so it also supports 7 of the most popular scripted languages: Java, C#, C/C++, D and Objective-C/C++.

The success of scripts comes from the instant gratification that interpreters provide: at the press of a key your code executes instantly. Compare this with the compilation and linkage cycles of Apache, Lighttpd or Nginx modules (which require server stops & restarts).

But since low performance comes at a cost, interpreters left the scene to invite compilers (Facebook HipHop PHP => C++ translator).

G-WAN C scripts offer the best of both worlds: the convenience of scripts and the speed of compiled code.

Comparing G-WAN C scripts / Apache + PHP / GlassFish + Java / IIS + C#

To evaluate the performance gap of poorly executed convenience, we invited the major scripted language vendors to review our version of the loan.c script ported in other language (see below) – and then we used an ApacheBench wrapper to test them all:

Seeing those results, some said that "G-WAN is faster only because it runs ANSI C scripts". This is obviously false: since G-WAN (serving static or dynamic contents) is faster than Nginx (serving static contents). G-WAN is just a better server.

When used as an application server G-WAN is 32 to 430x faster and uses much less CPU/RAM resources than all other servers (on the top of not dying as concurrency grows).

Anyway, Java is not ANSI C so to avoid being accused of "comparing apples to oranges" let's see how much faster G-WAN + Java can run as compared to Apache Tomcat – a server dedicated to Java.

Comparing G-WAN + Java to Apache Tomcat (a Java App. server)

Using the simplest possible Java script hello.java (instead of the loan-100 used above) to let Tomcat survive the experiment, G-WAN (loaded with a JVM weighting 20 MB RAM at startup) serves 10x more requests in 3.7x less time than Apache Tomcat while using 4.3x less RAM and 2.8x less CPU resources:

        Avg RPS    Elapsed Time for the Test      RAM     user / kernel / total (CPU jiffies)
       ---------  ---------------------------  ---------  -----------------------------------
G-WAN   759,726  00:27:28 = 1,648 seconds   46.90 MB      186,709    592,207   778,916
Tomcat    73,903   01:41:42 = 6,102 seconds    203.80 MB    1,582,074    630,299  2,212,373

This test "compares apples to apples": G-WAN + hello.java and Tomcat + hello.java. To do the same job, G-WAN + Java is 37x faster than Tomcat. And this is for a skinny "hello world", when Tomcat stops to respond under real-life jobs like the "loan" test above, then the costs are exponential: 750,000 RPS / 0 RPS = ∞

Conclusion: If you have to use Java then G-WAN's vertical scalability can help you to save a lot on horizontal scalability which requires more hardware (servers, cooling systems, network equipment like switches and load-balancers), floor space, staff and electricity – a resource which cost is expected to double in the next decade.

Calling G-WAN C scripts to the rescue

We have seen above that G-WAN makes Java fly much higher than Apache Tomcat. But if some critical code must perform and scale better then G-WAN C scripts will give you the boost you need. G-WAN + hello.c is now serving more requests in less time while consuming 5.5x less RAM and 3.5x less CPU resources than G-WAN + hello.java:

                   Avg RPS    Total Elapsed Time        RAM    user / kernel / total (jiffies)
                  --------- ----------------------- ---------- -------------------------------
 G-WAN hello.c     801,58500:25:51 = 1,551 seconds 8.60 MB   182,280    563,701   745,981
 G-WAN hello.java   759,726 00:27:28 = 1,648 seconds  46.90 MB   186,709    592,207    778,916
 Tomcat hello.java   73,903 01:41:42 = 6,102 seconds 203.80 MB 1,582,074    630,299  2,212,373
 Nginx 100.html     167,977 01:53:43 = 6,823 seconds  11.93 MB 1,000,270  1,910,443  2,910,713
 G-WAN 100.html     826,821 00:25:16 = 1,156 seconds   9.21 MB   155,884    563,147    719,031

You read well. G-WAN + hello.c process 11x more requests in 4x less time while they are consuming 24x less RAM and 3x less CPU resources than Apache Tomcat.

Right, unlike the test above we are now "comparing apples to oranges" (C vs Java) but G-WAN C scripts can be useful as they do the same job 44x faster than Apache Tomcat... for a mere "hello world" test.

The "hello world" program does very little, but the little it does illustrates how much more efficient ANSI C is as compared to Java. And the gap is much higher with more substantial tests than a simple "hello world". That difference is paid by end-users.

Whether you are running one single Web server or a Big Data server farm then G-WAN is your best option to make significant savings and to offer the most extraordinary performance without eye-waterering investments – and their recurring costs.

G-WAN faster (as an App. server) than Nginx (serving static files)

In the light of the tests above, it is now obvious that not all script engines are equal – even without concurrency. But the same goes for server technologies:

To sum it up: using the least efficient technologies comes at an hefty (recurring) cost.

Comparing G-WAN to Nginx, Lighttpd, Varnish and Apache Traffic Server (ATS)

Using a low-end Intel Core i3 laptop and a 100-byte static file, an independent expert [1] of the EPFL's Distributed Information Systems Laboratory has evaluated the performance of the best Web servers (and "Web server accelerators" like Varnish and ATS) by using this open-source ApacheBench (AB) wrapper:

Varnish provided several versions of its configuration file and, like Igor for Nginx, requested several benchmarks to be done.

Their efforts did not make any difference: G-WAN (without tuning and with more features than all others) is just much faster.

This test is less impressive than the SMP tests above because ApacheBench (AB) is a single-threaded client (which cannot saturate SMP servers like G-WAN).

Note that G-WAN v2.1, the version tested by Nicolas Bonvin, was 4x faster (serving more requests in less time) and used twice less CPU resources than Nginx – but it used more memory. This pushed us to at least equal Nginx's feat: G-WAN v2.8+ uses less memory than Nginx (while offering many more features like a 'wait-free' KV store – the first of its kind – as well as Java, C#, C/C++, D and Objective-C scripts to generate dynamic contents).

Apache / Cherokee / G-WAN / Lighttpd / Litespeed / Nginx / Varnish on 8-Core

More recently, using a high-end dual Intel 4-Core server, an indepedent sysadmin has compared the performance of the best Web servers (and cache like Varnish) CPU Core by CPU Core: