Weeks ago i performed some benchmarks to test Monkey project againts other webservers, i have included the test for one closed source web server which claims to be (or not to be?) the fastest on earth: GWan. My benchmark shows that Monkey is faster than GWan under certain conditions: number of request per second for a file >= 200KB.

One person around our community started to perform his own bencharks and shared his findings with Pierre the author of GWan. Looks like Pierre’s feelings were hurt and his ironic words expressed a strongly disagree because there is a project that performs better than G-Wan. He started to put in doubt my reputation and abused of the usage of my employer’s name:

What makes an expert “cheat”? http://gwan.ch/blog/20120728.html

Oracle and I

Pierre, let’s make the things clear here. Oracle is my employer and it have not any relationship with the Monkey open source project. It looks like a lack of professionalism to abuse of Oracle name to increase visibility and try to pseudo-expose your startup in a different manner, why to mention the ‘Oracle‘ word 12 times without need it ?, there are other ways to positionate your blog with the search engines.

 

G-Wan benchmaks

The G-Wan author decided the best way to bencharmk his software, so he went straigforward to benchmark under two specific static conditions:

  • KeepAlive : all test are made under keep alive
  • Small file sizes: the tests are performed with a file of 100 bytes

If the benchmark method is static as the used here, its useful when measuring current version againts development version, and see how it behave each other, and this is good. But you cannot use the same method to measure other web servers, let’s perform a brief analysis:

KeepAlive

The KeepAlive HTTP feature (default in HTTP/1.1), aims to keep the communication channel open between the client and the server to perform multiple request in a FIFO way, this approach reduces latency due to the avoidance of the TCP handshake and less network traffic. In a HTTP KeepAlive session for example, you can perform 1000000 request over the same persistent channel if the server allows that.

As stated before, using HTTP KeepAlive the transfer of user-level data is faster. But there is an important point that we cannot omit to mention when we are under a *benchmark* context: if the web server use threads to balance the work due to SMP or other need, using KeepAlive sessions we will *hide* the overhead of the connection balancing between workers and for hence the internal scheduling and time to start processing each request. So use KeepAlive for benchmarking is useful to measure just a part of the web
server core and *not* a fundament to determinate which is the fastest solution in the world.

From a real-life perspective, an HTTP Client (browser) using KeepAlive, rarely will perform more than 25 request over the same channel. So the tests under KeepAlive does not reflect how the Internet/HTTP world behave.

Small file sizes

G-Wan perform caching over requested files in memory when they are pretty small, so when testing a 100 bytes file, this is not hitting I/O and is not requiring to perform extra expensive system call. This is the common approach over almost all web servers, a few KB of extra memory is not bad to avoid I/O.

Caching is good, but testing againts the same small cached file, it’s just trying to determinate how fast is accessing memory buffer and sending out the same data.

Now having describe how the static benchmark is done, sounds like a good plan to determinate that G-Wan is the fastest solution available testing *only* under KeepAlive mode plus the same small cached file for all requests ?, not really.

Every person who wants to benchmark G-WAN and break one of those rules (KeepAlive/small file), will realize than Monkey and NginX performs faster than G-Wan.

G-Wan author claims that testing in non-keepalive mode is equal to “test the TCP IP stack rather than the user-mode code of the server”, which is totally wrong, a web server depends on TCP/IP stack and meassuring the performance of a web server is more than test a simple access to a memory buffer in user space.

Measuring tool

The G-Wan benchmarks are done using a wrapper utility called ‘ab wrapper’, described by Pierre as the “most capable tool” (http://gwan.ch/source/ab.c), the sad part is that it cannot perform well with files of a few KB, it get stuck when used with Weighttp backend or when is used without keepalive mode. It’s a good idea of tool as it takes snapshots of user-mode/kernel-mode stats once a concurrent round of request is finished.

As the code of ab.c it’s not legible and is spawning third party utilities to perform it’s job, i wrote a similar tool but based in proc filesystem, the tool name is ‘watch resources‘ (aka wr) and i have published the code on GitHUB:

http://github.com/edsiper/wr

So with this new tool i have performed a new set of tests of benchmark using concurrency as ab.c does and having more accurate memory usage for the results.

Benchmark: Monkey v/s G-Wan

This test have been made in the following way:

  • Wrapper by Watch Resources tool
  • Weighttp backend to stress the server
  • The URL tested hits a file of 200KB
  • Concurrency tests from 100 to 1000 (100 concurrents to 1000), increasing the level with 10 concurrents.
  • KeepAlive enable
  • Backend stress tool with 10 workers
  • Each round did 500.000 requests

Requests per second

  • Monkey did an average of 29231 requests per second
  • G-Wan did an average of 17222 request per second

Monkey did 42% more requests than G-Wan under the same conditions

Memory Usage

Its expected to see high level of memory consumption as the concurrency hits a major load in the server so the goal is to optimize the resources and reduce memory allocations when they are not necessary:

  • Monkey consumed an average of 3.4MB along the whole test
  • G-Wan consumed an average of 7.09MB along the whole test

G-Wan consumed 52% more memory than Monkey for the same test

User and system time

In the Linux Kernel (OS) design which comes from Unix family, exist two executions context or virtual address spaces: User space and Kernel Space (or System space). The User space belongs to the container of user applications and related resources plus an interface to communicate to the Kernel, as well the Kernel Space and its roles covers I/O, Memory allocations, Scheduling of user space tasks and others.

All job that occurred in a user process/task and which do not require a direct Kernel intervention is called user space, everything else is Kernel Space. We call User-time to the CPU cycles consumed by the user user task, and we call Kernel/System time to the CPU cycles consumed by the user space task through a system call to the Kernel.

So this metric shows how much CPU time is spent in user and system space, and depending of the point of view it could be good or not so good:

  • Monkey user time: 5267 milliseconds
  • G-Wan user time: 3830 milliseconds
  • Monkey Kernel/System time 36159 milliseconds
  • G-Wan Kernel/System time 50999 milliseconds

G-Wan have focused too much into reduce user-time being a non-friendly program running under the Linux Kernel. Monkey is a project devoted to run over a Linux Kernel and that’s the reason about why it runs pretty optimized. Having a basic knowledge of Linux system calls is not hard to achieve a great performance.

In the blog post mentioned earlier in the graphics comments it states:

As opposed to Monkey, with G-Wan, the Kernel is using CPU resources faster than G-Wan user mode CPU usage. As a result, the Linux Kernel is the bottleneck far before G-Wan

The previous comment, denotes a lack of understanding about how the Linux Kernel works internally, there is no knowledge of user/system spaces or scheduling. Referring to the G-Wan project history it comes from Windows so we can excuse the lack of knowledge on Linux. Rarely the Linux Kernel at this operation level is a bottleneck.

For short, in order to avoid misunderstandings i already have learned my lesson and i will provide more detailed benchmarks from now. In the meanwhile G-Wan have a couple of things to fix.. starting from it design ?.

Other points that i forgot to mention:

  • The test in Monkey took 25 minutes and 58 seconds. G-Wan took 44 minutes and 23 seconds
  • If you want to validate the information on this post, you can download the source reports and graphics from here.
  • I am emaling Pierre about this blog post

Last week we did the review of the mid-term evaluation of Google Summer of Code, on this second round of the program 2/3 of our students continue hacking Monkey, by now i can mention that we have fully implemented the following by students:

  • Monkey Library: Monkey now can be converted in a shared library, that means that you can use Monkey stack using the lib API
  • Monkey CGI Plugin: when we moved to a event-driven model in Monkey 0.10.0 (two years ago), we decided to remove the CGI support as it lack of performance, due to people requirements and one successful student who already finished his initial project, he decided to move forward with extra tasks, so the CGI support is officially back for the next release v1.1.
  • Redis Support: our web services framework called Duda, implements packages to add extra features or make the use of third party components in an easy manner, Redis package is ready for Duda and will be merged into mainline within the next week.

Ongoing tasks by the students are: complete the merge of library support in Monkey mainline, FastCGI support, Memcache support for Duda… exciting times 🙂

 

WE did it

After years of development we have reached Monkey v1.0, codename Monkey I/O.  Monkey have been a pretty stable web server for years, each release cycle have shown a mature in terms of fixes and improvements, as well the community have grown and we have more eyes over the code, as well different visions to achieve the goals. So far i am very pleased  about this new major release and i feel comfortable how it arrives in a very good year for the project.

This have been an exciting development cycle where many performance improvements have been made, as well code cleanup and general fixes. We have reach this new release keeping our focus of hit a great performance and be lightweight.

Being a well funded project and persistent community, Monkey project joined this year to the Google Summer of Code program as a mentoring organization. Currently we are guiding three students who are working hard in a few exciting implementations for our incoming versions.

Looking back…

Years ago people often complained about why to dedicate too much time in a new web server, when some open and reliable options were available such as Apache or NginX, we always looked for a real lightweight solution, and that is Monkey. Also when we started to do a major re-write of the core on 2008, we changed our mind about where the project is going so our new focus was always embedded: where the penguin (Linux) goes.. Monkey goes behind it. So it’s not a surprise that we are really focus on ARM based platforms.

What can Monkey offer to an embedded device ?

Nowadays HTTP is one of the widely used communication protocol by different application layers (obviously running on top of TCP/IP). We are not in the age of just serve static content, even generate a full dynamic content page through scripting it’s looking like old school, or it will be in less than five years. We are in the service-oriented models age, where unique resources perform a specific job and expose that functionality through a common communication application layer: HTTP.

Imagine different nodes around your local network sharing signal information from sensors or inclusive waiting for incoming commands to execute a specific task in the device, you have a few workarounds depending of the transport layer, but if you choose HTTP you need a flexible and lightweight solution.

Monkey is more than a common web server, is a complete event-driven HTTP stack with a flexible API which allow you to take full control over the incoming request and at any later stage of the cycle.

Making web services easy: Duda

Even for who is implementing web services in C have to deal with not-so-friendly concepts such as concurrency, thread keys, lock free, memory management, asynchronous events, etc. This generate and extra learning curve that can bring some delays in the project implementation where straightforward solutions are critical.

Due to the problem recently described, a  new solution and sub-project have born: Duda, a web services framework. Duda it’s build on top of Monkey and expose a friendly C API for the web services implementation. Weeks ago we have publish an blog post about it:

http://blog.monkey-project.com/2012/03/13/duda-web-services-framework-for-monkey/

As described in the article, Duda is hitting a great performance in terms of speed and resource usage:

Due to the nature of Duda and it’s new dependencies, we have decided to separate the project from the Monkey sources and handle it in a different scenario, you can find the new Duda home here: http://duda.io . The project is under active development and we will post more about it in the incoming weeks.

Final words

Thanks to everyone who have dedicated some time to take part of this, all work have been done in community with motivated people behind it, we will keep pushing hard in innovation and improvements. Just keep being part of it 🙂

If you are interested into participate in one of our projects or get involved in the community, don’t hesitate to contact us on irc.freenode.net #monkey or through our mailing list.

I need some assistance to purchase the right replacement for my car, if you can give me a hand it will be more than appreciated:

My car:

  • Kia Sportage GE ’97 (made in America)
  • Failure: flywheel

This is my flywheel:

the Flywheel have two sides, one with teeths and the other with a small squares. My teeth side is almost broken so i need to change the whole part, Googling a bit i found some vendors from the US that can ship the replacement but no one is able to confirm if the part is about both sides or just one of them. Its a very old car so they do not have full specs about each part.

Would you please point me to the right vendor  to purchase the whole part ?, any help is appreciated.

This year have been very exciting for Monkey Project, we are implementing cool features for our next v1.0 release and today we got the most great news that we could expect:

 

Your Organization Application for “Monkey HTTP Daemon” in
Google Summer of Code 2012 has been accepted.

Today begins a new stage for our project, after thousands of work hours and many years putting our best effort, we are finally being recognized internationally as a strong community with serious objectives and delivering a high quality product. This is a great opportunity to grow in different aspects of the project: organization, community and software improvement.

We invite to any student around the world to apply for our organization, we are committed to improve the server side of the web, we deal with embedded devices, performance, scalability, networking, and many cool stuff , innovation is one of our primary focus. Please review our project ideas site:

If you think that the web can still be improved, we are in the same page and we are looking to hear from you, meet us on irc.freenode.net #monkey or through our mailing list

[OBS: this is a repost from http://blog.monkey-project.com/2012/03/13/duda-web-services-framework-for-monkey/]

There is one fact: we are in exciting times for the project. Our HTTP core is consolidated since a while ago, it works in non-blocking mode, it has a nice indented configuration model and provide an advanced API to extend the core behavior, through this we support different behaviors by layers like IPv4, IPv6, SSL encription, security by subnets, basic auth, shell, log writer, etc.

There is a common question from people around the project: “where are you going ?” and the answer is: do you see that embedded Linux Box ?, there’s where we go. Common web servers lack of performance for embedded systems, most of them have focused in high production environments. our focus is different.

Our tech world is service oriented and i would say that implemented through a fat HTTP software stack, simple things are done in scripting languages that requires a process context or even be ran in a JVM environment, that is not lightweight and requires extra resources to be scalable. If you plan to implement a web service for embedded you should start forgetting about Java and PHP and think in lightweight options. Please Google a little about the options available and then continue reading here.

Continue reading

I have seen a few videos about the paradigms of the education model and how we lives… i cannot stop thinking about that things are really wrong about how we are educated, how we manage our life and how we will be fine working in the others dreams, but what about ours ?. Often when people grow smile less, get depressed and not all of them are able to see the light and get some happiness… this is not ok and should not be something common…

When you are a child your imagination does not have limits, when you grow you are instructed to limit the scope of your imagination to what you just have to learn, i do not say that teaching is bad, i say that the knowledge must be share but also people should be inspired to be connected  with the “creativity”, I consider the creativity a state of collective conscience where everything already exists, when you “create something” you are just being able to pick up a piece of that conscience, commonly named an “idea”.

Also if we see how we commonly work i would describe two scenarios: people working in a company and people working by their own. When you work for a company you are mostly working in the other person dreams, at least you share the same vision and feel part of what is being done. For people doing their own stuff  or working as independent, they are trying to reach their dreams. There is nothing wrong with the mentioned scenarios, except when you do not share the same vision or you are not able to “create” due to “limits” imposed.

Said that, i can conclude that the main problem is the education model, because you are educated about how to think, what to accept and then what to do to work. Would not be a coincidence that some successful people in the tech area broke their traditional educational model and then were able to create great things ?: Steve Jobs, Bill Gates, Mark Zuckerberg.. no one of them continue their studies, they just took ideas, connected points, imagine an improved work scenario, connected people.. etc.  I do not know if they were or are happy, but i am sure that they are satisfied with their archived goals…

Talking about educational models I cannot omit to mention One Laptop Per Child project, just to reminder it is not about cheap laptops, it is about access to the information and have an extra tool to help to explore and create.. you should review the success of this project with children in different countries…leave them a time alone and you will realize the amazing things that they do…

As this is just “my truth, I would suggest to watch the following videos, get your own conclusions:

This is not just about do what you want to do, it is about to wake up your conscience and listen your self… we can do something better for our kids and our selfs…

 

FYI: The Stanford University will start doing some public and free classes on their web site starting on January 2012, for more details check the following links:

Entrepreneurship

Civil Engineering

Electrical Engineering

Computer Science