Some weeks ago I attended the Embedded Linux Conference in San Diego, I've participate from the Showcase demonstrating Fluent Bit and a non-longer-secret-project that runs on top of it, more news in the incoming weeks.

The guys from NextThing Co were around giving for free C.H.I.Ps, for who's not aware about what it is, the C.H.I.P is a 9USD ARM Embedded Computer, it's pretty neat!

As soon you power up the device through the micro-USB, you can access the serial console and start playing with it; it comes with Debian 8, Wifi (AP mode supported), 512 MB Ram, 1GHz Processor, 4GB storage...within others (and for 9 dollars!). Definitely you should consider to get one!

I am putting my hands on to make packages now for Fluent Bit and Monkey HTTP Server...

Almost three months ago, I started a new Open Source Adventure and I called it adventure because I had to take decisions where different variables to evaluate were involved like: family, stability, country, long-term-plans, etc.

So how things are going now?, I would say pretty well!, there is a lot of fun things to do, I feel more happy contributing than just calling working. One thing that I am very pleasant about Treasure Data is about the work environment, despites I am a remote worker (I work from Costa Rica), I had the opportunity to visit the HQ offices in Mountain View a couple of times and I can say that people is very friendly.

td_logo3

Continue reading

This is a very late post about the Google Summer of Code Reunion 2014. I will not share too many details, just a summary of the most relevant things (for me).

The big difference on this year, was that people were celebrating the 10th anniversary of Summer of Code. So many things were different compared to previous Mentors Summit:

  • Not a Mentors Summit, a Complete Reunion. A lot of students participated this time. Personally I met and I talked to many students, it's very good to get feedback and see how was their experience, face-to-face feedback makes a big difference.
  • The event was hosted in the San Diego Marriott hotel. In previous years most of the activities were done on the Google Plex, this time was different and I can conclude it's because the large amount of people attended between Organizations, Past-Organizations and Students.
  • Gala night event: I had to admit that was fun to see many people from the open source community dressed in casual mode, had the opportunity to see people from other countries using their native customs (I am sure there is a best word for that but I cannot find it). The big surprise of the night was to see Linus Torvalds who shared his experiences on stage. For the record: he was very nicely with everyone who wanted to talk to him, take a pic or drink a beer.
  • Google rented the Great America park just for US!, our first impression was: is this just for us!???.

On the other side, I had the opportunity to met one of the Monkey core developers Sonny Karlsson, very nice person!. Would be awesome if open source projects had the opportunity to met people more frequently.

Another surprise was to met face-to-face people that I used to collaborate with. Many years ago I was lucky enough to join the One Laptop Per Child organization through Google Summer of Code. On that moment I met Walter Bender, Bernie Innocenti and Martin Abente. Most of them are hardly working on Sugar and still helping the XO laptops on development countries, isn't wonderful ? 🙂

15053059164_ababbb083d_z

15670979901_a557f189c0_z

15488112410_403535014d_z

15053551033_df39353e66_z

15649158946_edafd62cb2_z

15488112410_403535014d_z

15487743327_752212269e_z

More pictures available on the following link:

https://www.flickr.com/photos/edsiper/sets/72157648642056609/

The event was a real celebration, we also held many presentations and meetings, not only about GSoC but also about side projects within others.

In a few words, the event was great and very professional, I am always happy to share and listen about experiences from people that have years on Open Source or others that just joined, this environment really adds value and is very refreshing to have this kind of events.

Note: if you were interested into the Duda I/O presentation, here are the slides

[updates]
- Attached a Slideshare link to the Duda I/O presentation given at the lightning talks session

The next versions of Monkey HTTP Server and Duda I/O are very challenging, because both are very related and big changes are coming in terms of optimizations and general features.

As said many times, Monkey is the base HTTP stack for Duda I/O, for hence any improvement made on Monkey is inherit by Duda I/O, as well any extra feature is exposed too. Let's see what is expected for the following major versions:

Monkey

The code base continue being optimized, we care a lot of performance on all environments, not just high-end production servers, also in Embedded based systems where we got a lot of traction.

The big news besides performance, is the new event loop (polling) mechanism implemented, so starting from v1.6 we support Linux and also OSX. Yes, I know...what about our speech about Monkey is just for Linux ?, here is the answer: As a HTTP server Monkey it self does not have too much things to offer on an operating system that is not widely deployed on Internet Servers, neither on Embedded systems, but we need to look beyond and think in use cases, the main reason of Monkey being supported on OSX is Duda I/O.

Duda I/O is a fast web services framework, in short words: it let you build scalable C web services with a very simple API. Nowadays OSX is a base platform for developers, so our move is to provide a stable and fully featured web services stack to the Apple world. Wouldn't be great to write your scalable web services in C on your favorite OSX system and deployment later easily on Linux servers or devices ?, yes it is. This is very exciting for us and continue involving a lot of work, but fun 🙂 .

Duda I/O

When I mention a web services stack I refer to a professional product, fully documented and that have been used by production systems on the last two years. So major versions do not happen too frequently, we focus on long term support and stability.

The next version DST-2 (Duda Stable 2) will be released with great changes that covers a bit of everything:

  • Co-routines support (dthreads)
  • New routing model, do you like URLs like /users/:id/about/:phone/ ?, there you go.
  • Re-designed Console and optional Dashboard UI
  • Streams
  • Docker Support
  • More Documentation
  • Duda QA: Quality Assurance.

I mentioned Duda QA, so let me explain. Starting from DST-2 there is a new project that is part of Duda ecosystem, it deliver a complete Test system for Duda. For every Object API exists an unit test, as many changes are underway we need to make sure to avoid regressions. That means that as of now there are around 200 tests under 1300 lines of code...and growing. So every time we do some changes, you as an end-user can trust that we are not breaking the API or anything similar. There is nothing more frustrating than a broken API/stack, so keep calm as we are really taking care of that. More details about the project and status here:

https://github.com/monkey/duda-qa

In general many things are coming, there are other features and improvements that I did not mention but I am sure I will once I take some time to write the Release Notes, by now the works continue.

Monkey

I planned to have a quite Saturday, but after check my email in the morning, I found a possible security issue reported for Monkey (1.5 onwards), it may lead to a DDoS based on the new optional FDT feature, so it was time to research and release a fix as soon as possible.

So Monkey v1.5.3 was released which include the fix for the problem reported, i also take the opportunity to include other patchs sent by the reporter, so the goal was accomplished: act transparent and quick.

Duda I/O

Also during the week i have been working in a new look & feel for Duda I/O project, you cannot imagine how hard is to create an image for a Software that don't have a UI (yet). So the project just got 3 major things:

New logo

Duda I/O is a Stack, for hence from my perspective I think this represents it nature and simplicity:

not sure if people would love it but I am sure it will fit very well on Project T-shirts and Stickers 🙂

New Web Site

As the new logo arrived, was a good time to complete the new responsive web site:

Disclaimer: I bought a bootstrap template and did the adjustments, i am not a designer.

compared to the previous project site this is a major upgrade.

Google Summer of Code

We are in the pencils down phase of the Google Summer of Code program, that means most of our students are doing their final adjustments on their projects as the final evaluations may start this evening, all of them have done a very good job, they deserve the credit. More news about cool stuff that was made will be public in the incoming days.

It's a good jurney, life is moving a lot in a positive way, in all aspects...

I always believed that integration and focus are a success key for any kind of thing on this life. When talking about technology and servers specifically, integration may be a fundamental part to achieve different goals like good performance. Years ago was very valuable to have a software that can work on multiple Operating Systems, but there is a point that if you want to increase your performance limits you need to look forward for integration.

In the last years, Linux have become the best choice for Servers, it's well known that every big company trust in Linux and also commit to their development: Redhat, Oracle, Google, HP, IBM and many many others. So we can say that Linux is doing pretty well on professional Hardware, but what about user-space software ?.

If we look in the HTTP Servers (or Web Servers) context, we will find that most used solutions are made to support different platforms and on specific build-time for a specific Operating System, different approaches are used to work better, in servers performance and resources consumption are critical, and when moving those solution to Embedded even more, so every taken decision may sacrifice something, we do in OS portability.

Monkey HTTP Server and Duda I/O are written with focus on the Linux Kernel, every Unix operating system despite their POSIX API, provides extra system calls that aims to extend functionality and offer features. Of course this kind of integration have a cost which is basically: do not be portable to other Operating System, so depending of your human resources you need to decide how to invest better: we do in focus and integration, and it works great.

As a reference i would like to mention which specific features we use and i hope people who write servers or any kind of networking application would take this in count. Of course  these are dependent of specific Linux Kernel versions:

Linux 3.14

  • TCP AutoCorking: let Kernel do smart buffering when sending outgoing data, the old fashion way is to do manual Cork over sockets (more).

Linux 3.9

  • SO_REUSEPORT: be able to reuse port-addresses, e.g: multiple threads under same process can bind the same port and let the Kernel distribute load across them (more).

Linux 3.7

  • TCP FastOpen: Reduce the number of round-trip required in a TCP conversation (more).

Also other very helpful interfaces exists such as accept4(2), sendfile(2), epoll(7) and splice(2), these are not the only ones but that should be considered when writing a Server for Linux.

We are in good times where performance matters and make a good design to make distribute load across multi-core architectures is fun and challenging. Again, if you write with focus and integration will be a win:win relationship between your network server and the HW behind it.

Duda I/O web services framework project was started almost two years ago, initially as a Monkey HTTP Server extension to create scalable web services in C, nowadays both components together are a complete stack to develop any kind of event-driven web backend application that relies on HTTP or Websockets.  As mentioned in early post, the web is not just about images, CSS or any JS scripts, its about also about fast data transfer and scalable services that aim to optimize resources usage with low overhead.

Sincerely i would say that Duda I/O and Monkey projects are the Mozilla of the backend side, of course we are far of their commitment but i would say the mission is the same: promote openness, innovation & opportunity on the Web.

Scale a C project at a community level is hard, due to the target market, the number of early adopters is very low compared to any  GUI or Web (front end) project, but once it get's in and it works.. things changes in a positive way. At the moment there are a couple of Duda I/O instances running in production in private companies from US, Sweden and Germany, and i can conclude by their feedback the reasons of the adoption are pretty much the same:

  • High performance: all implemented web services requires some kind of real-time processing data coming from HTTP requests.
  • Open source stack: proof of good security handling, no back-doors. What really matters is that "It is open" (not talking about money).
  • Proof of years of development and commitment, the project will not disappear because of a one man project.
  • Good support: on early stages of evaluating the stack, their always receive good support through the mailing list or private emails.

Also much of the latest changes on the stack, comes from end-users feedback on topics such as documentation, development environment, API changes, etc. So this year have been a big step forward on terms of adoption and features improvements, next year will be even more fun and of course more hard work on:

  • Native deployment, native packaging for different Linux distributions.
  • Co-routines support to make easier to defer async work from callbacks.
  • SPDY & HTTP/2.0 .
  • New QA environment, stability and avoid regressions are high priority.
  • Clustering mode.
  • Sandboxing.
  • ...others.

Personally i am just happy with all goals achieved during this period and very thankful to be part of a small but great community.

thanks Google for the opportunity! , this week the results for the Mentoring Organizations for Google Summer of Code 2013 were released. I am very happy to announce that Monkey Project will be part of the program as second consecutive year.

We are a very small group of people developing an open source stack for web technologies and take part of the program means a big recognition, we are glad to be able to mentor students that will be hacking with us during three months.

Our last year experience was very positive, most of projects were developed successfully and those students have take an important role in our community and project development in general, so we have high expectations for this second round, as you can see there is a good scope of project ideas to develop in areas such as protocols (SPDY), OSX, Raspberry Pi, Web Services, etc.

You can get more details about the incoming work in our Monkey GSoC project ideas page.