I got a C.H.I.P! (9USD Computer)

Some weeks ago I attended the Embedded Linux Conference in San Diego, I’ve participate from the Showcase demonstrating Fluent Bit and a non-longer-secret-project that runs on top of it, more news in the incoming weeks.

The guys from NextThing Co were around giving for free C.H.I.Ps, for who’s not aware about what it is, the C.H.I.P is a 9USD ARM Embedded Computer, it’s pretty neat!

As soon you power up the device through the micro-USB, you can access the serial console and start playing with it; it comes with Debian 8, Wifi (AP mode supported), 512 MB Ram, 1GHz Processor, 4GB storage…within others (and for 9 dollars!). Definitely you should consider to get one!

I am putting my hands on to make packages now for Fluent Bit and Monkey HTTP Server

CMake: override subdirectory options

Last year I moved Monkey build system to CMake and when I started Fluent Bit I did it in the same way, a great experience.

Due to the nature of the projects mentioned where they can be build as libraries, I found that overriding the sub directories options was not too straightforward: if I have a project A with some options they should be listed like this in the CMakeLists.txt file:

option(WITH_FEATURE_AA  "Enable feature AA"   ON)
option(WITH_FEATURE_BB  "Enable feature BB"   OFF)

Now if I started a project B and I have included project A in a subdirectory, the files hierarchy should be:

project_B/
project_B/CMakeLists.txt
project_B/project_A/
project_B/project_A/CMakelists.txt

When building project_B based on it CMakeLists.txt rules, I may want to override some default options values of project_A, documentation is a bit confusing and you are likely to waste time trying to determinate why you cannot override the values, the guilty is the CMake Cache. As a workaround I found that building a simple macro in project_B/CMakeLists.txt solves the problem without complexity:

macro(SET_OPTION option value)
  set(${option} ${value} CACHE "" INTERNAL FORCE)
endmacro()

# Let's override the values
SET_OPTION(WITH_FEATURE_AA  OFF)
SET_OPTION(WITH_FEATURE_AA  ON)

# Now join the subdirectory
add_subdirectory(project_A)

If you want to see a working code example, checkout the following GIT repository:

http://github.com/edsiper/cmake-options

Fluent Bit: data from embedded/IoT matters

It’s not a surprise that year over year the number of connected embedded devices is growing exponentially; from one side many Hardware challenges are being addressed properly, we can find cheap devices that provides low power consumption with good connectivity, improved processors and flexible architectures in general. These are enough reasons to encourage the creation of innovative solutions for different markets such as automotive, medical and home automation within many.

When creating software on top of these new hardware, we often needs to address the next layer of challenges, one of them is data. The operating system and the set of applications that runs on your device often generate events that helps to troubleshoot of monitor general statuses, so collecting this information is a must.

There is one concept that is changing over the years, sometimes data is commonly known as logs, but logs are not longer considered a register as a set of bytes in the file system, logs are now a stream of events that usually can be collected through a serial or network interface.

Early this year at Treasure Data, we started researching the best ways to address all concerns described above for Embedded Software, for short how to solve data collection. I have to recall that data usually comes in different formats, while the Kernel writes to it Kernel messages interface, others applications writes to syslog or push some message to some network service.

Treasure Data solved data collection for servers with the creation of Fluentd, but this tool may not be applicable as a solution for all Embedded devices because of design and resources required, but there is a lot of acquired experience that can be used as the foundation for a new specialized solution, that’s why Fluent Bit have been created.



Along this year, the project have been growing and it’s becoming a great solution for data collection and data routing in general, the following table describe the usual scenarios on how Fluent Bit can collect data:

modes description
library Fluent Bit can be build as a shared or static library that can be used by your own program. So you invoke it API and ingest data to it.
metrics generator Fluent Bit auto-generate data from it built-in input metrics: CPU, Memory and Kernel log messages.
remote service This mode means that Fluent Bit runs as a service in a separate hardware and your embedded applications can send data to it.

In a more descriptive way, Fluent Bit provides input types for data, the current ones available as of v0.5 release are:

Input Plugin Description
CPU Calculate CPU usage in the system
Memory (metric) Gather memory usage in the system
Kernel log messages Gather Kernel logs from the Kernel messages interface
Serial Read data from the serial interface
STDIN Read data from the standard input interface
MQTT Listen for MQTT messages over the network.
XBee Listen for messages over a XBee Radio device.

Once the data is collected, it should go somewhere, that’s when the output plugins provide different options.

Output Plugin Description
Fluentd Fluentd is a famous open source data collector for servers, it provides an extensive list of interfaces for collecting and routing data to third party services.
Elastic Search Elastic Search is an open source database to search & analyze Data in Real Time.
Treasure Data Treasure Data provides an Analytics Cloud service for Big Data.
STDOUT Print event records to the standard output.

The project has reach a good level, flexible enough to behave as a puzzle in terms of which features to include and which not, it allows you to collect and route events easily across the network.

As this year I had the opportunity to attend and speak at a few conferences, I had learn from very experienced Embedded Engineers and end-users who shared their concerns and ideas on how data should be collected, this have been very useful to define current roadmap and general features available today.

Fluent Bit is Open Source and licensed under the terms of Apache License v2, feel free to contribute through our public repository on http://github.com/fluent/fluent-bit, or if you just want to get in touch, join our Slack team at http://slack.fluentd.org.

As said, embedded/IoT data matters, and Fluent Bit take cares of it, spread the word!

November Updates

It have been a long time since my last post, it’s a really busy year at work with enjoyable things to do!, but of course nothing is an excuse to stop sharing updates.

This year I have been working in several areas which includes software development, community strategies and spreading the word at several conferences. I think it have been the perfect mix of things to learn and improve. All of this consumes a lot of time, hopefully my wife and kids are very supportive on all of this, the balance is very important to keep everyone healthy.

Below you will find a short update for:

  • Fluent Bit
  • Conferences and Fluentd
  • Monkey & Duda I/O

Continue reading

Three months at Treasure Data

Almost three months ago, I started a new Open Source Adventure and I called it adventure because I had to take decisions where different variables to evaluate were involved like: family, stability, country, long-term-plans, etc.

So how things are going now?, I would say pretty well!, there is a lot of fun things to do, I feel more happy contributing than just calling working. One thing that I am very pleasant about Treasure Data is about the work environment, despites I am a remote worker (I work from Costa Rica), I had the opportunity to visit the HQ offices in Mountain View a couple of times and I can say that people is very friendly.

td_logo3

Continue reading

Ubuntu 14.10 + MacBook Pro Retina 13″ 2014

[Updated on March 10th, 2015]

I just got my new setup for work and as usual I performed Softwares changes: Install Ubuntu on the MackBook Pro retina. With the system by default I run on the following issues:

  • Suspend takes a long time
  • On Resume after suspend, the wireless driver stopped working
  • CPU scaling governor is powersave, but the CPU is always at maximum! (high temperature)

I manage to solve the problems with the following steps:

Continue reading

Starting a new Open Source Adventure

Two weeks ago was my last day at Oracle (after 7 years!) and now I am starting a new adventure.

I joined Treasure Data Inc as an Open Source developer to contribute to the Fluentd open source project and it related tools. Definitely this is a very exciting movement, the project is widely used and is the defacto Log/Events Collector for Servers, Mobile & Embedded. There is a lot of fun things to do and I will be mostly focused on it Golang, Embedded Linux versions and it Community within others, more news will come soon…

If you are a Fluentd user let me know, I’d be happy to hear from feedback and use cases, otherwise let’s talk anyways, I am sure it can solve your events collection needs :)

GSoC Reunion 2014

This is a very late post about the Google Summer of Code Reunion 2014. I will not share too many details, just a summary of the most relevant things (for me).

The big difference on this year, was that people were celebrating the 10th anniversary of Summer of Code. So many things were different compared to previous Mentors Summit:

  • Not a Mentors Summit, a Complete Reunion. A lot of students participated this time. Personally I met and I talked to many students, it’s very good to get feedback and see how was their experience, face-to-face feedback makes a big difference.
  • The event was hosted in the San Diego Marriott hotel. In previous years most of the activities were done on the Google Plex, this time was different and I can conclude it’s because the large amount of people attended between Organizations, Past-Organizations and Students.
  • Gala night event: I had to admit that was fun to see many people from the open source community dressed in casual mode, had the opportunity to see people from other countries using their native customs (I am sure there is a best word for that but I cannot find it). The big surprise of the night was to see Linus Torvalds who shared his experiences on stage. For the record: he was very nicely with everyone who wanted to talk to him, take a pic or drink a beer.
  • Google rented the Great America park just for US!, our first impression was: is this just for us!???.

On the other side, I had the opportunity to met one of the Monkey core developers Sonny Karlsson, very nice person!. Would be awesome if open source projects had the opportunity to met people more frequently.

Another surprise was to met face-to-face people that I used to collaborate with. Many years ago I was lucky enough to join the One Laptop Per Child organization through Google Summer of Code. On that moment I met Walter Bender, Bernie Innocenti and Martin Abente. Most of them are hardly working on Sugar and still helping the XO laptops on development countries, isn’t wonderful ? :)

15053059164_ababbb083d_z

15670979901_a557f189c0_z

15488112410_403535014d_z

15053551033_df39353e66_z

15649158946_edafd62cb2_z

15488112410_403535014d_z

15487743327_752212269e_z

More pictures available on the following link:

https://www.flickr.com/photos/edsiper/sets/72157648642056609/

The event was a real celebration, we also held many presentations and meetings, not only about GSoC but also about side projects within others.

In a few words, the event was great and very professional, I am always happy to share and listen about experiences from people that have years on Open Source or others that just joined, this environment really adds value and is very refreshing to have this kind of events.

Note: if you were interested into the Duda I/O presentation, here are the slides

[updates]
– Attached a Slideshare link to the Duda I/O presentation given at the lightning talks session

OSX Yosemite + Ubuntu 14.04 / dual boot

Before to start playing with the commands suggested here let me clarify my context:

  • I had a Mac Book Pro with OSX Maverick with dual boot working with Ubuntu 14.04 through rEFInd boot loader.
  • Today I upgraded OSX to Yosemite and after everything was updated rEFInd stopped working.

So my goal was to make able to have rEFInd working again to switch between operating systems. My primary OS is Linux so if you are in a similar situation this may help:

On a Twitter search I found someone linked the following solution:

http://pastebin.com/n1MCYxsF

That worked for me, I got a few warnings and I did not need to run the ‘mount’ command. Good luck!

Looking forward Monkey v1.6 and Duda I/O v2

The next versions of Monkey HTTP Server and Duda I/O are very challenging, because both are very related and big changes are coming in terms of optimizations and general features.

As said many times, Monkey is the base HTTP stack for Duda I/O, for hence any improvement made on Monkey is inherit by Duda I/O, as well any extra feature is exposed too. Let’s see what is expected for the following major versions:

Monkey

The code base continue being optimized, we care a lot of performance on all environments, not just high-end production servers, also in Embedded based systems where we got a lot of traction.

The big news besides performance, is the new event loop (polling) mechanism implemented, so starting from v1.6 we support Linux and also OSX. Yes, I know…what about our speech about Monkey is just for Linux ?, here is the answer: As a HTTP server Monkey it self does not have too much things to offer on an operating system that is not widely deployed on Internet Servers, neither on Embedded systems, but we need to look beyond and think in use cases, the main reason of Monkey being supported on OSX is Duda I/O.

Duda I/O is a fast web services framework, in short words: it let you build scalable C web services with a very simple API. Nowadays OSX is a base platform for developers, so our move is to provide a stable and fully featured web services stack to the Apple world. Wouldn’t be great to write your scalable web services in C on your favorite OSX system and deployment later easily on Linux servers or devices ?, yes it is. This is very exciting for us and continue involving a lot of work, but fun :) .

Duda I/O

When I mention a web services stack I refer to a professional product, fully documented and that have been used by production systems on the last two years. So major versions do not happen too frequently, we focus on long term support and stability.

The next version DST-2 (Duda Stable 2) will be released with great changes that covers a bit of everything:

  • Co-routines support (dthreads)
  • New routing model, do you like URLs like /users/:id/about/:phone/ ?, there you go.
  • Re-designed Console and optional Dashboard UI
  • Streams
  • Docker Support
  • More Documentation
  • Duda QA: Quality Assurance.

I mentioned Duda QA, so let me explain. Starting from DST-2 there is a new project that is part of Duda ecosystem, it deliver a complete Test system for Duda. For every Object API exists an unit test, as many changes are underway we need to make sure to avoid regressions. That means that as of now there are around 200 tests under 1300 lines of code…and growing. So every time we do some changes, you as an end-user can trust that we are not breaking the API or anything similar. There is nothing more frustrating than a broken API/stack, so keep calm as we are really taking care of that. More details about the project and status here:

https://github.com/monkey/duda-qa

In general many things are coming, there are other features and improvements that I did not mention but I am sure I will once I take some time to write the Release Notes, by now the works continue.