Porting a Ragel GPS NMEA Parser to AVR ATMega128
Synopsis: This article
shows the result of porting the GPS NMEA parser
introduced in
“A Ragel Parser for NMEA Sentences” to
an AVR ATMega128
with a Falcom FSA03 GPS
for source data. Furthermore, this port to the
AVR has refined the NMEA parser in several ways
including making more it robust and testing it
under non-GPS locked conditions. The code reads
GPS NMEA data on one ATMega128 TTL serial port and
pumps a JSON structure containing GPS location
data out the other other port, an XBee data
radio.
One of my goals with Ragel was to build powerful parsers that could live on small microcontrollers. Early on, when I started Ragel Recipies, I stated I wanted to see how far down the technology stack I could push Ragel including building parsers for microcontrollers. This is the successful culmination of that proof point. Ragel is indeed well suited for building parsers for embedded systems.
Ragel AVR Target
The target board is the RAD-GPS board I’ve designed which includes an ET-AVR ATMega128 stamp, an FSA03 GPS and an XBee data radio. The board has an onboard power supply that can take 6-30V input and a JTAG port for debugging which came in very handy to debug the port. Here’s an image of the target board. My plan is to use this board for the autonomous vehicle we’ll enter into the Sparkfun autonomous vehicle competition in mid-April.
The target C compiler is CodeVision AVR.
The previous Ragel NMEA parser I presented targeted GCC, so I needed to start with a basic skeleton for the AVR. With the CodeVision AVR wizard, I generated a basic application that included receive interrupt handlers for communication on both ATMega128 serial ports.
Refactoring CodeVision AVR Wizard Code
I then refactored the entire wizard output. One thing I’ve always disliked about CodeVision AVR wizard output is that it generates one large monolithic code file. That makes it cluttered to work with and in my case, I would have had to include all that code in the Ragel .rl definition. In any case, I was going to need to break it up.
So, I ripped out all the async comms and put those routines into their own .c file and added .h externs. Expecting to include a gps.c file into the project, I added a gps.h skeleton file and included it in the main .c file which I called autoveh.c (because I’m gunning to use this for my autonomous vehicle.)
At this point, it was a lot like I had torn up my kitchen...from a code perspective I had pipes sticking out, wires splayed everywhere.
Porting the Ragel .rl file to AVR
I copied the gps.rl file I developed in the previous Ragel Recipe: “A Ragel Parser for NMEA GPS Sentences” to my AVR project folder and began modifying it for the AVR target. The main thing I added was include files to the top of the gps.rl to target the AVR:
I then generated the parser from the OS X host where I had Ragel installed and shared source directory with my Windows VM where CodeVision AVR compiler lives. I added the gps.c file to the AVR project and started trial compiles with CodeVision AVR.
After working out various syntax errors, my next task was to integrate the GPS NMEA stream coming from the AVR serial port into the parser itself. The CodeVision AVR wizard generates an interrupt handler that fills a circular buffer with serial data. I needed to pull that data and feed it to the parser. There are multiple ways to do this, but the most straightforward way that I went for was to read an entire sentence out of the buffer an then pass that sentence into the parser. Ragel can parse fragments and maintain state which is nice, but in my case, that didn’t really buy me much. If I wanted to save memory or make it somewhat faster, I could have spent the time to feed the circular buffer directly to Ragel, but because that structure is modified at interrupt time, it would have taken more work to make that safe.
So, the main control loop of the firmware is simply reading GPS NMEA sentences from the ATMega128 serial port, making a single sentence and passing it into the Ragel parser. The Ragel parser tries to make sense out of the data and populates a data structure with location data. That structure, as explained in the previous article is cumulative. As various NMEA sentences are encountered and parsed, various portions of the structure get filled in and fields overwritten.
That way, the main control loop of the firmware can refer to that structure at any time the most current location data. NMEA sentences which contain redundant information, say latitude and longitude, get parsed and the structure updated, so lat/lon may get updated very frequently and other types of attributes, say altitude or speed, less frequently.
The main loop is constantly pumping out a JSON structure over the XBee serial interface with some of the critical data in it - mainly using this for debug right now, but ultimately, the JSON can be directly stuffed into a CouchDB or used directly by some other AJAX type application if needed.
When I’m testing, I have an XBee Regulated breakout board from Sparkfun connected to my OS X machine so I can monitor the output of the RAD-GPS board. I get a continuous stream of JSON output like this:
Most of these attributes are self-explanatory and map directly to GPS concepts, but there are a few extra attributes I added for diagnostic and management purposes. First, ‘vid’ is vehicle ID. My thinking is that if there are multiple devices to be tracked and all are transmitting on the same PAN ID of XBee, then it would be useful to sort them out. Haven’t worked out all the multiple XBee network things, but just making a data format that anticipates not being the only device out there. Secondly, “nmeaerrors” and “sentences” are counters. NMEA errors are sentences that for, whatever reason, the Ragel parser couldn’t parse. The “sentences” attribute is the number of sentences we’ve parsed...it helps indicate whether NMEA data is flowing at all.
This JSON structure is then streaming out the RAD-GPS XBee port to my OS X box and I can see what’s going on with the parser and GPS performance.
The Ragel .rl definition file for the NMEA parser on an AVR microcontroller then boiled down to this listing. Notice, if you compare this .rl with the .rl in the previous article, the Ragel machines that are defined are much more robust for handling non-locked GPS conditions where there are many fields that have no data in them “,,,,,” sort of output from the GPS. It also includes the NMEA error counts and will dump the syntax problems it encounters along the way....usually an indication my Ragel definition didn’t account for some sentence structure I wasn’t aware of a priori.
The main autoveh.c firmware loop which initializes the AVR micro and reads GPS data and passes it to the “parse_nmea()” function looks like this:
Lessons Learned
The main lessons I learned in the Ragle parser port to the AVR are:
Ragel can definitely be used to generate substantially complex parsers which are small enough to easily fit in a modern flash-based microcontroller like the AVR ATMega128. This opens up huge possibilities for writing network protocols in Ragel that can be shared between the embedded device and the servers and systems that embedded devices often must communicate with.
The micro must be fast enough to service serial data interrupts from the GPS and process and parse the sentences. The Ragel generated parser has also proven itself to be fast enough and have AVR cycles to do real tasks. From my experiments so far, the ATMega128 running at 16MHz is plenty strong enough to run Ragel parsers, specifically, a real-world, complex parser such as the NMEA parser. The parser I threw at the AVR is not a toy.
I’m encouraged to continue using this as a tracking platform and platform for autonomous vehicle control.
Landon Cox
www.ESawdust.com
Also see these other articles in the Ragel Recipes parser series:
“A Skeleton Command Line Intepreter in Ragel for Ruby”
“A Ragel Parser for NMEA GPS Setnences”
“What is Ragel? Why Ragel?”
One of my goals with Ragel was to build powerful parsers that could live on small microcontrollers. Early on, when I started Ragel Recipies, I stated I wanted to see how far down the technology stack I could push Ragel including building parsers for microcontrollers. This is the successful culmination of that proof point. Ragel is indeed well suited for building parsers for embedded systems.
Ragel AVR Target
The target board is the RAD-GPS board I’ve designed which includes an ET-AVR ATMega128 stamp, an FSA03 GPS and an XBee data radio. The board has an onboard power supply that can take 6-30V input and a JTAG port for debugging which came in very handy to debug the port. Here’s an image of the target board. My plan is to use this board for the autonomous vehicle we’ll enter into the Sparkfun autonomous vehicle competition in mid-April.
The target C compiler is CodeVision AVR.
The previous Ragel NMEA parser I presented targeted GCC, so I needed to start with a basic skeleton for the AVR. With the CodeVision AVR wizard, I generated a basic application that included receive interrupt handlers for communication on both ATMega128 serial ports.
Refactoring CodeVision AVR Wizard Code
I then refactored the entire wizard output. One thing I’ve always disliked about CodeVision AVR wizard output is that it generates one large monolithic code file. That makes it cluttered to work with and in my case, I would have had to include all that code in the Ragel .rl definition. In any case, I was going to need to break it up.
So, I ripped out all the async comms and put those routines into their own .c file and added .h externs. Expecting to include a gps.c file into the project, I added a gps.h skeleton file and included it in the main .c file which I called autoveh.c (because I’m gunning to use this for my autonomous vehicle.)
At this point, it was a lot like I had torn up my kitchen...from a code perspective I had pipes sticking out, wires splayed everywhere.
Porting the Ragel .rl file to AVR
I copied the gps.rl file I developed in the previous Ragel Recipe: “A Ragel Parser for NMEA GPS Sentences” to my AVR project folder and began modifying it for the AVR target. The main thing I added was include files to the top of the gps.rl to target the AVR:
I then generated the parser from the OS X host where I had Ragel installed and shared source directory with my Windows VM where CodeVision AVR compiler lives. I added the gps.c file to the AVR project and started trial compiles with CodeVision AVR.
After working out various syntax errors, my next task was to integrate the GPS NMEA stream coming from the AVR serial port into the parser itself. The CodeVision AVR wizard generates an interrupt handler that fills a circular buffer with serial data. I needed to pull that data and feed it to the parser. There are multiple ways to do this, but the most straightforward way that I went for was to read an entire sentence out of the buffer an then pass that sentence into the parser. Ragel can parse fragments and maintain state which is nice, but in my case, that didn’t really buy me much. If I wanted to save memory or make it somewhat faster, I could have spent the time to feed the circular buffer directly to Ragel, but because that structure is modified at interrupt time, it would have taken more work to make that safe.
So, the main control loop of the firmware is simply reading GPS NMEA sentences from the ATMega128 serial port, making a single sentence and passing it into the Ragel parser. The Ragel parser tries to make sense out of the data and populates a data structure with location data. That structure, as explained in the previous article is cumulative. As various NMEA sentences are encountered and parsed, various portions of the structure get filled in and fields overwritten.
That way, the main control loop of the firmware can refer to that structure at any time the most current location data. NMEA sentences which contain redundant information, say latitude and longitude, get parsed and the structure updated, so lat/lon may get updated very frequently and other types of attributes, say altitude or speed, less frequently.
The main loop is constantly pumping out a JSON structure over the XBee serial interface with some of the critical data in it - mainly using this for debug right now, but ultimately, the JSON can be directly stuffed into a CouchDB or used directly by some other AJAX type application if needed.
When I’m testing, I have an XBee Regulated breakout board from Sparkfun connected to my OS X machine so I can monitor the output of the RAD-GPS board. I get a continuous stream of JSON output like this:
Most of these attributes are self-explanatory and map directly to GPS concepts, but there are a few extra attributes I added for diagnostic and management purposes. First, ‘vid’ is vehicle ID. My thinking is that if there are multiple devices to be tracked and all are transmitting on the same PAN ID of XBee, then it would be useful to sort them out. Haven’t worked out all the multiple XBee network things, but just making a data format that anticipates not being the only device out there. Secondly, “nmeaerrors” and “sentences” are counters. NMEA errors are sentences that for, whatever reason, the Ragel parser couldn’t parse. The “sentences” attribute is the number of sentences we’ve parsed...it helps indicate whether NMEA data is flowing at all.
This JSON structure is then streaming out the RAD-GPS XBee port to my OS X box and I can see what’s going on with the parser and GPS performance.
The Ragel .rl definition file for the NMEA parser on an AVR microcontroller then boiled down to this listing. Notice, if you compare this .rl with the .rl in the previous article, the Ragel machines that are defined are much more robust for handling non-locked GPS conditions where there are many fields that have no data in them “,,,,,” sort of output from the GPS. It also includes the NMEA error counts and will dump the syntax problems it encounters along the way....usually an indication my Ragel definition didn’t account for some sentence structure I wasn’t aware of a priori.
The main autoveh.c firmware loop which initializes the AVR micro and reads GPS data and passes it to the “parse_nmea()” function looks like this:
Lessons Learned
The main lessons I learned in the Ragle parser port to the AVR are:
Ragel can definitely be used to generate substantially complex parsers which are small enough to easily fit in a modern flash-based microcontroller like the AVR ATMega128. This opens up huge possibilities for writing network protocols in Ragel that can be shared between the embedded device and the servers and systems that embedded devices often must communicate with.
The micro must be fast enough to service serial data interrupts from the GPS and process and parse the sentences. The Ragel generated parser has also proven itself to be fast enough and have AVR cycles to do real tasks. From my experiments so far, the ATMega128 running at 16MHz is plenty strong enough to run Ragel parsers, specifically, a real-world, complex parser such as the NMEA parser. The parser I threw at the AVR is not a toy.
I’m encouraged to continue using this as a tracking platform and platform for autonomous vehicle control.
Landon Cox
www.ESawdust.com
Also see these other articles in the Ragel Recipes parser series:
“A Skeleton Command Line Intepreter in Ragel for Ruby”
“A Ragel Parser for NMEA GPS Setnences”
“What is Ragel? Why Ragel?”
A Skeleton Command Interpreter in Ragel for Ruby
03/24/2010 20:35 Filed in: Ragel
Synopsis: Do you need
to write a command interpreter to accept input from a
user or even an automated protocol? Commands must be
parsed and an action taken when the command is
recognized. An error action or message must be
produced when a bad command is given. The following
is a Ragel
definition I developed as a common starting
point for a command interpreter. The target in
this case is generated in Ruby, but could easily
be adapted for C or Java targets. The code
demonstrates how to interpret three fictitious
commands and catch syntactically incorrect inputs
as well. It demonstrates Ragel error handling
actions that apply to command interpretation. The
Ragel code is liberally commented.
In this example command interpreter grammar there are 3 commands: "Bark", "Shout", "Grunt". Each command takes a numeric parameter up to 5 digits long.
When a specific command is recognized, a "found" action is called based on the type of command recognized. This “found” action is where you would perform some task based on the command that was recognized.
When a command is syntactically incorrect, an error action is called. The command error handler can be filled in to specifically deal with command errors for your application.
Finally, this code also demonstrates the start and end events that can be captured when a command is being parsed - start and end actions are fired at the appropriate times so you can fill in those handlers to do something special when command parsing starts or finishes.
To build the Ragel code and generate the state diagram, you can use this Bash script:
To run the resulting parser (Ragel will generate testcmd.rb):
expected output from running the generated Ruby is
The Ragel code for a simple command interpreter with error handling:
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
In this example command interpreter grammar there are 3 commands: "Bark", "Shout", "Grunt". Each command takes a numeric parameter up to 5 digits long.
When a specific command is recognized, a "found" action is called based on the type of command recognized. This “found” action is where you would perform some task based on the command that was recognized.
When a command is syntactically incorrect, an error action is called. The command error handler can be filled in to specifically deal with command errors for your application.
Finally, this code also demonstrates the start and end events that can be captured when a command is being parsed - start and end actions are fired at the appropriate times so you can fill in those handlers to do something special when command parsing starts or finishes.
To build the Ragel code and generate the state diagram, you can use this Bash script:
To run the resulting parser (Ragel will generate testcmd.rb):
expected output from running the generated Ruby is
The Ragel code for a simple command interpreter with error handling:
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
A Ragel Parser For NMEA GPS Sentences
Synopsis: This is
a Ragel definition
file, gps.rl, which implements a parser for NMEA
GPS sentences $GPGGA, $GPRMC, $GPGSA, GPGLL,
GPGSV, GPVTG. The target parser is generated in C
and the main() is a set of test functions that
send the parser through some paces based on actual
GPS strings I gathered from a custom GPS PCB with
a Falcom FSA03 GPS
that I built.
As promised in my introduction to Ragel Recipes, some recipes would be full-blown Ragel parsers and some would just be Ragel snippets. This is the former category - a complete Ragel NMEA parser.
The target parser was built using GCC on an OS X machine, but is completely portable to any Linux platform. I haven’t spent any time on Windows with Ragel to add anything wise about using it with Windows. Ragel 6.6 (Dec 2009) was used to compile the .rl file.
You can take this parser, replace main(), read NMEA sentences from your systems serial port where you have a GPS connected, and feed each line to the parser. The way I implemented this code is there is a location data structure that accumulates the current location data as it is reading the sentences, so your app, at any one time only has to examine the data structure currentPosition to get everything that has been learned by the GPS - the structure contains things like latitude, longitude, speed, heading and a lot more.
Instructions for building this are simple (assuming you have installed the latest Ragel and GCC):
To run the code, from a shell, just execute ‘gps’:
More validation code would need to be added to main() to make it a completely automated test for more rigorous use in a production setting.
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
As promised in my introduction to Ragel Recipes, some recipes would be full-blown Ragel parsers and some would just be Ragel snippets. This is the former category - a complete Ragel NMEA parser.
The target parser was built using GCC on an OS X machine, but is completely portable to any Linux platform. I haven’t spent any time on Windows with Ragel to add anything wise about using it with Windows. Ragel 6.6 (Dec 2009) was used to compile the .rl file.
You can take this parser, replace main(), read NMEA sentences from your systems serial port where you have a GPS connected, and feed each line to the parser. The way I implemented this code is there is a location data structure that accumulates the current location data as it is reading the sentences, so your app, at any one time only has to examine the data structure currentPosition to get everything that has been learned by the GPS - the structure contains things like latitude, longitude, speed, heading and a lot more.
Instructions for building this are simple (assuming you have installed the latest Ragel and GCC):
To run the code, from a shell, just execute ‘gps’:
More validation code would need to be added to main() to make it a completely automated test for more rigorous use in a production setting.
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
What is Ragel? Why Ragel?
03/18/2010 20:01 Filed in: Ragel
Synopsis: I started
this category on Ragel Recipes for several reasons.
For a long time, I’ve wanted to learn
Ragel for various
parsing tasks often encountered, and two, I wanted
to see how far down the technology stack I could
push it. I wanted to find out if I could use it
effectively in an embedded environment like an
AVR. This “Ragel Recipes” category will capture
some of what I’m learning about Ragel as I
implement it lower and lower in the stack and also
at the host level. Embedded devices very often are
not standalone - they need to communicate via some
protocol with some host. Ragel is suitable for
writing parsers on both ends of the spectrum.
What is Ragel?
Simply put: Ragel is a tool which can generate a parser based on a description language, but can also make calls to user functions as it parses a input stream. It can spit out parsers in numerous popular languages. A more involved definition and description of Ragel can be found on the Ragel home page. [Credit for Ragel goes to Adrian Thurston].
My Motivation for Learning Ragel
When you first start looking at the Ragel user guide it can be pretty daunting. Learning Ragel is a significant investment of time and effort so you need to have a painful problem to solve to make it worth it.
For me, the painful problem was simple: After 27+ years as a software engineer, I’m tired of writing parsers by hand. When you are wet behind the ears, writing a parser is a nice challenge...sophomore CS stuff. Over the years, you keep running up against the same problems - needing to parse an input stream - and it’s nice to have learned how to hand parse stuff early on in the career.
However, after you’ve long since lost count of how many parsers you’ve written in your career, well, that’s where Ragel comes into its own. When you don’t have anything left to prove, learn Ragel.
My Practical Uses of Ragel
I first started thinking about using Ragel when I was considering next steps with the autonomous vehicle we’re working on. The vehicle microcontroller has a GPS and data radio on it. I had already worked up a cool looking user interface using TouchOSC on my iPhone (an as yet to be written blog article) and wanted to send configuration commands to my vehicle over the data radio.
I was faced with two tasks - 1) reading the GPS stream and keeping track of current location bits and 2) reading and responding to the command input stream via the data radio. Both required parsers and both required doing something based on the stream as it was flowing in. Writing yet another NMEA parser for the GPS didn’t seem very worthwhile but I also didn’t feel like cobbing someone else’s - I wanted to “own” it. Secondly, the comm protocol between my iPhone TouchOSC app and the vehicle was going to be a custom protocol - I needed to invent one.
Since everything in this application was about reading a continuous stream of data and responding to it as fast as possible, Ragel was a natural choice. Ragel parsers are known to be fast, nearly as fast as the best hand-written parser, and small, so there was a hope it would fit into a microcontroller flash.
Also, since Ragel can generate parsers in C, C++, and Ruby, it easily covered my needs both at the embedded level and at the host level. In other words, I could learn the ins-and-outs of Ragel and apply it in multiple domains so my learning curve is leveraged. I think that is one of the most beautiful things about using Ragel - it’s extremely versatile.
This Ragel Recipes category of the ESawdust blog is mainly a stash of Ragel things I’m learning and a basic portal for sharing that code with others who might benefit from the technology.
Whenever I learn something new, it always helps to have a real project in mind and start working on it. So, what’s in these recipes are either code snippets or full-bore Ragel definitions for various problems I’ve applied it to. In some cases, I’ll put up a recipe for no other reason than I want to easily refer to it later...it’s my Ragel notebook.
The Ragel Recipe Roadmap
1) Ragel Parser for NMEA GPS
The first Ragel Recipe is a full-on NMEA GPS parser I wrote - target: C language. The reason for targeting C in this case, is part of my roadmap for using Ragel - I want to use this parser on a small AVR microcontroller to parse the NMEA data stream and maintain current location data for my autonomous vehicle. This meant that the parser could not be huge like some other parser generators would emit and it had to be in C so I could build the firmware.
2) Ragel parsers for Linux log data
I spend a lot of time pawing through Linux logs - firewall, mail, messages, snort, and so on. I want to build some custom automation tools that can help me answer various, specific questions I have when I go into a log. “Why were there 3,424 messages in an hour?” “What were the preponderance of messages about?” Might possibly use some “R” to help with stats, but use Ragel to help dig and pull the components out of the sea of log data.
I suspect numerous recipes will simply be Ragel definitions, mini machines, for specific types of log messages. Nothing earth shattering, but over time, a small portfolio of Ragel state machines will build up here in the recipes. These might more aptly be described as Ragel “phrases” than “recipes.”
Recipes for parsing linux log data will almost all be targeting Ruby parsers as Ruby’s my first choice for system admin work on Linux (if I have a choice.)
3) Ragel parsers on an AVR
This is a huge thing for me. I really want to see how feasible it is to use a major parser generator like Ragel to solve the day-to-day parsing problems an embedded developer faces. Rather than hand-coding parsers every time I turn around, I would like to turn to a single tool, Ragel, that I know can cover the smallest targets like AVR as well as the hosts they talk to. This roadmap is designed to find the benefits and limitations of this approach. I’ll put Ragel to the test on an AVR and see how far I can get.
Conclusion
I’m not claiming to be an expert Ragel user, but just one who is willing to share his experience. These recipes are the result of “pressing on” with Ragel and I feel that I have been rewarded so far. I hope you will be encouraged to learn Ragel as you check back for new recipes and follow the progress.
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
What is Ragel?
Simply put: Ragel is a tool which can generate a parser based on a description language, but can also make calls to user functions as it parses a input stream. It can spit out parsers in numerous popular languages. A more involved definition and description of Ragel can be found on the Ragel home page. [Credit for Ragel goes to Adrian Thurston].
My Motivation for Learning Ragel
When you first start looking at the Ragel user guide it can be pretty daunting. Learning Ragel is a significant investment of time and effort so you need to have a painful problem to solve to make it worth it.
For me, the painful problem was simple: After 27+ years as a software engineer, I’m tired of writing parsers by hand. When you are wet behind the ears, writing a parser is a nice challenge...sophomore CS stuff. Over the years, you keep running up against the same problems - needing to parse an input stream - and it’s nice to have learned how to hand parse stuff early on in the career.
However, after you’ve long since lost count of how many parsers you’ve written in your career, well, that’s where Ragel comes into its own. When you don’t have anything left to prove, learn Ragel.
My Practical Uses of Ragel
I first started thinking about using Ragel when I was considering next steps with the autonomous vehicle we’re working on. The vehicle microcontroller has a GPS and data radio on it. I had already worked up a cool looking user interface using TouchOSC on my iPhone (an as yet to be written blog article) and wanted to send configuration commands to my vehicle over the data radio.
I was faced with two tasks - 1) reading the GPS stream and keeping track of current location bits and 2) reading and responding to the command input stream via the data radio. Both required parsers and both required doing something based on the stream as it was flowing in. Writing yet another NMEA parser for the GPS didn’t seem very worthwhile but I also didn’t feel like cobbing someone else’s - I wanted to “own” it. Secondly, the comm protocol between my iPhone TouchOSC app and the vehicle was going to be a custom protocol - I needed to invent one.
Since everything in this application was about reading a continuous stream of data and responding to it as fast as possible, Ragel was a natural choice. Ragel parsers are known to be fast, nearly as fast as the best hand-written parser, and small, so there was a hope it would fit into a microcontroller flash.
Also, since Ragel can generate parsers in C, C++, and Ruby, it easily covered my needs both at the embedded level and at the host level. In other words, I could learn the ins-and-outs of Ragel and apply it in multiple domains so my learning curve is leveraged. I think that is one of the most beautiful things about using Ragel - it’s extremely versatile.
This Ragel Recipes category of the ESawdust blog is mainly a stash of Ragel things I’m learning and a basic portal for sharing that code with others who might benefit from the technology.
Whenever I learn something new, it always helps to have a real project in mind and start working on it. So, what’s in these recipes are either code snippets or full-bore Ragel definitions for various problems I’ve applied it to. In some cases, I’ll put up a recipe for no other reason than I want to easily refer to it later...it’s my Ragel notebook.
The Ragel Recipe Roadmap
1) Ragel Parser for NMEA GPS
The first Ragel Recipe is a full-on NMEA GPS parser I wrote - target: C language. The reason for targeting C in this case, is part of my roadmap for using Ragel - I want to use this parser on a small AVR microcontroller to parse the NMEA data stream and maintain current location data for my autonomous vehicle. This meant that the parser could not be huge like some other parser generators would emit and it had to be in C so I could build the firmware.
2) Ragel parsers for Linux log data
I spend a lot of time pawing through Linux logs - firewall, mail, messages, snort, and so on. I want to build some custom automation tools that can help me answer various, specific questions I have when I go into a log. “Why were there 3,424 messages in an hour?” “What were the preponderance of messages about?” Might possibly use some “R” to help with stats, but use Ragel to help dig and pull the components out of the sea of log data.
I suspect numerous recipes will simply be Ragel definitions, mini machines, for specific types of log messages. Nothing earth shattering, but over time, a small portfolio of Ragel state machines will build up here in the recipes. These might more aptly be described as Ragel “phrases” than “recipes.”
Recipes for parsing linux log data will almost all be targeting Ruby parsers as Ruby’s my first choice for system admin work on Linux (if I have a choice.)
3) Ragel parsers on an AVR
This is a huge thing for me. I really want to see how feasible it is to use a major parser generator like Ragel to solve the day-to-day parsing problems an embedded developer faces. Rather than hand-coding parsers every time I turn around, I would like to turn to a single tool, Ragel, that I know can cover the smallest targets like AVR as well as the hosts they talk to. This roadmap is designed to find the benefits and limitations of this approach. I’ll put Ragel to the test on an AVR and see how far I can get.
Conclusion
I’m not claiming to be an expert Ragel user, but just one who is willing to share his experience. These recipes are the result of “pressing on” with Ragel and I feel that I have been rewarded so far. I hope you will be encouraged to learn Ragel as you check back for new recipes and follow the progress.
Landon Cox
www.ESawdust.com
Full list of Ragel Recipes
asdfasdf
