RSS

Tag Archives: redis

In defense of dynamic languages

03 May

There are a good many truths and there are a better set of likelihoods. Given the current state of dynamic languages today they are less performance than static and functional languages, however, it is also true that dynamic languages are more productive than static and functional languages. (I am not talking about savants)

Don’t optimize your code at the first stage. First make it right, then (if necessary) make it fast (while keeping it right). –erlang programming rules

It is likely that regardless of the size of your project, the size or makeup of your team, or the breakthrough that you think the project represents… that your project is going to have average results at best. The Google’s, FaceBooks and Twitters of the world are extreme edge cases. As proof, look at the iPhone app store. There are over 600,000 apps and only a very small fraction of those apps have the following that Angry Birds does.

So before you go off in a corner reinventing the wheel in your favorite language consider this. WHat is going to be your return on investment? I cannot blame you for learning a new language or tool that would enhance your marketability or even just for hobby sake. But if your intent is to make some money and maybe a little independence they you really need to consider your ROI. And if you’re making money then rewriting your killer app in whatever killer fast programming language is available (and popular) will make make plenty of sense.

This is why I’m hot on python and python’s django, tornadoweb, flask; perl and perl’s mojolicious; ruby and ruby’s sinatra and rails; redis, sqlite, zeromq.

PS: While I’m not a fan of erlang, partly because of what it represents, I really like it’s Programming Rules and Conventions(PRC). By comparison python’s PEP-8 is amateurish. The PRC starts off with ideas like the one quoted above and giving you ideas on how best to approach the problem. This is like python’s PEP-20 but again it’s like signing your name with a crayon instead of a fountain pen.

 
Leave a comment

Posted by on 2012/05/03 in architecture, ProgLang

 

Tags: , , , , , , , , , , , ,

JSON and more JSON

10 Mar

A few articles ago I wrote about processing CDRs from an asterisk server. One of the design decisions I made was using very simple JSON messages as the message payload from the publisher to the subscriber. As most are familiar with JSON it’s a key/value container represented in UTF-2/ASCII. The format of the message is well documented and widely implemented.

There are a number of alternatives to JSON, like BSON, s-expr, msgpack and a few others. I settled on JSON because it was trivial to hand code a JSON payload without having to make any decisions and that it mapped to/from hash datatypes in python and ruby.

In the article I wrote the publisher in C and the subscriber in Ruby.

The publisher was an not an easy decision but by the time I committed to C I had a plan and it’s just one page of code for JSON, Redis and a but of parsing. (this code had to be fast and small for the reasons discussed in the article)

The subscriber was a different. It did not have to be all that fast… but in hindsight I’m thinking that the Ruby code worked as a proof of concept and now I want to reduce the memory footprint of the subscriber, however, as I look at the output of “top” I see that it has consumed only 54K of memory.  Not bad. I’ll have to watch it over time to see of there are any leaks.

All that aside, I was thinking that I would implement the subscriber in C. I knew that one of the complications of  JSON+C was that C did not have a native datatype that the JSON could be parsed into. So the code would have to work more like strtok().  I supposed that’s not much of a problem but it’s going to add code on my side to implement. Big deal- 20 or 30 lines of code.

But what was also interesting was the wide variety of implementation in the different libraries that the JSON project links to. Some of the projects were small and some were outrageously huge. I suppose there is more than one way to skin a cat. At least in this case while the intellection pursuit is interesting that’s about it. The Ruby solution was memory friendly for the moment.

 
Leave a comment

Posted by on 2012/03/10 in architecture, ProgLang

 

Tags: , ,

Reliable Asterisk CDRs

10 Mar

This is going to be a long and technical article pertaining to the capture of CDRs (call detail record) using the custom extension file and an AGI script.

I designed, built, deployed, and maintain a number of asterisk servers which are used as part of a VOIP arbitrage system.

The first generation system, which inherited, was a single system that housed the Asterisk server, database server for CDR and other billing and reporting data, and a PHP webapp that reported on the data. The system worked (a) when the volume was low; (b) when the overall amount of data was low. Needless to say I was brought in when the “system” (1) started hanging (2) losing call detail records (3) webapp could not return result before the browser would timeout. It was a mess.

The new system uses multiple systems (n+1). There can be a limited number of asterisk servers connected to a dashboard. The dashboard is where all the data is stored, where the ETL is performed, and where the reporting is initiated. The following design supports about 5000 channels on fairly moderate hardware; will auto-restart/recover if there is a crash; and will operate independently from the dashboard.

Here’s how it works. In the VOIP reporting business we live and die by the accuracy of the CDR. In VOIP there is no association equivalent of Visa or MasterCard that sets the rules or arbitrates the discrepancies. It’s always going to be “on you”. Therefore it’s always important to get the data from the switch. RDBMS people like to call the transaction ACID. Something very similar applies here.

In the basic Asterisk installation there are a number of ways to get the CDRs from the system. You can export them directly into flat files or directly into one of several brands of SQL databases like SQLite3. The problem with this approach is that the database is expensive in terms of resources and the flat file is inefficient because it’s one big file. This is additionally cumbersome when you’re trying to report and monitor in realtime.

My strategy is twofold. (1) Export the CDRs to a small flat file and change the flat file once a minute. (2) Then send the flat file to the dashboard server for processing. This is surprisingly efficient and it allows the system to continue to process calls if the dashboard is rebooting or in maintenance mode.

While approach has been wildly successful there is still some for improvement. The first improvement went live today.

Today’s challenge: When Asterisk receives an incoming call it authenticates the source and then tries to locate a route for the call (or the destination).  The routing of the call takes place in a file called extensions_custom.conf. In this file you’ll see some “code” that is more of a macro or script then an actual programming language. This macro tells asterisk when to do with the incoming call and at the end of the call to hangup. There are some other more complex functions like interactive voice prompts and voicemail but we’re just interested in routing. When the call completes we have to initiate a “hangup” and then we need to record the CDR.

So based on the approach above when the call was terminated (hangup) control would be passed to a 3rd layer script (through the AGI interface). This script could be written in any language and it would collect all of the data from the call and append the CDR to the flat file.

So let’s review:

  1. call initiated
  2. authenticate the call source
  3. check the extensions config for an appropriate route
  4. when the call is complete hangup
  5. send the CDR to a PHP script through the AGI interface

Step #5 looks like this:

exten => _X.,n,Hangup
exten => h,1,Set(CDR(userfield)=Hangupcause:${HANGUPCAUSE} Qos:${RTPAUDIOQOS})
exten => h, n, AGI(cdr_new.php, ${SIPCALLID}, ${CDR(dcontext)}, ${SUPPLIER},${CDR(start)}, ${CDR(duration)}, ${CDR(billsec)}, ${CDR(disposition)}, ${HANGUPCAUSE}, ${V_NETWORK}, ${CDR(lastapp)}, ${DEST})

The module that was replaced was “cdr_new.php”. The new module took the same parameters and was called “cdr_pub”.

The problem with the original PHP code was that it processed the incoming data and then created the target filename… and then opened the target for in order to append a record. It’s been working great but we are to a point where we might be losing some CDRs. (this is not definitive, just intuition) With 5000 channels running that means that there can be as many as 5000 instances of the routing process. That means when 5000 calls terminate at once there is a rush to append their CDRs. It’s simply not efficient for PHP to block when appending to the file. Not to mention that there is a lot of overhead for the PHP interpreter to load with each call completion.

The performance issues:

  • the latency to load php with each call completion
  • the possible deadlocks when more than one process tries to append to the same file at the same time. Blocking and resolution are not guaranteed.

The new plan. I rewrote the PHP script in C. Even with the few libraries I needed it’s not more than 20 or 30K. Since it’s native C it loads very fast. So this program gets all of the data from the AGI in the form of command line parameters and data in the STDIN. Then, instead of rushing to append the data to a file the small program sends or “publishes” the CDR to a redis pub/sub queue. There is a single, external, application that “subscribed” to the redis queue an when a message event arrives that external app will write the CDR to the flat file. Since there is only one external app appending to the flat file it cannot have the same problems.

One side note. If the publisher fails then the message event is posted to the syslog. And if the subscriber fails to append to the flat file then it also posts an event onto syslog. If something goes horribly wrong (with the exception of disk space) then we should have a chance to replay the calls in the dashboard by scrubbing the syslog file.

PS: once side note. This configuration also limits the number of simultaneous channels. Therefore if the CDR recording process blocks of any reason that will prevent the system from accepting the next call when the system is running at capacity for that source.

PS: the subscribe app was written in ruby. Installing ruby on my production asterisk server was not my first choice but it was worth it. The Ruby code was compact and it handled exceptions nicely. There were some idioms that I liked a little more than python. And while some of the development took place in Ruby 1.9.3 and the default version on the server was 1.8.7 I did have some challenges getting it to run and I needed to install some additional packages…… which as a side note confirms all of my previous beliefs about full stack awareness.

PS: One last note. When deciding on the publisher implementation and after abandoning C based on it’s lack of a JSON library that made sense I tried go and then considered java, other JVM-based and several dynamic languages… In the end C was the only choice because of it’s size, load latency and runtime.

 
5 Comments

Posted by on 2012/03/10 in nosql, ProgLang, VOIP

 

Tags: , , , , , ,

Lua is “good enough” development at it’s worst

08 Mar

I’m trying to rewrite my Asterisk CDR capture program from PHP to just about anything else. I inherited the PHP version of the program from a programmer who did not know anything about concurrency or performance. Inside the Asterisk configuration there is a file “extensions_custom.conf”.  This file contains all of the scripts for the individual PBX extensions. And when the call is completed the last thing the script does is capture the CDR.

The first thing that you might say is that I should use the built-in CDR recording. I would except for two important reasons.  My extension file contains many customizations so I need the CDR to reflect that information accurately. The second is that the built-in capture does not really address concurrency from the reporting/admin side of the equation. Regardless of the database or storage that info needs to be taken off-box so that it can be reported on… think map/reduce.

Back to the PHP code I have the following challenges: (a) it’s PHP and the interpreter needs to be loaded with each call completion. (b) the output is a textfile in a log directory. The file is a standard unix file with no concurrency mechanisms or file lock detection and avoidance. Needless to say the PHP version must go.

My first replacement was going to be built with golang. I wrote a separate article on that experience. In summary 3rd party libs FAIL.

My second choice was Lua. For one reason the redis team was using it for it’s scripting and so I wanted to learn it. The work was not hard but the documentation was not written like any language manual I had ever read previously. The best and worst part seems to be that the built-in functions are strictly limited. The idea being that 3rd parties would deliver the missing pieces.

At one time or another that was probably true, however, while trying to get the missing bits to work I found that the LuaForge project is practically dead. The individual projects are starting to scatter to the 4 corners.

As a side note it seems that a great many developers are still using Lua 5.1 even though Lua 5.2 has been available for about a month. The common response has been that not much had changed to warrant the change; although there were some compatibility issues.

In the end I simply do not understand the Lua ecosystem and since this is not that critical because I can code this project in C directly. I wanted to give Lua a fair chance after all the previous comments I’ve made. Basically Lua is still crap.

 
Leave a comment

Posted by on 2012/03/08 in architecture, Complaint Depatment, nosql, open source, ProgLang, Tools

 

Tags: , ,

golang 3rd party libs let me down

08 Mar

[UPDATE 2012-03-28] Google released version 1.

I’m working on a presentation project and one of the demo apps is a pub/sub that I need for a VOIP customer.

Currently the app is written in php. It’s executed when each phone call completes. And it currently writes directly to a log file. Both attributes are bad which I was hoping my demo app would correct. And while I’ve considered C I’d rather not for the moment.

golang has a fast and functional API and language. The target executable is statically linked and a reasonable size. My concern is that the beanstalkd and redis libs are old. So before I pushed my first message I’ve abandoned golang for lua.

 
1 Comment

Posted by on 2012/03/08 in open source, ProgLang, Tools

 

Tags: , ,

Redis EVAL() in 2.6.0

18 Jan

Our friends on the Redis commit team are proponents of the Ruby language when not coding in Lua, tcl or C. And so the EVAL() function example code is written Ruby. That’s all fine and well… but what about <my_lang>?.

So I spent all of 30 seconds on a python version of the same code. Chances are pretty good that the code will work. I do not know for certain because 2.6.0 is not ready yet and I’m not in a position to install unstable yet. Of course I could run it in userspace but that’s another topic.

import redis
r = redis.Redis()
RandomPushScript = """
 local i = tonumber(ARGV[1])
 while (i > 0) do
  res = redis.call('lpush',KEYS[1],math.random())
  i = i-1
 end
 return res
"""
r.delete('mylist')
print r.eval(RandomPushScript,1,'mylist',10)
# __END__

One of the crappy things about python is the indents. It makes copy/paste to a place like wordpress semi-functional.

 
2 Comments

Posted by on 2012/01/18 in beta

 

Tags: , ,

My New Python Project Setup

30 Dec

[update 2012-01-18] postgres has been updated to 9.1.2; the latest version as of today.

[update 2012-01-17] feel free to ignore my comments about Lua. While Lua might sit in an interesting place between Python and Java in an embedded/scripting place. The fact that lunatic-python does not compile and lupa depends on LuaJIT2 which is compatible with Lua 5.1 and the current Lua version 5.2 was recently released… and the comment from the LuaJIT team about adoption was a little snarky. I gotta think about something else.

[update 2011-12-29] I forgot to add twitter’s bootstrap CSS/JS. I’ll cover that in a future post when I also discuss modern-package-template

It’s pretty simple to set things up. There are some prerequisites and some basic install packages that need root access but the intent is to get the config in userspace as soon as possible. This article covers VM slices at Rackspace using Ubuntu 11.10.

First: Install and update:

  • allocate the OS
  • select the OS and wait for it to complete.  You’re going to receive an email with the root password
  • login, change the root password
  • create an “admin” privileged user (usually my name or “builder”)
  • add this user to the sudo
  • change it’s password
  • edit /etc/ssh/sshd_config and disable root login
  • update the package definitions (apt-get update)
  • upgrade the packages (apt-get upgrade)
  • reboot

Second: Install the required roo packages

  • postfix – when prompted select the default values
    • apt-get -y install postfix
  • apt-get -y install python-setuptools daemontools daemontools-run python-dev mailutils mutt build-essential uuid-dev python-nose vim htop sysstat dstat ifstat screen locate apache2-utils unzip siege python-virtualenv bwm-ng libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev fail2ban openssl libssl-doc openvpn libssl-dev libgcrypt11-dev lighttpd lighttpd-dev libevent-dev libcurl4-openssl-dev  libreadline6-dev beanstalkd tree
  • apt-get install postgresql-9.1 postgresql-client-9.1 postgresql-doc-9.1 postgresql-plperl-9.1 postgresql-plpython-9.1 postgresql-server-dev-9.1
  • easy_install pip
  • easy_install mercurial
  • easy_install pycurl
  • pip install virtualenvwrapper

That’s it, essentially, for the second layer, however, here’s an explanation of the modules from a macro perspective:

  • python-setuptools – make the installer, easy_install, available
  • daemontools daemontools-run – there are so many ways to implement a ‘daemon’ this tools make it simple to make daemon deployment simple
  • python-dev python-nose python-virtualenv – basic prereqs for python development. virtualenv is needed so that packages can be installed in userspace
  • mailutils mutt – generate emails
  • build-essential uuid-dev  - basic developer tools
  • vim screen – editor and console tool
  • htop sysstat dstat ifstat locate unzip bwm-ng – debug /monitoring tools
  • libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev – libs used when rendering usage graphics
  • fail2ban – detect login attempts and put the IP in time-out
  • openssl libssl-doc openvpn libssl-dev libgcrypt11-dev libcurl4-openssl-dev - crypto
  • lighttpd lighttpd-dev – web server that should be in front of the framework
  • libevent-dev – kevent, kpoll libs
  • apache2-utils siege – performance simulation tools
  • beanstalkd – message queue

And finally the third layer, the userspace framework layer. But before you start installing packages you need to create the virtual environment:

  • cd ${HOME}
  • mkdir -p src
  • cd ${HOME}/src
  • virtualenv currentenv
  • . ./currentenv/bin/activate
Now install the third layer.
  • pip install tornado
  • pip install flask
  • pip install flask-rest
  • easy_install pip
  • pip install pycurl
  • pip install simplejson
  • pip install tornado
  • pip install Fabric
  • pip install PasteDeploy
  • pip install PasteScript
  • pip install modern-package-template
  • pip install requests
  • pip install gevent
  • pip install pystache
  • pip install nose
  • pip install redis-py
  • pip install pymongo
  • pip install hoover
  • pip install pyzmq
  • pip install pyyaml
  • pip install beanstalkc
  • pip install django
  • pip install django-redis-cache
  • pip install clint
  • pip install djangorestframework
  • pip install pyparsing
  • pip install flup

I’m hoping that there is a practical use-case for embedding Lua in Python. There are an few interesting projects like lunatic-python and lupa. Normally I would not consider Lua for anything beyond “hello world”, however, the redis team is embedding Lua, it seems like a very lightweight codebase, it can be embedded in just about any language (do a google search).

  • cd ${HOME}
  • mkdir -p tmp
  • cd tmp
  • wget http://www.lua.org/ftp/lua-5.2.0.tar.gz
  • tar zxvf lua-5.2.0.tar.gz
  • cd lua-5.2.0
  • make linux
  • sudo make install

NOTE: the lunatic project does not compile under Lua 5.2. So this thread is postponed for now.

  • pip install lunatic-python

Alternatively I tried [lupa] but that requires LuaJIT 2.0 which is currently in beta (version 9)

  • cd ${HOME}
  • mkdir -p tmp
  • cd tmp
  • wget http://luajit.org/download/LuaJIT-2.0.0-beta9.tar.gz
  • tar zxvf LuaJIT-2.0.0-beta9.tar.gz
  • cd LuaJIT-2.0.0-beta9
  • make
  • sudo make install
  • sudo ldconfig

Then install lupa.

  • pip install lupa
NOTE: hoover is a client library for loggly.com.  You’ll need an account if you want to use this service.

In closing, I would like to include a few more libraries, however, the current version in apt-get is too old. I’d prefer installing them from scratch. They are necessary packages so for the time-being I’m just going to list them. They should be installed when installing the first layer and by the root user (or sudo)

  • ZeroMQ - trivial to build and deploy if you follow the instructions
    • cd /tmp
    • wget http://download.zeromq.org/zeromq-2.1.11.tar.gz
    • tar zxvf zeromq-2.1.11.tar.gz
    • ./configure
    • make
    • make install
    • ldconfig
  • MongoDB – (can actually be installed in userspace)
    • cd /tmp
    • wget http://fastdl.mongodb.org/osx/mongodb-linux-x86_64-2.0.2.tgz
    • mkdir -p ${HOME}/bin
    • cd ${HOME}/bin
    • tar zxvf mongodb-linux-x86_64-2.0.2.tgz
    • sudo mkdir -p /data/db
    • sudo chown `id -u` /data/db
    • ./mongodb-linux-x86_64-2.0.2/bin/mongod
    • … or …
    • cd ${HOME}/bin
    • find ./mongodb-linux-x86_64-2.0.2/bin/ -type f -exec ln -s {} \;
    • ./mongod
  • Redis – trivial to build and deploy if you follow the instructions
    • cd /tmp
    • wget http://redis.googlecode.com/files/redis-2.4.6.tar.gz
    • tar zxvf redis-2.4.6.tar.gz
    • cd redis-2.4.6
    • make
    • make install
  • SQLite – a simple SQL DB
    • cd /tmp
    • wget http://www.sqlite.org/sqlite-autoconf-3070900.tar.gz
    • tar zxvf sqlite-autoconf-3070900.tar.gz
    • cd sqlite-autoconf-3070900
    • ./configure
    • make
    • make install
    • ldconfig
  • ISO8583 – ISO8583 lib

Good luck!

PS: You should consider scripting this installation so that the deploy can be automated.  Specially via Fabris, chef, or puppet.

 
1 Comment

Posted by on 2011/12/30 in architecture, beta

 

Tags: , , , , , , , , , , , ,

OpenBSD 5.0 + Mojolicious + Redis + Beanstalkd

03 Nov

[update 2011-11-06] Recently I installed Mojolicious 2.22 without a hiccup.  Tonight I tried to install the latest 2.24 release.  The challenge tonight was that I needed to upgrade Test::Pod and Test::Pod::Coverage. I just guessed which modules were old based on the output from the build, however, it was 90% luck and 10% intuition. While I hate deep dependencies I wish they would have said something about this before leaving me to fend for myself. This is the sort of thing that make admin/DevOps work dangerous.

[update 2011-11-04] I’m trying to run my test program and I’m finding errors with module dependencies. I’m making the corrections inside the doc.  CRAP! One more thing I found. This time it was my fault. One of the Mojo guys asked me to check the clock/Timezone.  I thought I had. Crap!  I missed it. The clock was wrong and in fact it was 4 days behind. Since the Mojo files were technically 4 days in the future they would not build properly.  When I re-installed NTPD and corrected the clock… Miraculously it installed from CPAN.

[update 2011.11.03] looks like I managed to get everything to install. It’s not totally painless but it’s also not as involved as making code changes.

I really like OpenBSD (OBSD) and I’ve used it for years. The only comparison to other OS’ that I want to make is that while these guys are pushing development forward with new ideas they truly embrace the “Unix” way by making use that “the sum of the parts is bigger than the whole”. What I mean by that is that they build larger applications by incorporating specialized smaller ones. They are also more interested in security and correctness rather than security by constant change or obscurity.

Enough of that.

Armed with a current version of OBSD 5.0 installed in a working VMware instance… I wanted to install The tools I’d need for my next application idea. (skynet-pl). The installation of the primary framework tools was really simple.

Install redis and beanstalk proper (you’ll need some basic packages from the CD; not listed here)

cd /tmp
wget https://github.com/downloads/kr/beanstalkd/beanstalkd-1.4.6.tar.gz
tar zxvf beanstalkd-1.4.6.tar.gz
./configure
gmake
gmake install

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/redis-2.2.12.tgz
cd /tmp
wget http://redis.googlecode.com/files/redis-2.2.15.tar.gz
tar zxvf redis-2.2.15.tar.gz
cd redis-2.2.15
gmake
gmake install

(gmake test; is recommended, however, you need to install TCL first)

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/p5-Mojolicious-1.16.tgz
cd /tmp wget http://cpan.metacpan.org/authors/id/S/SR/SRI/Mojolicious-2.22.tar.gz tar zxvf Mojolicious-2.22.tar.gz cd Mojolicious-2.22 /usr/bin/perl Makefile.PL touch Makefile.PL Makefile gmake gmake install

Install the libs from the CPAN (cpan and redis should be running in the background before you try to install the packages.)

curl -L http://cpanmin.us | perl - --sudo App::cpanminus
sudo -s 'cpanm Test::Pod'
sudo -s 'cpanm Test::Pod::Coverage'
sudo -s 'cpanm Mojolicious'
sudo -s 'cpanm AnyEvent::Redis'
sudo -s 'cpanm AnyEvent::Beanstalk'
sudo -s 'cpanm EV'
sudo -s 'cpanm JSON'

Well, that installs everything, however, it is not current.  Redis and Mojolicious are still back level. I hope to resolve this shortly.

PS: I tried to force Mojolicious 2.22 to be installed with the ports collection. FAIL. I also tried to install the latest redis too.  FAIL. I suppose I should try “OpenBSD 5.0 + TornadoWeb + Redis + Beanstalkd” next, however, initial tests are not looking good.

 
6 Comments

Posted by on 2011/11/03 in web

 

Tags: , , , , , ,

Mojolicious plus Redis equals RestMQ-pl

02 Nov

I’m playing with Mojolicious. One of the fun projects I want to implement is a perl version of RestMQ. It was the main reason that I used then I requested a CPAN id. One of my previous complaints was, and remains, that the Python and Ruby versions of this project look elegant and fun. (this also means that a lot of the code is hidden in either the framework or the language; which I hate).

I’ll be posting and submitting this project eventually, however, in the meantime I had not been able to get things working properly. There was a timing issue with MojoX::Redis and Mojo. It was not working well at all. I read and read all of the docs over and over again. In fact I had abandoned MojoX::Redis altogether in order to try AnyEvent::Redis. (it also failed).

Finally I reread a page in minute detail. The first piece of example code looked more like a standalone snippet. Which did not make any sense in this context because it was supposed to be a MojoX::Redis implementation in Mojolicious. (The wiki has been wrong more than once before)

But then; there it was:

If you are planning to use MojoX::Redis in Mojolicious’ daemon mode, create MojoX::Redis instance with ioloop attr:

my $redis = MojoX::Redis->new(ioloop => Mojo::IOLoop->new);

Once I add this to my code and then added the stop and start calls to my function:

post '/q/:queue' => sub {
  my $self = shift;
  my $result = undef;
  my $queue  = $self->param('queue');
  my $value = $self->param('value');
  if (! defined $queue) {
    $self->app->log->debug('queue was not in the URL ('.$queue.')');
    $self->render_not_found;
  } else {
    my $uuid = undef;
    my $lkey = undef;
    my $q1   = $queue . $QUEUE_SUFFIX;
    my $q2   = $queue . $UUID_SUFFIX;
    $redis->incr($q2 => sub{ 
                             my ($redis, $res) = @_; 
                             $uuid = $res->[0]; 
                             $redis->ioloop->stop;
                            })->start;
    $self->app->log->debug('the uuid is ('.($uuid||'undefined').')');
    $lkey = $queue . ':' . $uuid;
    $redis->execute("sadd" => [$QUEUESET, $q1]);
    $redis->execute("set" => [$lkey, $value]);
    $redis->execute("lpush" => [$q1, $lkey]);
    $self->app->log->debug('the uuid q is ('.($q2||'undefined').')');
    $self->render(text => '{ok, ' . $lkey . '}');
  }
};

it worked great. I still do not think this code is sexy and it’s no where near the appearance of the Python or Ruby code(I should probably remove the debug statements). But it works… and it seems to be pretty fast too.

 
3 Comments

Posted by on 2011/11/02 in beta, web

 

Tags: , , , , ,

Mojolicious and MojoX::Redis

29 Oct

I’ve been looking at the code for MojoX::Redis for a couple of days now and I’m impressed and depressed at the same time.

First the good news. Like many projects it’s open source. The better news is that it looks cool. The code is nicely formatted and if anyone was following the PEP equivalent for perl then you’d say it was adhered to.

On the sad news side of things. While there is some POD doc at the end of the main project file that’s it. The code is not documented at all. And the worst of it is that the code is the exact reason why people hate this crap. This person clearly knows the ins and outs or perl and he demonstrated that aptitude well. But if you asked me to reverse engineer it… it’s going to take a while and a few cases of wine or beer.

The best feature is that it implements non-blocking in a way that complements Mojolicious, however, the first side effect is that the main thread continues to run while the first request is processing. When really the benefit of this sort of functionality is to let peer events run not the current main thread. Since it was not documented in any meaningful way this had to be experienced first hand… and after reviewing the test code I’m not sure that my conclusions from my code are correct.

Anyway, here is an explanation as I see it in pseudo code.

1) do some redis function like INCR expecting a response
2) do a get on the same key
3) compare the results and they will always fail because
the results from #1 have not completed by the time #2 completes.

Some code that demonstrates this

my $retval1 = undef;
$redis->execute("incr" => [$mykey] => sub{my ($redis,$res)=@_; $retval1=$res;});
my $retval2 = undef;
$redis->execute("get" => [$mykey] => sub{my ($redis,$res)=@_; $retval2=$res;});
die "they do not match" if $retval1 != $retval2;

The side effect here is that it simply does not work. The only way to make this work is something like this:

my $retval1 = undef;
my $retval2 = undef;
$redis->execute("incr" => [$mykey]
         => sub{
                my ($redis,$res)=@_;
                $retval1=$res;}
                $redis->execute("get" => [$mykey]
                      => sub{my ($redis,$res)=@_; $retval2=$res;});
              );
die "they do not match" if $retval1 != $retval2;

The effect in the above code is that since the sub() that is called upon completion of the incr() is called when the incr() is completed. The same for the subsequent call to the get(). The last die() will still have the same effect of getting control before the redis calls have completed execution. So fo for this to be effective the die() needs to be inside the sub() of the get(). Phew!

I looked at the test cases in MojoX::Redis and there were some interesting examples. There was an implementation of the redis pipeline in the form of a multi() transaction. This could be interesting since one could do an incr() and a get() in the same pipeline, however, if you needed the result in order to perform future calls then you’d have the same timing problems with the response not being ready or available in local memory for future calls.

An async lib of the redis tools seems novel but it makes certain use-cases very difficult and verbose. For example I was playing with the sinatra example of RestMQ. Sinatra being ruby has many of the same warts, and certainly the ruby version of the lib was not evented like MojoX::Redis so I do not expect that it’s going to get much work done. (I really like their demo version because it is so little code and it so accurately depicts the mission that it’s hard not to like the elegance. Even though it’s ruby.) But the reality is that it is still constrained.

In summary, while Mojolicious is nice and simple to use(I still like it). The simple use-cases are simple but as soon as you advance to the next step this will get tricky. If you track after the “get it to work correctly” and then think about performance you could end up rewriting the project to make it performant. So be mindful.

post '/q/:queue' => sub {
  my $self = shift;
  my $result = undef;
  my $queue  = $self->param('queue');
  my $value = $self->param('value');
  if (! defined $queue) {
    $self->app->log->debug('queue was not in the URL ('.$queue.')');
    $self->render_not_found;
  } else {
    my $uuid = undef;
    my $lkey = undef;
    my $q1   = $queue . $QUEUE_SUFFIX;
    my $q2   = $queue . $UUID_SUFFIX;
    $redis->execute("incr" => [$q2] => sub{
                                           my ($redis, $res) = @_;
                                           $uuid = $res->[0];
                                           $self->app->log->debug('the uuid is ('.($uuid||'undefined').')');
                                          });
    $lkey = $queue . ':' . $uuid;
    $redis->execute("sadd" => [$QUEUESET, $q1]);
    $redis->execute("set" => [$lkey, $value]);
    $redis->execute("lpush" => [$q1, $lkey]);
    $self->app->log->debug('the uuid q is ('.($q2||'undefined').')');
    $self->render(text => '{ok, ' . $lkey . '}');
  }
};

When this code executes… the following output is on the console:

rbucker@mvgw:~/hg/metaventures/gwtwo$ ./alt/restmq.pl  daemon
[Sun Oct 30 00:59:50 2011] [info] Server listening (http://*:3000)
Server available at http://127.0.0.1:3000.
[Sun Oct 30 00:59:51 2011] [debug] Your secret passphrase needs to be changed!!!
[Sun Oct 30 00:59:51 2011] [debug] POST /q/myqueue/ (Wget/1.12 (linux-gnu)).
[Sun Oct 30 00:59:51 2011] [debug] Dispatching callback.
Use of uninitialized value $uuid in concatenation (.) or string at ./alt/restmq.pl line 47.
[Sun Oct 30 00:59:51 2011] [debug] the uuid q is (myqueue:UUID)
[Sun Oct 30 00:59:51 2011] [debug] 200 OK (0.003678s, 271.887/s).
[Sun Oct 30 00:59:51 2011] [debug] the uuid is (19)

My observation is that the output from the sub() is in the log after the rest of the output. Also the error complaining about line 47 is because $uuid is currently undefined when the line executes. Therefore the callback is not merging the execution and therefore any sensible use requires that the code be nested. And that sucks.

 
Leave a comment

Posted by on 2011/10/29 in database, web

 

Tags: , ,

← Older posts
 
One Page Docs

Creating a library one page at a time.

One Page Bugs

Reducing the friction of writing and fixing bugs or features.

Follow

Get every new post delivered to your Inbox.

Join 223 other followers