RSS

Tag Archives: tornadoweb

In defense of dynamic languages

03 May

There are a good many truths and there are a better set of likelihoods. Given the current state of dynamic languages today they are less performance than static and functional languages, however, it is also true that dynamic languages are more productive than static and functional languages. (I am not talking about savants)

Don’t optimize your code at the first stage. First make it right, then (if necessary) make it fast (while keeping it right). –erlang programming rules

It is likely that regardless of the size of your project, the size or makeup of your team, or the breakthrough that you think the project represents… that your project is going to have average results at best. The Google’s, FaceBooks and Twitters of the world are extreme edge cases. As proof, look at the iPhone app store. There are over 600,000 apps and only a very small fraction of those apps have the following that Angry Birds does.

So before you go off in a corner reinventing the wheel in your favorite language consider this. WHat is going to be your return on investment? I cannot blame you for learning a new language or tool that would enhance your marketability or even just for hobby sake. But if your intent is to make some money and maybe a little independence they you really need to consider your ROI. And if you’re making money then rewriting your killer app in whatever killer fast programming language is available (and popular) will make make plenty of sense.

This is why I’m hot on python and python’s django, tornadoweb, flask; perl and perl’s mojolicious; ruby and ruby’s sinatra and rails; redis, sqlite, zeromq.

PS: While I’m not a fan of erlang, partly because of what it represents, I really like it’s Programming Rules and Conventions(PRC). By comparison python’s PEP-8 is amateurish. The PRC starts off with ideas like the one quoted above and giving you ideas on how best to approach the problem. This is like python’s PEP-20 but again it’s like signing your name with a crayon instead of a fountain pen.

 
Leave a comment

Posted by on 2012/05/03 in architecture, ProgLang

 

Tags: , , , , , , , , , , , ,

Your Next Web Application Framework?

12 Apr

Suppose you are the person who has to make the decision as to what language and framework your startup is going to use to deploy it’s application.  What would you choose? There are so many interesting and qualified frameworks that are already powering a good portion of the internet.

(You don’t have to know the language and framework but enough to argue what makes it idea.)

What would it be?

For example: I read an article several years ago that strongly recommended Erlang. At the time it was a great idea. The author suggested that using Erlang would attract smart people and keep the actual number of respondents to something manageable. Since then I implemented several applications that in hindsight: (a) impossible to attract new talent. (b) the more time that passes the more fragile the app gets because my detail recollection is fading (c) and it lacks common tools that would make allow generalized apps to give access to “operators” instead of programmers.

Other examples include: perl, ruby, python… mojolicious, rails and sinatra, flask, tornadoweb, and django.

I have my ideas… what are yours?

 
Leave a comment

Posted by on 2012/04/12 in architecture, future, web

 

Tags: , , , , , , , , , , ,

REST semi-realtime transactions

29 Jan

The freelance pattern implemented with TornadoWeb and ZeroMQ.

I recently implemented one of the broker reliable patterns as described by the ZeroMQ guide. It’s something very similar to beanstalkd’s but left to the reader to implement. This in itself is not a bad thing but it is more code to design, write and test; and had you the budget to hire these guys directly you would get the best broker money could buy. But how reliable is this model. Really?

I’m not a big fan of the broker model. It’s a lot of extra code to write for the broker itself. It’s also a single point of failure. And then there is the error handling as the client and worker negotiation the status of a transaction only to renegotiate it when the broker fails. And then there are all those places where transactions can queue up and all that code that is written that does not need to be. (the crux of this article)

In a brokerless model each client connects to each server (many to many) and in a traditional socket implementation that would not be possible. But it is with ZMQ. (read the guide). So a user app can connect to more than one server at a time and the client will “fan-out” the send() to the next server.

ctx = zmq.Context()
socket = zmq.Socket(ctx, zmq.REQ) 
socket.setsockopt(zmq.HWM, 1)
socket.connect('http://127.0.0.1:5555')
socket.connect('http://127.0.0.1:5556')
socket.connect('http://127.0.0.1:5557')
. . .
socket.send('a message for you')
socket.send('a message for you')
socket.send('a message for you')

What is going to happen here is that this code is going to send one message each to each of the servers assuming that there is an actual connection. Because the socket defines multiple endpoints. And it’s all very orderly and as expected.

The documentation talks about only round robin-ing active connections… sadly a call to connect() without a bind is still considered a valid connection and so this port would still receive a transaction but not actually send it to the server. Meaning that some transactions are going to be delayed. Just how long depends on the restart time for the downed server.

So on the upside… when everything is running smoothly, the transactions are going to be distributed nicely. Each server will be given some work to perform. The workers are still standard userspace applications that do not need any special threading or processing. Just bind to a socket endpoint and wait for incoming work. Do the work and send a response.

When things go wrong or when you might restart a server manually, that endpoint address is still in the client side. Should a transaction be headed that way and the connection had not been reestablished then that message will block until that port instance reconnects. If the server is running via daemontools then it should restart any second. The transaction in the queue will be scooped up and procession will resume. The number of transactions queued per connection depends on the high water mark setting.

I say ’1′ transaction in the queue because we set the HWM (high water mark) when creating the connections. This is probably a good setting for realtime systems where losing transaction in an invisible queue is the least desirable event. You might also be able to add NOBLOCK on the send() function to get some other actionable events. It really depends on the applications tolerances.

At first I did not like the idea of losing the transaction(s) but I’m warming to the idea that the codebase will be smaller and possibly more reliable overall.

 
Leave a comment

Posted by on 2012/01/29 in architecture, Tools

 

Tags: , ,

webapp application stack

19 Dec

I’m trying not to be a “bitter betty” over the time spent on Mojolicious so I’ve been refreshing my stack in order to prepare for future endeavors. In the meantime the application I’ve been building for a client required that the GUI be split into an API and a GUI rather than just an integrated GUI. At this point I have completed the API development and it’s as extensible and scalable as I could hope for. Now it’s time for the GUI stack.

There are several missions here. a) python; b) tornadoweb; c) keep in mind the person making changes to this app will not likely be a pythonista (thank goodness). That’s it. Now I need to pick the rest of the tools. If there is some interest on your part then do some googling. They are easy to find.

  • PyCharm IDE – not all of the future developers need or want to understand everything from the command line.
  • virtualenv – handle different versions of python.
  • easy_install and pip – needed for installing the different dependencies, however, distutils might be best for deploying your app
  • pycurl – dep
  • simplejson – dep (at some point it was incorporated directly into python)
  • tornadoweb – web framework
  • fabric - repetitive CLI tool
  • modern-package-template – python project folder structure
  • libevent -
  • requests – simple http client APIs
  • gevent – needed by requests
  • pystache (from mustache) – templating without logic of any kind – I like it because it’s supported by so many platforms including JavaScript for client side and perl too. The best part is that any mustache file can be reused.
  • nose – unit testing.
  • daemontools – daemonize any userspace app.
  • redis – but I’m not actually accessing redis directly although there is some room for cacheing.

and my GUI elements:

  • bootstrap – this cooked Less / CSS is great. The best part is the best practices but the files are easy enough to spoil if necessary.
  • and it’s dependencies

It’s a pretty simple setup from here and creating your first project is easy too. After some experimentation you’ll figure out the order that works best for you.

 
Leave a comment

Posted by on 2011/12/19 in architecture, web

 

Tags: , , , , , , , , , , , , , , ,

“Mojolicious” has lost it’s Mojo

11 Dec

I’ve recently encountered my second issue with Mojolicious; since the Mojo team had recently added some core developers and reinstated their ticket system via GitHub I decided to open a ticket.

The first issue I presented to the team was their use of “daemon” as a command line param to launch your webapp, however, it unexpectedly ran the app in the foreground. My understanding of “daemon” and the behavior of any number of apps is that “-d” or daemon-mode meant background execution.

The issue I am currently working on is the new 2.37 version. It’s the first build from the new core team and that was a good thing, however, it would not build on my system (OpenBSD 5.0). So I reported it. The response that I received directed me to the CPAN team who verified that it worked.

Less obvious was the notion that I needed to check my dependencies. So I suggested exactly that… to which I received a reply that a) we were off topic and b) that there are no dependencies.

Well, ok… we were a little off topic… but there is such a thing as customer service. And while the cpan install file may not actually set the dependencies… they are listed on the CPAN page. So why not add that to the install?

So I’m up to my eyeballs in this. I’ve spent the last few months (calendar time) trying to wrap my head around this thing called Mojolicious. I happen to like perl and the CPAN. But I’m also frustrated with my two encounters with the Mojolicious team. They are neither smart enough or pretty enough to warrant this sort of snarkey-ness,

In the last 24-hours I’ve spent some time looking at flask, Werkzeug, Cyclone, and of course I already use Django and TornadoWeb. I also like python. The dependencies are easier here too.

 
1 Comment

Posted by on 2011/12/11 in web

 

Tags: , , , ,

Mojolicious or TornadoWeb?

26 Nov

It depends on what you are really trying to accomplish.  In a current project I’m using TornadoWeb to get the transaction through the ZeroMQ broker to a worker. (standard request/response broker implementation).

Since I’m using TornadoWeb I needed to implement about 50 lines of glue code and I needed to hijack TornadoWeb’s IOLoop instance with the IOLoop from ZeroMQ. And since the work is only a few dozen lines of code … it’s all up for debate.

In retrospect I think I should have gone with Perl+Mojolicious+beanstalk. It would have completely eliminated my own broker .. another 100 lines of code. Then there was the direct access to redis in tornadoweb.

MEH! While I’m pragmatic that if I did not have to wtote an extra 150 lines of code then I would nto have to debug them either.

This race between the two platforms is so close! (on any given Sunday)

 
Leave a comment

Posted by on 2011/11/26 in web

 

Tags: ,

Django-App-Engine is not Django

24 Nov

I’m not going to give these guys any more press than they deserve.  I tweeted that “django-app-engine development halted”. As I tweeted the message I thought they were referring to the DjangoProject, however, I was wrong! Seems the Django project is still going strong, I don’t know who these other guys are or what their endgame is…

Anyway, Django is a strong platform. I do not use it much these days because I’m focusing on TornadoWeb and Mojolicious but it is rock solid. It’s admin tools can bootstrap just about app you might consider building.

 
Leave a comment

Posted by on 2011/11/24 in web

 

Tags: , ,

Getting Mojolicious and AnyEvent::Beanstalkd to work

18 Nov

I have been having a hard time getting Mojolicious and AnyEvent::Beanstalkd to integrate. Given the commonality of AnyEvent one would think that it would a) work off the shelf; and b) there would plenty of documentation or examples. Sadly neither are true and I had to test several permutations, give up, and then try one last method (which worked).

It basically comes down to this. $cv->recv() does not work inside a Mojolicious. It causes a recursion error that AnyEvent just barely explains. The first alternative, that worked, was a matter of nesting all of the callback functions and as long as the stack is shallow that would work but it’s not pretty and it’s certainly unmaintainable.

In the end the code was replaced with $txn->cb(sub { $_[0]->recv }); and then the beanstalkd commands were flattened; and everything seems to be working nicely now. There was no explanation as to why this worked.

After looking at the AnyEvent code I think the code was able to “falsely” detect the $cv->revc() as a recursive call where the other format is undetectable.  I’m not certain whether this is good or bad but it works.

 . . .
 my $txn = AnyEvent::condvar;
 $txn->begin;
 $client->use('restmq', sub {$txn->end;});
 $txn->begin;
 $client->watch($guid, sub {$txn->end;});
 $txn->cb(sub { $_[0]->recv });

 $txn = AnyEvent::condvar;
 $txn->begin;
 my $job2 = $client->put( $msg, sub {$txn->end;});
 $txn->begin;
 $client->reserve( sub { my $job = shift;
   my $decode = $job->decode;
   my $uuid = (@$decode)[0][0]->{'uuid'};
   my $retval = sprintf "data> %s, %s\n", $job->data, $uuid;
   $client->delete($job->id);
   $client->ignore($guid);
   $client->quit;
   $self->render(text => $retval);
   $txn->end;
 });
 $txn->cb(sub { $_[0]->recv });
 $self->render_later;

The complete code is here. I’ll be posting my scratch code shortly and I’ll be updating and commenting the code too.

 
Leave a comment

Posted by on 2011/11/18 in web

 

Tags: , , ,

Benchmark – Hello World – TornadoWeb vs Mojolicious

17 Nov

I was running a simple hello world benchmark. It was meant to be as simple as possible. Mojo clearly wins the LOC comparison but TornadoWeb wins the race.  Mojo produced 20TPS and TW produced 22TPS. I used the basic run commands and I used siege from JoeDogs.

Mojo:

#!/usr/bin/env perl
use Mojolicious::Lite;
# Simple delayed rendering
get '/' => sub {
 my $self = shift;
 $self->render_text('3 seconds delayed!');
};
app->start;

TornadoWeb:

 
#!/usr/bin/env python
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
 def get(self):
 self.write("Hello, world")
application = tornado.web.Application([
 (r"/", MainHandler),
])
if __name__ == "__main__":
 application.listen(8888)
 tornado.ioloop.IOLoop.instance().start()
 
Leave a comment

Posted by on 2011/11/17 in beta, web

 

Tags: , ,

OpenBSD 5.0 + Mojolicious + Redis + Beanstalkd

03 Nov

[update 2011-11-06] Recently I installed Mojolicious 2.22 without a hiccup.  Tonight I tried to install the latest 2.24 release.  The challenge tonight was that I needed to upgrade Test::Pod and Test::Pod::Coverage. I just guessed which modules were old based on the output from the build, however, it was 90% luck and 10% intuition. While I hate deep dependencies I wish they would have said something about this before leaving me to fend for myself. This is the sort of thing that make admin/DevOps work dangerous.

[update 2011-11-04] I’m trying to run my test program and I’m finding errors with module dependencies. I’m making the corrections inside the doc.  CRAP! One more thing I found. This time it was my fault. One of the Mojo guys asked me to check the clock/Timezone.  I thought I had. Crap!  I missed it. The clock was wrong and in fact it was 4 days behind. Since the Mojo files were technically 4 days in the future they would not build properly.  When I re-installed NTPD and corrected the clock… Miraculously it installed from CPAN.

[update 2011.11.03] looks like I managed to get everything to install. It’s not totally painless but it’s also not as involved as making code changes.

I really like OpenBSD (OBSD) and I’ve used it for years. The only comparison to other OS’ that I want to make is that while these guys are pushing development forward with new ideas they truly embrace the “Unix” way by making use that “the sum of the parts is bigger than the whole”. What I mean by that is that they build larger applications by incorporating specialized smaller ones. They are also more interested in security and correctness rather than security by constant change or obscurity.

Enough of that.

Armed with a current version of OBSD 5.0 installed in a working VMware instance… I wanted to install The tools I’d need for my next application idea. (skynet-pl). The installation of the primary framework tools was really simple.

Install redis and beanstalk proper (you’ll need some basic packages from the CD; not listed here)

cd /tmp
wget https://github.com/downloads/kr/beanstalkd/beanstalkd-1.4.6.tar.gz
tar zxvf beanstalkd-1.4.6.tar.gz
./configure
gmake
gmake install

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/redis-2.2.12.tgz
cd /tmp
wget http://redis.googlecode.com/files/redis-2.2.15.tar.gz
tar zxvf redis-2.2.15.tar.gz
cd redis-2.2.15
gmake
gmake install

(gmake test; is recommended, however, you need to install TCL first)

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/p5-Mojolicious-1.16.tgz
cd /tmp wget http://cpan.metacpan.org/authors/id/S/SR/SRI/Mojolicious-2.22.tar.gz tar zxvf Mojolicious-2.22.tar.gz cd Mojolicious-2.22 /usr/bin/perl Makefile.PL touch Makefile.PL Makefile gmake gmake install

Install the libs from the CPAN (cpan and redis should be running in the background before you try to install the packages.)

curl -L http://cpanmin.us | perl - --sudo App::cpanminus
sudo -s 'cpanm Test::Pod'
sudo -s 'cpanm Test::Pod::Coverage'
sudo -s 'cpanm Mojolicious'
sudo -s 'cpanm AnyEvent::Redis'
sudo -s 'cpanm AnyEvent::Beanstalk'
sudo -s 'cpanm EV'
sudo -s 'cpanm JSON'

Well, that installs everything, however, it is not current.  Redis and Mojolicious are still back level. I hope to resolve this shortly.

PS: I tried to force Mojolicious 2.22 to be installed with the ports collection. FAIL. I also tried to install the latest redis too.  FAIL. I suppose I should try “OpenBSD 5.0 + TornadoWeb + Redis + Beanstalkd” next, however, initial tests are not looking good.

 
6 Comments

Posted by on 2011/11/03 in web

 

Tags: , , , , , ,

← Older posts
 
One Page Docs

Creating a library one page at a time.

One Page Bugs

Reducing the friction of writing and fixing bugs or features.

Follow

Get every new post delivered to your Inbox.

Join 223 other followers