RSS

Monthly Archives: September 2011

Agile is still dead and has been since 1991

30 Sep

[updated 2011.09.30] yet another response to Agile is good.

When you have so much of you career invested in something like Agile, XP etc… it can be hard to see the forest for the trees. I had a consulting job in The Haag many years ago. IBM was the incumbent contractor at the customer site (a bank) but after 5 years on the job they had not written a single line of functioning code. In the office there were two teams of software people… both behind closed doors. The first team was the Data team and the second team was Functional. They rarely spoke and they never shared information. I was there for a week, introduced the client to OO and we had a functioning prototype. Smart people do smart things, You cannot make an underachiever exceptional by using Agile. Either they get “it” or they don’t.

I just commented on a blog. I’m sure there is some validity to his post beyond observing that Agile Scrum is broken. It certainly is not what it was originally intended but for that you have to go way back to Dave and Andy from PragProg. I have not found the original links and references myself but I recall enough from my reading at the time. Today’s Agile does not look anything like what it was.

Scrum deflects from individual accountability. But the failure of agile/scrum is probably more psychology than technology. There are essentially three groups of people. 1) the high achievers that you want everyone to emulate; 2) the average devs; 3) and the low achievers. The high achievers hate these processes because it simply adds friction to their day. The low achievers love it because it deflects individual responsibility (think Survivor or Big Brother; the best do not always win. Floater is a legitimate strategy). And the average achiever is ambivalent and can be tipped either way.

Everyone is different and creative in their own way. You cannot herd cats with agile or scrum.

Agile does not treat people with respect. It dictates with strict rules what and how things are to be done. When the reality is that we have individual and collective responsibility. And most of that is encapsulated in an unwritten “bill of rights” that we carry around with us. At least the high achievers do. Translating that BOR to everyone else with a wide brush is simply too general an approach. Improvement from the under achievers comes from training, education, and more then anything else experience and experience from making mistakes.

The real chalenge here is that businesses are trying to grow their ranks as fast as they can. Many times that means hiring people that they would not hire if there was not such a shortage. This is a different set of problems from Agile and Scrum. This problem can be fixed by being selective, selecting people with aptitude for the work, selecting tools that lower the bar of entry, and managing people rather then allowing them to manage themselves.

 
8 Comments

Posted by on 2011/09/30 in business, management

 

Tags: ,

Still managed a Mojo upgrade

30 Sep

Even in my half inebriated state I managed to upgrade my Mojolicious development installation from version 1.98 to 1.99. I cannot wait to restart my latest perl project… with only 2 parts written so far.

$ sudo -s ‘cpanm Mojolicious’

CPAN/cpanminus might be the second killer app. (gems and peaks simply do not measure up)

 
Leave a comment

Posted by on 2011/09/30 in ProgLang, updates

 

Tags: ,

What will be the effect of internet memory?

23 Sep

I thought I would have a deep philosophical viewpoint but after walking the dog and considering the obvious it’s not that difficult. The challenge is that the generations that follow me already have a more open viewpoint of the online social experience. They are no less opinionated, fearful, or cliquish than my generation but their memories are much shorter.

I’m not sure if that is because my-gen stores it’s knowledge base in local or near “cache” and the next-gen uses google.

So what is going to happen you go to take that all important career making interview and the interviewer pulls out a stack of blog posts or pictures that demonstrate questionable moral character or outright disdain for everything lawful and good.

Will it be easy enough to say “I’ve changed my opinion since then” or “I was just acting out childhood angst” or my new favorite… “I was publishing without the filter of a professional editor“? We are being judged all the time.. Whether it’s because we contributed to a project code tree without checking in comments or PEP-8, submitted a bug report with poor grammer that was fixed anyway, or wrote openly about a crush on a teacher or professor. (these are not my personal experiences).

But what is to come of it all? Specially if you were interviewing for a local fast food establishment or secretary of state. It seems that it might be on a sliding scale.

So as you read my articles… I’m not trying too hard to express too strong of an opinion that I cannot change my mind as I often do. I won’t be slamming “the law” as they keep me safe and that job is for the regular press and Assange.

The internet memory is long and potentially unforgiving… and always subjective and subject to interpretation. Good luck to us all.

 
1 Comment

Posted by on 2011/09/23 in web

 

Tags:

Message Queues and nothing but Message Queues

21 Sep

[Update 2011.09.21] The ink had hardly dried on this post when I decided to quickly evaluate gearman. It’s immature and direction internally seems dizzy. So pass on this one too.

I’m reading up on MQs again while I’m waiting for a conference call. I do not want to disrupt my code before the demo (rule #1 of demo-club). So I’m making yet another list of all of the MQs out there.

  • ActiveMQ
  • RabbitMQ
  • Amazon SQS
  • Google Queue/Task
  • Gearman
  • ZeroMQ
  • Sparrow
  • Starling
  • Kestrel
  • RestMQ
  • Oracle Advanced MQ
  • IBM Websphere MQ
  • MicrosoftMQ
  • JBoss Messaging
  • Sun Open Message Queue
  • Apache Qpid

There are several ways to compare these MQs. Core source language, client language support, implementation detail, inspiration or initial design, current activity, licensing, cost, deploy OS, performance/TPS, use-cases, dependencies, deep dependencies, mindshare.

So here is that list again.

  • ActiveMQ – apache and meant for Java JMS although there are libs for other languages it really depends on the actual message payload.
  • RabbitMQ – build with erlang, while it works well and is the MQ for ejabberd it’s pretty heavy weight.
  • Amazon SQS – it’s not free or at least cost effective. They have had huge outages. For security reasons this has to be onsite.
  • Google Queue/Task - For security reasons this has to be onsite and only supports java, python and possibly go.
  • Gearman – There is potential here. It seems to have been rewritten in C from perl. It is open source and free to use. They appear to be active and there is a CLI making access easy. This is worth looking into. (the only downside is that it’s written in Java)
  • ZeroMQ – 0mq leaves a lot to the developer or architect to decide and fill in the blanks. My original implementation used beanstalkd and it was razor fast and trivial to implement. 0mq has plenty of gotchas and the burden is on the developer. I’d use it again but it’s not going to be #1.
  • Sparrow - ruby? This is just not going to make the list.
  • Starling – another ruby implementation.
  • Kestrel – argh… this one is implemented in scala. So you get all of the non-functional java components (unverified) and long namespaces.
  • RestMQ – I like the pythonic implementation, however, while twisted is a solid project it appears to be conflicting with my main project based on tornadoweb.
  • Oracle Advanced MQ – not free
  • IBM Websphere MQ – not free
  • MicrosoftMQ – not free
  • JBoss Messaging – too many dependencies. This is akin to J2EE’s JMS. And for that I might as well use ActiveMQ.
  • Sun Open Message Queue – They do not live here any more.
  • Apache Qpid – supports AMPQ and trying to get 100% compliant. I suppose this is interesting in that disparate systems will be able to communicate. Looking at the source tree there is just so much code and since I do not want to dedicate this kind of time to it… I think it’s an easy PASS.
  • beanstalkd – The last release was over a year ago. I’ve posted on their group hoping to get some sort of answer. There has been some activity on their website. The APIs have a TTR novelty API that I like. Specially in my fire and delete application.

One of the novel things about the apache team is that when they acquire a technology it’s long before there is a wrapper around it. For example Apache Camel.

So here is the state of things.

  1. RestMQ – I’m going to continue my attempt to rewrite RestMQ in perl using Mojolicious. If I get some decent results then I’ll publish them/it.
  2. Gearman – deserves some immediate investigation. I’m hoping to make it through the painful documentation.
  3. Beanstalkd – while it has not had much recent action I’m hoping to see some info in my inbox shortly. I like their broker and while they do not specifically support req/resp it’s not really needed and can worked around.
  4. ZeroMQ is just an API. My broker is getting more and more complicated. Message routing is also getting complicated.
 
2 Comments

Posted by on 2011/09/21 in Tools, web

 

Tags: , , , , , , , , , , ,

pydoc – the anti-killer appp

21 Sep

Some time ago I wrote about perldoc being perl’s killer app. I still think that’s true. Recently I’ve been going through my python code and attempting to add useful comments that could be picked up by some auto documenting mechanism. Clearly they exist… python.org is full of doc and I’m certain it’s rendered.

Included with python, since 2.1, is pydoc. At first glance it appears to be an analog to perldoc; and for the most part it isn’t.

  • pydoc will generate text or html – perldoc will convert the poddoc to html, latex, man, text
  • pydoc has a graphic UI embedded – but it requires TK (and x-windows)
  • pydoc has an embedded http server – could leak important data if the pydoc is internal use only.
  • perldoc generates static files – pydoc generates dynamic files
  • pydoc recurses the class dependency tree – if the stack is deep the document is long (there is a lesson in this)

In a way, it’s nice that pydoc scans your code looking for your comments in your classes etc. The format makes sense and it’s not all that terrible. There are even ways to comment individual functions instead of just one monster perldoc (as typically demonstrated)

So perldoc is still the killer app.

ref: perlpod – perldoc.perl.org http://bit.ly/nNhp58

 
Leave a comment

Posted by on 2011/09/21 in ProgLang, Tools

 

Tags: , , , ,

mojolicious – first app (part 2)

21 Sep

I was going to call it a night but I just wanted to make a little more progress on this first app. I’d like to think I’m tired and cranky because I’m not seeing it. Mojolicious generated the following as my first app:

#!/usr/bin/env perl
use Mojolicious::Lite;

# Documentation browser under "/perldoc" (this plugin requires Perl 5.10)
plugin 'PODRenderer';

get '/welcome' => sub {
  my $self = shift;
  $self->render('index');
};

app->start;
__DATA__

@@ index.html.ep
% layout 'default';
% title 'Welcome';
Welcome to Mojolicious!

@@ layouts/default.html.ep
<!doctype html><html>
  <head><title><%= title %></title></head>
  <body><%= content %></body>
</html>

That’s everything. It’s their version of hello world. The app started right away, however, it was my fault.  I pointed my browser to http://127.0.0.1:3000// and the page that popped up was some sort of 404 error page. It was nice enough… but much larger than my screen so I had to scroll to see everything.

Which is when I realized that I was not seeing my project. After a quick code review I changed my browser to load http://127.0.0.1:3000//welcome and my welcome page arrived. However since there was no default page (aka index) or default redirect. I started looking.

Skimming the documentation I found something that looked like:

get '/*' => sub {
   . . .
};

This had promise as I added a redirect.

get '/*' => sub {
  my $self = shift;
  $self->redirect_to('/welcome');
};

My only problem is/was that it would not recognize or redirect http://127.0.0.1:3000/. I had to read a lot more doc and make a few assumptions as to what was going on. The mojo docs indicate that the pattern could have been a regex, however, their pattern matching is more efficient just because it is and based on the way most people use it. And in their pattern matcher ‘/*’ indicates that they are greedy matching placeholders and not a pattern per say. (I think that the * indicates that there has to be at least one byte/char)

The strange thing is that ‘/’ and ‘/*’ are not the same string and both will not match the empty path. Only ‘/’ will.

So in my app I have a ‘/welcome’ in order to say hello. A ‘/*’ to get all of the non-empty paths so that I can redirect them to ‘/welcome’ and I have a ‘/’ in order to catch the empty path. What a pain in the ass! At least I only have a few pages.

ELAPSED TIME: 30min

 
Leave a comment

Posted by on 2011/09/21 in nosql, web

 

Tags: , ,

mojolicious – first app (part 1)

21 Sep

[Update] I should have mentioned that the elapsed time was only about 15 min; maybe less.

I’m a perl programmer from way back and while it has been 2 years since I’ve written any perl of consequence I’ve decided that my client’s application was going to be implemented in perl… just for the fun of it. (I’ve been writing a lot of python+tornadoweb+redis+mongodb and I’m very comfortable in this space. Mojolicious is a half step outside my comfort zone because I really do not want to rewrite this app in python if I don’t have to. I’m a big believer in DRY)

I’m certain that everything is installed. I used the same installation for documentation’s sake recently:

wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.0.tgz
# need to untar and then relocate the files to the path

# MOJOLICIOUS - (as root)
curl -L cpanmin.us | perl - App:cpanminus
curl -L cpanmin.us | perl - Mojolicious
curl -L cpanmin.us | perl - Redis
curl -L cpanmin.us | perl - ZeroMQ
curl -L cpanmin.us | perl - JSON
curl -L cpanmin.us | perl - MongoDB

And we should be ready for the first app.

The application folder structure is going to look like this:

.
|-- COPYRIGHT
|-- INSTALL
|-- LICENSE
|-- README
|-- bin
|-- data
|-- doc
|-- src
`-- test

It’s far from perfect and it’s missing the Mojolicious application (webapp). So let’s create a mojo app (adapted from the docs):

Nope, not yet. First you have to know whether you are going to create a lite_app or a full-blown app.

mojo generate lite_app

or

mojo generate app

The difference is quite extensive. The second selection is probably best suited for anything more that a a few dynamic pages. And the lite app seems suited for all-in-one file deployment. Right now I’m not sure which is best for this design problem, however, I know that I can convert later; I can cut-paste everything and with any luck I won’t be repeating myself.

So before I make that decision. What does my application look like.

  • I need a login screen and session state so that the data is private for my client. One user is sufficient.
  • I need to upload a CSV file to a file system and queue that file for processing.
  • I need a background process that looks for queued files and then processes them.
  • The process is as simple as a GET against a URL, storing the result in another CSV and then getting the next record from the input file.
  • Once all of the records have been processed the second file is converted to a PDF of mailing labels.
  • The PDF can be downloaded from inventory.
  • and finally the user should be able to delete the PDF, the original CSV and the intermediate CSV.

So that’s pretty simple. I need a few pages:

  • login
  • error
  • queue status
  • upload
  • are you sure (popup??)
  • completed (optional, could always be inline)

So for the time being lite_app it is. This is what I did:

mkdir webapp
cd webapp
mojo generate lite_app lisapp.pl

Notice that I specified my application name (lisapp.pl). Starting the app was a breeze eve though it bugs me that the mojo team misused daemon and does not even want to be open minded about it. Oh well, it’s a good platform in spite of ‘daemon’. And then I noticed the output:

$ ./lisapp.pl daemon
[Wed Sep 21 00:13:47 2011] [info] Server listening (http://*:3000)
Server available at http://127.0.0.1:3000.

Notice that the output is stating that it’s listening on ‘*’ and 127.0.0.1.  What’s up with that? When I did a ‘netstat -ln’ I only had one instance of port 3000.

tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN

I’m not certain I understand the motives here. But I have to say that it drop be nuts because I knew that I had to test on 0.0.0.0 since my server is at Rackspace. And at first glance I only saw the 127.0.0.1. So it’s on me to pay attention and on them to be more clear.

When I pointed my browser to the URL it popped. As I expected since this is a very lightweight app. So for now I will take a break. Next time I will add the login screen. It’s part of the 3rd screencast and pretty simple. I also need to read-up on the under command. It seems to be where all the power comes from. Keep in mind that the home page is going to be a login screen. No data should leak out. Everyone logs in.

 
1 Comment

Posted by on 2011/09/21 in database, nosql, web

 

Tags: , , , , , , ,

Everything is in the cloud – it’s all damage control now

16 Sep

There was a time when I was really concerned about what information was “out there”. Then I tried to control that information. And finally I realized it was hopeless. Now it’s just a matter of trying to keep it clean and ongoing damage control. Not that I’ve done anything wrong or embarrassing in my life…

The fact remains that the likes of google, yahoo, “social media”, advertisers, and so are… while they might really want my demographic information, spending habits, income, neighbors, surfing habits, they will accept my anonymous information just the same. And I give them plenty of both.

The way it works… you visit a website. The site owner has contracted with someone like google for some sort of service. Whether it’s advertising, sales or just some general tracking information it’s not important. That service drops a cookie on your browser. Later, you visit another site. That site contracts with the same service provider… and looks for a cookie that they might have dropped previously. When they find it they can connect the dots.

This sort of function is everywhere. It’s in everything we see and do. From the transponders in our cars, cell phones, the GPS in our cameras. It’s everywhere.

So why not put everything in the cloud? What is it that I think I can hide? I’m a lawful person and so I’m not going to advertise that I went through a red light, but some day there is going to be an intersection device that is going to register the infractions… made by the car, not necessarily the person.

I do not care that I use GMail and Google Docs. That all of my music is in iTunes match/iCloud. That I backup everything to Dropbox and CrashPlan. The information privacy war is over. We lost. Now all we can do is damage control.

 
1 Comment

Posted by on 2011/09/16 in security, web

 

Tags:

How does [mongoDB] concurrency work

15 Sep

This is a short post to call attention to this [mongoDB concurrency]. I do not doubt, for a second, that concurrency is difficult. It’s probably very hard. Part of which depends on the overall architecture, resultset size, record version collision, replication, sharding, and conflict resolution; among others.

I do not have a particular concern other than it is an important problem and that it’s going to effect write performance and potentially some reads. (This is very similar to the GIL problem in Python). There is a ticket on their system that references this issue and plans to resolve the issue by pushing the lock to the collection instead of the DB. While this is more granular I’m not sure it resolves the real issue with a workable solution. (And I do not have a formal recommendation)

It’s not clear to me whether this problem references a lock across all shards or just the one shard. Therefore, could the problem be alleviated by adding shards? (doubt that)

I can say this, however, most RDBMS servers have addressed this problem by setting the locking level via a pragma and that it is tunable. In order to know whether this is an issue for you; you will need to know the distribution of calls [read, insert, update and delete] and the TPS rates too.

I’m less convinced that the day of the RDBMS is over. (Recently PostgreSQL 9.1 was released)

 
Leave a comment

Posted by on 2011/09/15 in database, nosql

 

Tags: , , ,

Perl RestMQ

15 Sep

I’m starting a Perl version of the python tool restmq. Restmq is really cool but my only complaint is that it uses cyclone; the tornadoweb server lookalike written for twisted.

Sure, I’d be justified porting it to tornadoweb but after spending time with mojolicious it seems like a better fit and more fun.

Contributors are welcome: http://github.com/rbucker881/perrestmq

 
Leave a comment

Posted by on 2011/09/15 in open source

 

Tags: , ,

← Older posts
 
One Page Docs

Creating a library one page at a time.

One Page Bugs

Reducing the friction of writing and fixing bugs or features.

Follow

Get every new post delivered to your Inbox.

Join 223 other followers