During a phone screen this weekend I was asked to describe all of my payments experience in a 2-3 page cover letter. I quickly wrote an outline and started filling in the blanks and submitted my first draft. This morning I printed the first draft which was now 7 pages. I have since cleaned up the spelling and much of the grammar. It’s not meant to be a memoir and some descriptions are subjectively technical; and I’ve left out details that professionals should already know. Anyway here it is.
—-
The following text represents the many payment systems I designed, implemented, supported, updated, managed, and contributed to in some way. It should be needless to say that I have worked on other projects in other vertical markets and other languages. I trust you will see the value that I bring to the business as well as the technology. One final note. These are my personal accomplishments. Sometimes I was part of a team and sometimes I worked alone it just depended on scheduling, resources, SME, etc.
In 1993 I started working, as a contractor, for NaBanco (acquired by First Data) as a contractor. I designed and implemented a TSR, written in assembler, for their FoxPro/DOS hospitality application. The TSR was designed to connect to each of the property’s Zon terminals and download it’s transactions. It would then post the transactions in the FoxPro database. Later the FoxPro app would send all of the aggregated data to the NaBanco’s host via the TSR. One last thing that the TSR would do (in the days prior to the popular internet) was a trivial email service for HQ to communicate with the properties.
After this project was finished my manager recommended me to the HR department. I interviewed with and was hired to design and develop the ValueLink platform. This was a closed loop stored value system. The First client, BlockBuster Video, needed a working platform ASAP. Once the hardware was selected I went about defining the toolset. Having evaluated Informix, which was currently running on NaBanco’s debit system, I decided on Oracle with PRO*C and a RAD GUI development tool from Computer Associates.
There were a number of tough challenges in designing this system. At the time I did not have any experience on Sun hardware and while I had worked on databases for years I did not know much about SQL other than the evaluation I had just performed. Additionally I had to learn multi-threading, multiplexing transactions over X.25, and everything that comes with OLTP production support. And while I had experience with the Zon terminal there was still a lot more to learn.
The next challenge was the helpdesk. I implemented the first desktop app with a toolset from CA (Computer Associates). The app lacked performance based on the PCs at BlockBuster’s offices in Ft Lauderdale. I used a 2400 baud dial up modem to connect the two locations. Shortly after the project went live I hired a VB programmer to rewrite the application, however, since the application was also going to be used internally we were going to have a lot more users connected than I wanted. So I implemented a REST/SOAP-like server using Java and Java WebServer from Sun. It worked brilliantly and was later used by the IVR subsystem.
Finally, I was introduced to Perl. I used Perl to implement two major systems. The first was the card account creation in order to generate plastics and send them to manufacturing and I also used Perl for generating product performance reports (TPS reports).
In the end I was able to implement a fast, flexible, and reliable system that now transacts over 700 TPS every single day(with plenty of headroom) and hosts thousands of merchants and over 500M accounts.
This platform’s most notable accounts include: BlockBuster Video, Walmart, Starbucks, and the USPS.
WildCard Systems was a client of First Data, however, during the early stages of their discovery it was decided that First Data was not going to be able to deliver. Mostly because they were going in a different direction. Since many of the people who were engaged in the conversation were friends it was easy for me move over.
At WildCard I was tasked with designing a different type of open loop stored value card system. I had implemented the first multi-wallet system that was to be used by insurance companies in order to pay or deliver money to the insured. While WildCard eventually circled back to HSA, FSA and eligibility applications they moved away from direct insurance applications.
The authorization system was implemented in two parts. The first part was a java based front end system that would connect to the association, reformat the transaction (the process of message normalization), adapt to network impedance, and then execute the particular transaction request against a set of T-SQL stored procedures and complex data configuration with rules. This front-end system was eventually certified to work with: Visa, Amex,MasterCard, Discover, First Data Resources. The overall platform replaced Visa’s LAC platform.
Early on it was discovered that the state of the art PC was not going to keep up with our needs so I implemented a rudimentary replication engine in java. This application would sync 4 master-master database servers in different data-centers over a dedicated WAN connection. Eventually others in the department as well as Microsoft tried their hand at replication.
I designed a template language that could emit html, pdf, txt, and csv files. This was written in Perl and was intended to limit the roundtrips to the DB. As a domain specific language it was non-trivial to produce reports and the demand was greater than the staff could produce. Eventually all of the data had to be replicated to a farm of 5 database servers in order to produce the reports.
One of the newer projects I worked on was “WebDog”. This internal-use webapp performed a number of functions supporting the operations staff. (1) it was a production migration management system, where developers wanting to submit code for production would write a ticket that had to be approved and the app managed the workflow. (2) it monitored all of the SQL Server databases. (3) It monitored all of the front end processors. (4) the most important thing it monitored was the approval ratio. When the ratio was out of spec we knew there was a release problem. (5) lastly it was responsible for deciding which SQL Server was the current master.
This killer app was conceived on a beach in Nantucket; modeled after Star Trek, deployed on FreeBSD, used MySQL, written in Perl, receiving requests via apache and mod_perl, and templated responses with Mason.
Notable clients included: AAA, Bank of America as well as the Visa Buxx brand.
After leaving WildCard I decided to work on a side project. One of the last discussions we had at WildCard had to do with TPS rates. The existing system was only working at about 25-TPS at 100% CPU Utilization (8 CPU with 16GB RAM). I posited that (1) there was a problem with our SAN. It has been reported that period SAN drives suffered from brown-outs. (2) there many examples based in truth bashing MicroSoft and SQL Server. Oracle was so much more performant. (3) T-SQL was a pig, all of the code was essentially doing hash lookups O(1) using a relational search O(lg(n)).
So I submitted two papers to SleepyCat, the makers of Berkeley DB. The papers represented payment system designs based on BDB and BDB-XML. I received two honorable mentions. I also implemented one of my designs using Java and BDB. I was able to get 1500TPS on a single core, single spindle drive.
**sidebar** by this time in my professional development I had discovered erlang. The notion that if a language like erlang can offer 9-sigma, if implemented correctly, in a phone switch environment then how different could that be in payments. 9-sigma would be a great platform/language to implement payment.
What attracted me to eDiets was a similarity to a side project I was working on, however, one of the projects I implemented for the company was a prototype erlang merchant gateway. This allowed their internal payment system to connect to different acquirer systems. The first prototype was implemented in erlang and later it was replaced with a java implementation as an ATG plug-in. The team was excited about the erlang potential, however, management steared the company toward more java.
I joined MetaVentures to support their existing CRM platform for Verifone magstripe devices. The Perl application communicated device configuration and transaction details to/from the Verifone devices. Since I had payment knowledge I was tasked to design and implement a complete end to end payment system. This included; POS, HSM, merchant gateway, and PCI compliance. The HSM and merchant gateway were implemented in erlang. The POS is a mix of languages including Perl, C, SQL and bash.
While the erlang systems were interesting to construct it was uneventful. Certifying with multiple acquirers was as simple as changing the message templates. They have been running without interruption since they were installed. There are necessary enhancements, however, none of the current team members really want to spend any time on erlang. (to be continued later)
The POS was interesting in that it needed to support a kiosk mode browser in javascript which used websockets to communicate with local webservice daemons that were connected to barcode scanners, scales, customer facing displays, pin pads, and a magstripe reader.
The gateway was certified with RBS, Global, and First Data. And is PCI compliant.
Insight Card Systems implemented a Ruby/Rails platform for account and card management. At specific times of the day it would perform account balance updates to a service provider and the service provider would send transaction details back. The system suffered from a number of problems including reporting performance and reporting accuracy. Even though I was the director of development I was optimizing the SQL and training the programmers on new ways to get more performance out of their platform, and making production operations decisions. Furthermore I implemented proper release process in order to reduce downtime and improve release quality.
As the director I had a number of other roles and assignments. I needed to hire more staff and bring development in-house. (currently outsourced). I also had to redesign a system that had 5-10x capacity with the same hardware that was currently at 100% capacity. And I had to address client expectations and customization.
I started Florida Freelance because of the economic times we live in. I had a couple of contracts that I knew I could work on. The first was a VOIP arbitrage system that generated about 1M minutes a day in call volume. This was an integrated Asterisk switch and a connected dashboard. I was tasked with redesigning the system because the original system was dropping calls, losing calls, performing badly, and could not handle the volume they needed. While this project is not a payment system is does demonstrate my ability to scale.
My second client, a bill.me.later-like company in Stockholm Sweden; hired me based on my experience. They wanted me to contribute to their existing platform and help them design new applications in areas I had detailed experience. Their platform is implemented in erlang, however, I built several interfaces in java and C as part of another plan to unify their message passing and logging. I also performed a complete PCI audit of their HQ and operations centers in Stockholm.
**sidebar** One of the interesting features of erlang is hot-code replacement. The erlang core allows developers to replace modules on the fly without interruption. However, while many erlang programmers think this is a cool feature it is actually a detriment to payment systems. Hot-plugging code causes transactions in flight to become unreproducable due to the version mismatch of sub-modules through the transaction. From an operations POV, if you are going to switch master/slave or HA configurations in order to release new versions… then you might as well restart the app. This way you are assured that the app will restart.
A recent client in Portland Oregon, asked me to perform a number of projects. The first was a one-day design and overall roadmap for their future issuing platform and to see whether I was compatible with the CEO. A few months later they asked me to perform a due diligence on a potential payment vendor’s platform. And finally to design a custom issuing system for them in the EU. This was to include to EMV for chip. Shortly after beginning this part of the project I was tasked to design the same for China Union Pay.
Another client in Atlanta Georgia; has decided to rewrite their erlang gateway and HSM. While the system has been running this entire time it still suffers from the inability to enhance the application. Initially they wanted to implement the new platform in C but I convinced them that Python/tornadoweb/redis was a good choice. They recently certified with WorldPay on the first attempt. The entire project took less than a month.
There was a brief moment when I was having second thoughts about Python. The team was made up of Perl programmers, however, their tech lead was not grocking it and wanted a chance to contribute and python was going to be a lot easier for him to learn and easier still for the others to adopt.
So that’s about everything payments. I look forward to fielding any questions you might have.
