Banking on Perl

Lisbon -- 3rd August 2009

Paul Johnson

www.pjcj.net

Perl in Large Corporations

Perl helps in keeping lots of large organisations running
So why are so few willing to talk about it?
Perhaps for some it really is a business secret
But for most it probably isn't

Perl in UBS

For UBS use of Perl isn't a business scret
I'll talk about some of the ways Perl is used in UBS
But without giving away any real business secrets
I have permission to give this presentation

UBS

"world-leading wealth management company that sits down with you to understand your needs and goals"
www.ubs.com
Marketing division of the Sirius Cybernetics Corporation

UBS

UBS formed from a merger between Union Bank of Switzerland and Swiss Bank Corporation, which had previously merged with Warburgs
Still technologically quite distinct
WM&SB and IB
Usage of Perl has emerged along separate lines

3 minutes

Glue

Banks use lots of technologies
They need to be joined together
Perl is good at joining things together
Perl is often known as a glue langauge
Joins other components together

New Development in Perl

Applications

Bringing business advantage
Used by business users

Little projects

To keep the IT world running
Sysadmin tasks
Monitoring
Automating procedures
Used by technical users
Sometimes fullblown projects with budgets, timescales, reviews etc.
Sometimes just people doing their jobs

Case studies

A project I've worked on for the last four years
Other ways Perl is used at UBS

Here comes the science bit!

Banks move lots of money around
By moving money to the right places, banks can earn or avoid paying interest
This makes bankers happy

Go, Johnny, Go!

This process is called:
Cash Management
Perl has helped UBS implement its Cash Management solution
Together with a copious dollop of CPAN

5 minutes

Outsourcing

Primary development is outsourced
Customised product
Needs to be integrated

Systems

databases
webservers
communications
high availability
monitoring
logging
archiving
deployment

Databases

Oracle
use DBI;

Webservers

IHS - IBM's rebranding of apache
Websphere
No special modules

Communications

message transfer
mail
SMS
error reporting
IRC
file transfer

Data Munging

Files come in in all sorts of formats
They need to be standardised
pack and upack
use Spreadsheet::ParseExcel

MQSeries

message transport
use MQSeries;
Kindly written and maintained by our friends at Morgan Stanley

High availability

System handles many millions of [IYFC] per day
Insert Your Favourite Currency
If it breaks, the bank loses many thousands of [IYFC] per not-very-long
system needs to stay up
backup server just waiting to spring into action
mount the disks and start it up

Monitoring

How do we know when something goes wrong?
When should we failover?
Monitor!
Based on Nagios
Plugin infrastructure and plugins
Monitor:

system
application
even business logic

Logging

Logs are important to find out what went wrong
Core application produces many Gb of logs per day
Far to large to be useful
Required restart of aplication every night to rotate logs
Restarting often caused problems in the middle of the night
Made the sysadmins grumpy

Logging

I wrote a wrapper around the logging system
Made the core application write to named pipes
Read the pipes and write to proper log files
Plus all sorts of snazzy stuff

Logging

use Log4Perl;
Based on select
Multiple processes
Made the application much more robust
Made the business folk happy
and the sysadmins
Made the logs useful

Archiving

The logs are still enormous
So they need to be archived
Zipped up and stored until they are old enough to be thrown away

8 minutes

Deployment

Sun packages
Package creation
Automated deployment on test machines
Version control with ~~SVK~~ git

Extra development

Database Investigation
Management Information System
Legal Archiving
Automated Testing
More Communications

Database Investigation

Second system
Half external
Internal part based on Catalyst
DBIx:Class, TT, JSON

Management Information System

Daily data copied from DB and sent via file transfer
SQL took 40 hours per day
Rewritten in perl and takes less than 2 hours

Legal Archiving

Send data to a UBS system
C or Java interfaces
XS

Automated Testing

All the Test::* modules
Devel::Cover
WWW::Mechanize
Test::WWW::Selenium
Reading and writing from all interfaces

More Communications

Based on Tibco EMS
C++ and Java interfaces
... but in practice only Java
Groovy!

CPAN

Algorithm::Diff aliased Alien::SeleniumRC Any::Moose AppConfig Archive::Tar Archive::Zip Array::Compare autobox B::Hooks::EndOfScope B::Hooks::OP::Check Bit::Vector B::Utils Cache::FastMmap Carp::Assert Carp::Assert::More Carp::Clan Catalyst::Action::RenderView Catalyst::Controller::FormBuilder Catalyst::Devel Catalyst::Engine::Apache Catalyst::Engine::HTTP::POE Catalyst::Model::DBIC::Schema Catalyst::Plugin::Authentication Catalyst::Plugin::Authorization::Roles Catalyst::Plugin::ConfigLoader Catalyst::Plugin::JSONRPC Catalyst::Plugin::Server Catalyst::Plugin::Session Catalyst::Plugin::Session::State::Cookie Catalyst::Plugin::Session::Store::FastMmap Catalyst::Plugin::StackTrace Catalyst::Plugin::Static::Simple Catalyst::Plugin::XMLRPC Catalyst::Runtime Catalyst::View::JSON Catalyst::View::TT CatalystX::Component::Traits CGI::FormBuilder CGI::Simple Class::Accessor Class::Accessor::Chained Class::Accessor::Grouped Class::C3 Class::C3::Adopt::NEXT Class::C3::Componentised Class::Data::Accessor Class::Data::Inheritable Class::Inspector Class::MOP Class::Singleton Class::Trigger Clone Compress::Raw::Bzip2 Compress::Raw::Zlib Config::Any Config::General Config::Tiny Context::Preserve CPAN CPAN::Mini C::Scan CSS::Tiny Data::Dump Data::Dumper Data::Flow Data::OptList Data::Page Data::UUID Data::Visitor Date::Calc DateTime DateTime::Locale DateTime::TimeZone DBD::Pg DBD::SQLite DBI DBIx::Class DBIx::Class::Schema::Loader DBIx::ContextualFetch DBIx::Log4perl DBIx::ProcedureCall Devel::Caller Devel::Cover Devel::Declare Devel::GlobalDestruction Devel::Leak Devel::PartialDump Devel::StackTrace Devel::Symdump Digest::MD5 Digest::SHA Email::Address Email::Send Email::Simple Error ExtUtils::CBuilder ExtUtils::Depends ExtUtils::MakeMaker FCGI FCGI::ProcManager File::ChangeNotify File::Copy::Recursive File::Find::Rule File::HomeDir File::Modified File::Remove File::Temp File::Which Hook::LexWrap HTML::Lint HTML::Parser HTML::TagCloud HTML::Tagset HTTP::Body HTTP::Request::AsCGI HTTP::Response::Encoding HTTP::Server::Simple Image::Size IO::Compress IO::String IO::stringy IO::Tty IO::Zlib JSON JSON::Any JSON::XS libnet libwww::perl Lingua::EN::Inflect Lingua::EN::Inflect::Number List::MoreUtils Locale::Maketext::Simple Log::Log4perl MIME::Base64 MIME::Types Module::Build Module::Find Module::Install Module::ScanDeps Moose Moose::Autobox MooseX::Declare MooseX::Emulate::Class::Accessor::Fast MooseX::Meta::TypeConstraint::ForceCoercion MooseX::MethodAttributes MooseX::Method::Signatures MooseX::Params::Validate MooseX::SemiAffordanceAccessor MooseX::Traits MooseX::Traits::Pluggable MooseX::Types MooseX::Types::Structured Mouse MRO::Compat namespace::autoclean namespace::clean Net::Server Net::SSLeay Number::Compare Number::Format Object::Signature OLE::Storage_Lite PadWalker Params::Util Params::Validate PAR::Dist Parse::CPAN::Meta Parse::Method::Signatures Path::Class PathTools Perl6::Junction Perl::Tidy Pod::Coverage POE POE::Test::Loops PPI PPI::HTML Proc::ProcessTable Return::Value RPC::XML Scalar::List::Utils Scope::Guard Set::Object Shell::Source SOAP::Lite Sort::Versions Spreadsheet::ParseExcel SQL::Abstract SQL::Abstract::Limit String::RewritePrefix Sub::Exporter Sub::Install Sub::Name Sub::Uplevel Task::Weaken Template::Plugin::Number::Format Template::Timer Template::Toolkit TermReadKey Term::ReadLine::Perl Test::ClassAPI Test::Deep Test::Differences Test::Exception Test::Harness Test::JSON Test::LongString Test::Mock::LWP Test::MockObject Test::NoWarnings Test::Object Test::Pod Test::Pod::Coverage Test::Simple Test::SubCalls Test::Tester Test::use::ok Test::Warn Test::WWW::Mechanize Test::WWW::Mechanize::Catalyst Test::WWW::Selenium Text::Aligner Text::Diff Text::Glob Text::SimpleTable Text::Table Text::TabularDisplay Text::Template Tie::IxHash Tie::ToObject TimeDate Tree::DAG_Node Tree::Simple Tree::Simple::VisitorFactory UNIVERSAL::can UNIVERSAL::isa UNIVERSAL::require URI Variable::Magic WWW::Mechanize XML::Parser YAML YAML::LibYAML YAML::Perl YAML::Syck YAML::Tiny