Monday, December 26, 2011

howto: Observer pattern

The observer pattern comes very handy when dealing with complex systems. It allows for class-to class communication with a very simple structure. Even more important is the ability to separate functionality in different modules, for example running a single 'broker' as a wrapper to some api and letting multiple strategies subscribe to relevant broker events. There are some ready-made modules available, but the best way to understand how this process works is to write the whole system from scratch. In many languages this is a very tedious task, but thanks to the power of Python it only takes a couple of lines to do this.

The following example code creates a Sender class (named Alice). Sender keeps track of interested listeners and notifies them accordingly. In more detail, this is achieved by a dictionary containing a function-event mapping, Sender.listeners.
A listener class can be of any type, here I make a bunch of ExampleListener classes, named Bob,Dave & Charlie. All of them have a method, that is that is subscribed to Sender. The only special thing about the subscribed method is that it should contain three parameters: sender, event, message. Sender is the class reference of the Sender class, so a listener would know who sent the message. Event is an identifier, for which I usually use a string. Optionally, a message is the data that is passed to a function.
A nice detail is that if a listener method throws an exception, it is automatically unsubscribed from further events.


'''
Created on 26 dec. 2011
Copyright: Jev Kuznetsov
License: BSD

sender-reciever pattern.

'''

import tradingWithPython.lib.logger as logger
import types

class Sender(object):
    """
    Sender -> dispatches messages to interested callables 
    """
    def __init__(self):
        self.listeners = {}
        self.logger = logger.getLogger()
        
        
    def register(self,listener,events=None):
        """
        register a listener function
        
        Parameters
        -----------
        listener : external listener function
        events  : tuple or list of relevant events (default=None)
        """
        if events is not None and type(events) not in (types.TupleType,types.ListType):
            events = (events,)
             
        self.listeners[listener] = events
        
    def dispatch(self,event=None, msg=None):
        """notify listeners """
        for listener,events in self.listeners.items():
            if events is None or event is None or event in events:
                try:
                    listener(self,event,msg)
                except (Exception,):
                    self.unregister(listener)
                    errmsg = "Exception in message dispatch: Handler '{0}' unregistered for event '{1}'  ".format(listener.func_name,event)
                    self.logger.exception(errmsg)
            
    def unregister(self,listener):
        """ unregister listener function """
        del self.listeners[listener]             
                   
#---------------test functions--------------

class ExampleListener(object):
    def __init__(self,name=None):
        self.name = name
    
    def method(self,sender,event,msg=None):
        print "[{0}] got event {1} with message {2}".format(self.name,event,msg)
                   

if __name__=="__main__":
    print 'demonstrating event system'
    
    
    alice = Sender()
    bob = ExampleListener('bob')
    charlie = ExampleListener('charlie')
    dave = ExampleListener('dave')
    
    
    # add subscribers to messages from alice
    alice.register(bob.method,events='event1') # listen to 'event1'
    alice.register(charlie.method,events ='event2') # listen to 'event2'
    alice.register(dave.method) # listen to all events
    
    # dispatch some events
    alice.dispatch(event='event1')
    alice.dispatch(event='event2',msg=[1,2,3])
    alice.dispatch(msg='attention to all')
    
    print 'Done.'
    
    
    



Wednesday, December 14, 2011

Plotting with guiqwt

While it's been quiet on this blog, but behind the scenes I have been very busy trying to build an interactive spread scanner. To make one, a list of ingredients is needed:

gui toolkit: pyqt -check.
data aquisition: ibpy & tradingWithPython.lib.yahooData - check.
data container: pandas & sqlite - check.
plotting library: matplotlib - ehm... No.

After tinkering with matplotlib in pyqt for several days I must admit its use in interactive applications is far from optimal. Slow, difficult to integrate and little interactivity. PyQwt proven to work a little better, but it had its own quirks, a little bit too low-level for me.
But as it often happens with Python, somebody, somewhere has already written a kick-ass toolkit that is just perfect for the job. And it looks like guiqwt is just it. Interactive charts are just a couple of code lines away now, take a look at an example here: Creating curve dialog . For this I used guiqwt example code with some minor tweaks.

And of course a pretty picture of the result:

...If only I knew how to set dates on the x-axis....

Friday, November 4, 2011

How to setup Python development environment

If you would like to start playing with the code from this blog and write your own, you need to setup a development environment first. I've already put a summary of tools and software packages on the tools page and to make it even easier, here are the steps you'll need to follow to get up and running:

1. Install PythonXY. : this includes Python 2.7 and tools Spyder, Ipython etc.
2. Install Tortoise SVN. This is a utility that you need to pull the source code from Google Code
3. Install Pandas (time series library)

This is all you need for now.
To get the code, use 'Svn Checkout' windows explorer context menu that is available after installing Tortoise SVN. Checkout like this (change Checkout directory to the location you want, but it should end with tradingWithPython):
If all goes well, the most recent version of the files will be downloaded. I'll be writing more code and improving current one, you'll be able to stay in sync with my code by using 'svn update' context menu.

The final step is to launch Spyder (through pythonXY launcher or start menu) and add the directory just above the 'tradingWithPython' (in my example C:\Users\jev\Desktop') dir to python path . Do this with 'tools'->'PYTHONPATH manager'.
Ok, all done, now you can run the examples from \cookbok dir.

Friday, October 28, 2011

kung fu pandas will solve your data problems

I love researching strategies, but sadly I spend too much time on low-level work like filtering and aligning datasets. In fact, about 80% of my time is spent on this mind numbing work. There had got to be a better way than hacking all the filtering code myself, and there is!
Some time ago I've come across a data analysis toolkit pandas especially suited for working with financial data. After just scratching the surface of its capabilities I'm already blown away by what it delivers. The package is being actively developed by Wes McKinney  and his ambition is to create the most powerful and flexible open source data analysis/manipulation tool available. Well, I think he is well on the way!

Let's take a look at just how easy it is to align two datasets:

from tradingWithPython.lib import yahooFinance

startDate = (2005,1,1)

# create two timeseries. data for SPY goes much further back
# than data of VXX
spy = yahooFinance.getHistoricData('SPY',sDate=startDate)
vxx = yahooFinance.getHistoricData('VXX',sDate=startDate)

# Combine two datasets
X = DataFrame({'VXX':vxx['adj_close'],'SPY':spy['adj_close']})

# remove NaN entries
X= X.dropna()
# make a nice picture
X.plot()

Two lines of code! ( this could be even fit to one line, but I've split it for readability)
Here is the result:

Man, this could have saved me a ton of time! But it still will help me in the future, as I'll be using its DataFrame object as a standard in my further work.

Monday, October 17, 2011

Tools & Cookbook

I've added two pages specifically to help new users to get started.
Tools: here you'll find all the info you need to set up a development environment.
Cookbook: Overview of recipies I've written. The code itself is hosted on Google Code.

Saturday, October 15, 2011

How to download a bunch of csv files

I'll be analyzing VIX futures data, so to start with, I need all the files from CBOE. They use a consistent format for file names, which is easy to generate. The following script creates a data directory and then downloads all csv files for the period 2004-2011. As you can see, not much coding is required, less than 30 lines!


from urllib import urlretrieve
import os

m_codes = ['F','G','H','J','K','M','N','Q','U','V','X','Z'] #month codes of the futures
dataDir = os.getenv("USERPROFILE")+'\\twpData\\vixFutures' # data directory

def saveVixFutureData(year,month, path):
    ''' Get future from CBOE and save to file '''
    fName = "CFE_{0}{1}_VX.csv".format(m_codes[month],str(year)[-2:])
    urlStr = "http://cfe.cboe.com/Publish/ScheduledTask/MktData/datahouse/{0}".format(fName)

    try:
        urlretrieve(urlStr,path+'\\'+fName)
    except Exception as e:
        print e

if __name__ == '__main__':
    if not os.path.exists(dataDir):
        os.makedirs(dataDir)

    for year in range(2004,2012):
        for month in range(12):
            print 'Getting data for {0}/{1}'.format(year,month)
            saveVixFutureData(year,month,dataDir)

    print 'Data was saved to {0}'.format(dataDir)

Friday, October 14, 2011

Interactive work with IPython

Pretty good screencast on python basics/
There is more of this good stuff which can be found here.

Sunday, September 25, 2011

Getting Started

First of all, I must admit that it took me quite some time even to consider switching from Matlab to Python. I've been working with Matlab for more than ten years, and I must say that it is an excellent tool for research. However, both in my professional and private work there where quite some cases where Matlab was not the best tool to do the job. This forced me to learn things like Labwindows and C# for gui programming, php for website building. Not to forget that Matlab is not a free (not as in speech, nor as in beer) product.
Still, the main reason for me trying to make the switch, is  Python being a 'swiss knife of programming'. It can be used to make excellent GUIs with Qt, conduct research with SciPy, program dynamic websites and so on.

All posts on this blog will be based on the following tooling:


  • As a base distribution I'm using PythonXY, which is an excellent package containing (almost) everything one needs to do scientific programming with Python.
  • To connect to Interactive Brokers you'll also need IbPy.