EnglishSvenska

Calculate Personal Financial worth with Python

I wanted to know how much my financial assets are worth across one or more bank accounts, so I created a small python script for it.

I just type all my assets into a list like this:

assets = {
'AAPL': {
'own': 40,
'current_value': yahoo_share('AAPL')
},
'Coca-Cola': {
'own': 35,
'current_value': yahoo_share('KO')
},
'NOF': {
'own': 170,
'current_value': yahoo_share('NOF.OL')
},
'Nordnet Superfonden Sverige': {
'own': 6.4099,
'current_value': morningstar("http://www.morningstar.se/Funds/Quicktake/Overview.aspx?perfid=0P0000J24W")
},
'SEB Japanfond': {
'own': 491.8263,
'current_value': morningstar("http://www.morningstar.se/Funds/Quicktake/Overview.aspx?perfid=0P00000LU7")
},    
}

Then I run the script and get an output looking a bit like this:
Skärmavbild 2015-09-12 kl. 13.40.51

(Not my actual positions 🙂 )

Next step is to program a get rich quick algorithm. Until then the code for calculating assets is published here:
https://github.com/sebnil/Pengar

Tagged with:

Responsive one-page WordPress site

I have not been doing wordpress sites for a while but today I changed that. Nalisha Chouraria needed a simple personal webpage and I got it done in a day by using some shortcuts (mainly Twitter bootstrap css).

nalishac.com
Skärmavbild 2015-09-06 kl. 18.50.15

 

Tagged with: ,

HIL testing in Arduino

I am trying out Hardware-in-the loop simulation on a new Arduino project for a client. Test driven development is unfortunately not very big in the Arduino community (yet) so I decided to implement something by myself. The setup is simple:

  1. One Arduino is running the application software.
  2. Another Arduino connects to inputs and outputs on the first Arduino. This Arduino will include all the test code.

2015-05-14 16.08.09

A test case could for example be, when the user presses a button a LED should light up. The second Arduino will output a signal to the button input on the first Arduino, and then check if the LED pin output is high. Example code is shown below.

void loop() {
// Simulate that user presses button
digitalWrite(BUTTON_PIN, 1);
// check that the LED lights up
assert(LED_PIN, 1);
delay(500)
// check that some actuator starts running
assert(ACTUATOR_PIN, 1);
// Simulate that user releases button
digitalWrite(BUTTON_PIN, 0);
// Led and actuator should turn off
assert(LED_PIN, 1);
assert(ACTUATOR_PIN, 1);
// stop execution
while (1) {}
}
bool assert(uint8_t pin, bool expectedState) {
bool state = digitalRead(pin);
if (state != expectedState) {
Serial.print("## FAILURE: pin ");
Serial.print(pin);
Serial.print(" == ");
Serial.print(state);
Serial.print(" != ");
Serial.print(expectedState);
Serial.println();
return false;
}
else {
Serial.print("## OK: pin ");
Serial.print(pin);
Serial.print(" == ");
Serial.print(state);
Serial.println();
return true;
}
}

Why the hassle?

It might seem unnecessary, (and it is for simple problems), but it does increase code quality and decreases the risk of bugs being introduced when writing new features to the code.

Making it better

I would like to write my test code in Python on a laptop and control an Arduino (via Firmata for example). Then I would have proper tools for testing and generating test reports. For now the Arduino solution is sufficient though.

Tagged with:

Python for sftp and mysql backup

I needed to backup some sftp sites and mysql from a remote server to my local server at home. Piece of cake in Python.

After that I add it as a Jenkins script to schedule periodic backups:

Skärmavbild 2015-05-05 kl. 23.18.45 jenkins_no_bg

import shutil
import os
import paramiko
import pysftp
import select
import logging
logging.basicConfig(level=logging.DEBUG)
def sftp_backup(ssh_host=None, ssh_username=None, ssh_password=None, source_directory=None, local_directory=None):
with pysftp.Connection(ssh_host, username=ssh_username, password=ssh_password, log=True) as sftp:
sftp.chdir(source_directory)
# first remove the local directory to make room
try:
logging.info('Removing local directory: {}'.format(local_directory))
shutil.rmtree(local_directory)
logging.info('Done removing local directory: {}'.format(local_directory))
except:
logging.info('Can\'t delete {}. Probably does not exist'.format(local_directory))
# then create the directory
if not os.path.exists(local_directory):
logging.info('Creating empty local directory: {}'.format(local_directory))
os.makedirs(local_directory)
logging.info('Done creating local directory: {}'.format(local_directory))
# recursively copy to local_directory
logging.info('Starging to download from {} to {}'.format(source_directory, local_directory))
sftp.get_r('', local_directory)
logging.info('Done')
def dump_mysql_to_file_via_ssh(ssh_host=None, ssh_user=None, ssh_password=None, mysql_host=None, mysql_user=None,
mysql_password=None, mysql_databases=None, output_file='dump.sql'):
logging.debug('dump_mysql_to_file_via_ssh')
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=ssh_host, username=ssh_user, password=ssh_password)
stdin, stdout, stderr = ssh.exec_command('mysqldump --host={mysql_host} -u {mysql_user} -p{mysql_password} --databases {mysql_databases}'.format(
mysql_host=mysql_host,
mysql_user=mysql_user,
mysql_password=mysql_password,
mysql_databases=mysql_databases
))
logging.info('Begin writing to file {}'.format(output_file))
file = open(output_file, 'w')
# Wait for the command to terminate
while not stdout.channel.exit_status_ready():
# Only print data if there is data to read in the channel
if stdout.channel.recv_ready():
rl, wl, xl = select.select([stdout.channel], [], [], 0.0)
if len(rl) > 0:
# Print data from stdout
r = stdout.channel.recv(1024)
file.write(str(r))
file.close()
logging.info('Done writing to file.')
ssh.close()
if __name__ == '__main__':
dump_mysql_to_file_via_ssh(
ssh_host='ssh.example.com',
ssh_user='',
ssh_password='',
mysql_host='',
mysql_user='',
mysql_password='',
mysql_databases='',
output_file='sebastiannilsson.com.sql'
)
sftp_backup(ssh_host='ssh.example.com', ssh_username='', ssh_password='', source_directory='', local_directory='')
Tagged with:

Algorithmic trading with Python

I have been experimenting with algorithmic trading for a couple of weeks. Zipline is a Python library for backtesting trading algorithms and I would like to share one of the algorithms I made.

import talib
from zipline.api import record, order_target, history, add_history
import dateutil
import logging
from zipline.utils.factory import load_from_yahoo
from zipline.finance.slippage import FixedSlippage
from zipline.algorithm import TradingAlgorithm
from zipline.finance import commission
logging.basicConfig(level=logging.DEBUG)
# initialize algorithm
def initialize(context):
logging.debug('enter initialize')
context.set_slippage(FixedSlippage())
context.set_commission(commission.PerTrade(cost=5))
context.LOW_RSI = initialize.low_RSI
context.HIGH_RSI = initialize.high_RSI
context.rsi_window = initialize.rsi_window
add_history(context.rsi_window, '1d', 'price')
context.i = 0
context.invested = False
# default parameters for algorithm
initialize.rsi_window = 15
initialize.low_RSI = 30
initialize.high_RSI = 70
# Will be called on every trade event for the securities you specify.
def handle_data(context, data):
logging.debug('enter handle_data')
context.i += 1
if context.i < context.rsi_window:
return
# get the last RSI value
prices = history(context.rsi_window, '1d', 'price')
sec_rsi = talib.RSI(
prices[context.security].values,
timeperiod=context.rsi_window - 1)
# buy and sell flags
buy = False
sell = False
if sec_rsi[-1] < context.LOW_RSI and not context.invested:
# RSI under 30 indicates oversold, time to buy
order_target(context.security, 1000)
logging.debug('Buying {}'.format(context.security))
context.invested = True
buy = True
elif sec_rsi[-1] > context.HIGH_RSI and context.invested:
# RSI over 70 indicates overbought, sell everything
order_target(context.security, 0)
logging.debug('Selling {}'.format(context.security))
context.invested = False
sell = True
# record data for each time increment
record(secRSI=sec_rsi[-1],
price=data[context.security].price,
buy=buy,
sell=sell)
logging.info(context.portfolio.cash)
def run_algorithm(
security='AAPL',
start_date='20100101',
end_date='20150101',
initial_cash=100000,
rsi_window=15,
low_RSI=30,
high_RSI=70):
logging.debug('run_algorithm begin')
# dates
start = dateutil.parser.parse(start_date)
end = dateutil.parser.parse(end_date)
# get data from yahoo
data = load_from_yahoo(stocks=[security], indexes={}, start=start, end=end)
logging.debug('done loading from yahoo. {} {} {}'.format(
security, start_date, end_date))
# create and run algorithm
algo = TradingAlgorithm(
initialize=initialize,
handle_data=handle_data,
capital_base=initial_cash)
algo.security = security
initialize.low_RSI = low_RSI
initialize.high_RSI = high_RSI
initialize.rsi_window = rsi_window
logging.debug('starting to run algo...')
results = algo.run(data).dropna()
logging.debug('done running algo')
return results
if __name__ == '__main__':
import matplotlib.pyplot as plt
# run algorithm and get results
results = run_algorithm(
security='AAPL',
start_date='20100101',
end_date='20150101',
initial_cash=100000,
rsi_window=15,
low_RSI=30,
high_RSI=70)
# get s&p500 and nasdaq indexes
index_data = load_from_yahoo(
stocks=['^gspc', '^ixic'],
indexes={},
start=results.index[0],
end=results.index[-1])
# portfolio value, stock holdings and S&P 500 index
fig = plt.figure(figsize=(12, 6))
ax11 = fig.add_subplot(311)
ax12, ax13 = ax11.twinx(), ax11.twinx()
ax13.spines['right'].set_position(('axes', 1.07))
ax11.set_ylabel('portfolio value', color='blue')
ax12.set_ylabel('holdings', color='green')
ax13.set_ylabel('S&P 500', color='red')
# portfolio value
ax11.plot(results.index, results.portfolio_value, color='blue')
# holdings (number of stocks owned)
holdings = [0 if t == [] else t[0]['amount'] for t in results.positions]
ax12.plot(results.index, holdings, color='green')
ax12.set_ylim([min(holdings) - 30, max(holdings) + 30])
# index
ax13.plot(index_data.index, index_data['^gspc'], color='red')
# algo visualization
ax21 = fig.add_subplot(312)
ax21.set_ylabel('stock price', color='blue')
ax22 = ax21.twinx()
ax22.set_ylabel('rsi', color='red')
# stock
ax21.plot(results.index, results.price, color='blue')
# add sell and buy flags on top of stock price
ax21.plot(
results.ix[results.buy].index,
results.price[results.buy],
'^',
markersize=10,
color='green')
ax21.plot(
results.ix[results.sell].index,
results.price[results.sell],
'v',
markersize=10,
color='red')
# rsi value
ax22.plot(results.index, results.secRSI, color='red')
# add lines to show under- and over value indicator
ax22.plot([results.index[0], results.index[-1]], [30, 30], 'k-')
ax22.plot([results.index[0], results.index[-1]], [70, 70], 'k-')
# portfolio value, stock value and index in percentage
ax31 = fig.add_subplot(313)
ax32, ax33 = ax31.twinx(), ax31.twinx()  # share x for other plots
ax31.set_ylabel('algo %', color='blue')
ax32.set_ylabel('snp index %', color='green')
ax33.set_ylabel('stock %', color='red')
ax33.spines['right'].set_position(('axes', 1.07))
# portfolio value
ax31.plot(
results.index,
results.portfolio_value / results.portfolio_value[0] * 100 - 100,
color='blue')
# index
ax32.plot(
index_data.index,
index_data['^gspc'] / index_data['^gspc'][0] * 100 - 100,
color='green')
# stock value
ax33.plot(
results.index,
results.price /
results.price[0] * 100 - 100,
color='red')
plt.show()

If you get it working you should see a plot similar to this one:
Skärmavbild 2015-04-04 kl. 22.20.20

If you are observant, it is easy to see that the performance of this algorithm is not good enough to be used on a real portfolio, and it is more of a test.

Links:

Tagged with: ,

Wall computer

What to do with a worn out Samsung series 9 ultraslim laptop? Turn it into a wall computer! Perfect for showing weather and traffic information.

Tagged with: , ,

Master thesis: Remote controlled truck - Proof of concept for designing a remote system for a Volvo truck

A proof of concept for designing a remote system for a Volvo truck. Designing and buildning a remote control to be able to drive a Volvo truck from outside the truck. Investigating wireless technology, safety aspects, designing and building PCB, designing and building mechanics for remote control, systemizing the whole system and writing embedded code. Simulink was used to glue it all together.

The thesis is available from the Chalmers University online library

Tagged with:

Location, location, location

Läge är kanske den viktigaste parametern när man söker bostad, men varför presenteras den så dåligt på boplats.se? Med Tampermonkey la jag till ett skript som tar bostadsadressen och beräknar avstånd och cykeltid till valfri adress. För mig var det viktigt att min bostad inte var för långt från mitt arbete.

Utan mitt skript ser Boplats.se ut såhär:

boplats.se utan tampermonkey

Och med mitt skript blir Boplats.se en nästan bra webbplats:

bokplats.se med tampermonkey

Hur gör man:

  1. Installera Tampermonkey i Google Chrome
  2. Kodiera mitt skript.
  3. Modifera destinationsadress i koden
  4. Jag tror att skriptet aktiveras automatiskt så fort man går in på Boplats.se. Om inte så trixa i Settings i Tampermonkey.

Tampermonkey-kod:

// ==UserScript==
// @name       Boplats.se enhancer
// @namespace  http://sebastiannilsson.com/blogg/location-location-location/
// @version    0.1
// @description  Adds google maps integration to Boplats.se
// @match      https://boplats.se/
// @copyright  Whatever
// @require http://code.jquery.com/jquery-latest.js
// ==/UserScript==
$(document).ready(function() {
var i = 0;
var elements = $('#dgList.tbl_list tr');
elements.each(function() { 
if (i++ == 0)
return;
var origin = $(this).find('td:eq(1)').text() + ", " + $(this).find('td:eq(2)').text();
var origin2 = $(this).find('td:eq(1)').text() + ", Göteborg";
var destination = "Bergskroken 3, Mölndal";
origin = encodeURIComponent(origin);
origin2 = encodeURIComponent(origin2);
destination = encodeURIComponent(destination);
$.ajax({
url: "http://sebastiannilsson.com/boplats-google-maps/index.php?origin="+origin+"&destination="+destination+"&mode=bicycling",
dataType: 'text',
context: $(this)
}).done(function(s) {
console.log("done: http://sebastiannilsson.com/boplats-google-maps/index.php?origin="+origin+"&destination="+destination+"&mode=bicycling");
if (s== "NOT_FOUND") {
$.ajax({
url: "http://sebastiannilsson.com/boplats-google-maps/index.php?origin="+origin2+"&destination="+destination+"&mode=bicycling",
dataType: 'text',
context: $(this)
}).done(function(s2) {
console.log("done2: http://sebastiannilsson.com/boplats-google-maps/index.php?origin="+origin2+"&destination="+destination+"&mode=bicycling");
$(this).append( "<td>"+s2+"</td>" );
});  
}
else {
$(this).append( "<td>"+s+"</td>" );
}
});
});
});
Tagged with:

Rapidly building a python gui application

One of my favorite scripting languages is python, and for a couple of good reasons, and the most important reason rapid development from idea to a working application. I will show this with a example by building a simple python app with a gui and then make it ready for deployment by creating an executable file.

Prerequisites:

  • Python 2.7 (and not 3.* since wxpython support is still under development)
  • wxpython will handle the gui stuff.
  • wxformbuilder is a gui editor that can generate wxpython code for us. In most cases this is faster than writing this code by hand.
  • pyinstaller to freeze the application (create an .exe on Windows or a .app on osx)
  • A text editor (sublime, notepad++, pycharm or anything similar)
  • Good enough python skills to know how to install new modules. I will not explain how to install all the modules listed above.

1. Building the gui

Start wxformbuilder. By default a new project is already open. Set filename to gui and code generation to Python:

0 wxformbuilder 6 set project prop

Add a frame:

1 wxformbuilder form1.1 set form name

Add a wxBoxSizer. This will hold our gui components and stack them vertically:

2 wxboxsizer

Add a text and a button:

3 wxbitmapbutton

Name them m_text and m_button (or whatever):

5 set prop text4 set prop button

Bind an event for button clicks since we want to do something when the user clicks the button:

4.1 set prop button event

Save the wxformbulder project to a new folder. Then click Generate code from the menu.

7 generate code

The content of your project folder should contain the wxformbuilder project file and the generated file:

8 files

2. Integrate gui into Python code

The idea is to import the gui file, extend the functions and override the events. Lets use this code as a skeleton:

import wx
import gui
# Extend the gui with some new functionality
class MyFrame(gui.MainFrame):
def __init__(self, parent, title):
gui.MainFrame.__init__(self, parent)
# this is the event we defined in wxformbuilder, and now override from gui.py
def on_button_click_event(self, event):
print('on_button_click_event')
# Create wxpython app
class MyApp(wx.App):
def OnInit(self):
self.frame = MyFrame(None, "Hello Wxpython")
self.SetTopWindow(self.frame)
self.frame.Show(True)
print("wxApp created.")
return True
if __name__ == "__main__":
app = MyApp(redirect=False) # do not redirect stdout to the gui
app.MainLoop() # render gui continuously

Try running the code and you should see something like this:
9 running the skeleton

Now lets modify it to do something when user clicks the button:

import wx
import gui
# Extend the gui with some new functionality
class MyFrame(gui.MainFrame):
def __init__(self, parent, title):
gui.MainFrame.__init__(self, parent)
# this is the event we defined in wxformbuilder, and now override from gui.py
def on_button_click_event(self, event):
print('on_button_click_event')
# Create wxpython app
class MyApp(wx.App):
def OnInit(self):
self.frame = MyFrame(None, "Hello Wxpython")
self.SetTopWindow(self.frame)
self.frame.Show(True)
print("wxApp created.")
return True
if __name__ == "__main__":
app = MyApp(redirect=False) # do not redirect stdout to the gui
app.MainLoop() # render gui continuously

Now a button click should trigger the event and print something to the console:

10 running the finnished code

Building an executable (freezing) the application

Since Python is a script language we (usually) don't compile it into machine code. But if you want to deploy your application or send it user which do not have Python installed, you can freeze your application into an executable file (.exe in windows or .app in OS X). There are multiple tools to do this but I choose to use pyInstaller since it works on both windows and OS X, and can pack your application into a single executable instead of multiple files.

In principle, using pyInstaller is quite straight forward. From the terminal (OS X) or command prompt (windows), you can use this command to build your file:
# pyinstaller -y --windowed --onefile --icon=icon/banana.icns --name="' + filename + '" hello_wxpython.py

Build executable from a Python script

I usually use a custom script when I want to build an executable. You can download the pyinstaller_helper module from the github link at the end.

import pyinstaller_helper
pyinstaller_helper.build({
'script': 'hello_wxpython_tutorial.py',
'application_name': 'hello wxpython',
'version': '1.0.0.1',
'company_name': u'Example',
'product_name': u'Hello wxPython',
'internal_name': 'product_name',
'original_filename': u'Hello wxPython',
'file_description': 'Lorem ipsum.',
'legal_copyright': 'your@email.com',
'legal_trademark': ''
})

Running build_executable.py on OS X will generate an .app file, and on windows a .exe file. The script will create an executable that is specific for that platform, so for me it looks like this:

11 dist code

Download the code

This tutorial is on github.

Build a GUI in python using the Qt framework

If you would like to learn how to build a GUI using the Qt framework instead of Wxpython then try this sponsored link:

https://www.udemy.com/python-gui-programming/

Use the coupon code: sebastiannilsson.com and if you are one of the first 50 you get it for free.

Tagged with: ,

Formula Student Baltic Open 2014

Sensommaren och hösten kunde ha blivit utan praktiskt ingenjörsjobb om det inte vore för att jag blev övertalad att bygga räserbil. Målet var Baltic Open i Estland och dit skulle vi ta CFS13 och CFS13e (alltså både den bensin- och eldrivna bilen).

Tagged with: ,