Setting up a Kubernetes cluster on Digital Ocean with SSL

Introduction

It is now possible to create/maintain Kubernetes clusters on Digital Ocean quite easily.
In this post I will show you how to create a cluster with a pod that has a public endpoint; this endpoint will be secured with a Let’s Encrypt SSL certificate.
Some basic Kubernetes knowledge is assumed so i won’t go into detail explaining what a deployment, service or pod is, or how to work with kubectl. N.B. Kubernetes support on Digital Ocean is still in beta, so it maybe not available to you.

Overview

The application we are going to deploy is called ‘whoami’ which has an endpoint that outputs some information about the request that is
sent to it.
We will expose this endpoint through https://whoami.example.com (N.B. example.com is a fake domain name and is used for illustration purposes, replace it with domain you actually own)

In the picture below is the end result is shown:
The parts that are of interest for external access are the ingress, ingress controller and the load balancer.

An ingress is a Kubernetes object defined in our application namespace that defines the rules to which service (or pod) external requests are routed.
In this case we define an ingress with the following rule:


“If an incoming request contains ‘whoami.example.com’ in the host header, the request is routed to the whoami service.”


The actual routing is done by an ingress controller which in most cases uses also a load balancer to route incoming requests.
The controller we will use is Traefik, which also handles SSL termination. This means also that the traffic between the controller and the service/pod is flowing through HTTP.
Traefik also handles the automatic acquiring and renewal of the Let’s Encrypt certificates

Prequisites:

  • You have a Digital Ocean account and created a project in the control panel.
  • You own the domain example.com and you are able to adminster its DNS records through Digital Ocean.
  • You have kubectl installed on your computer.
  • A Digital Ocean API token with read/write access, which is needed later for the configuration for the acquiring of the certificate.

Setting up a cluster in Digital Ocean

Through the Digital Ocean control panel create a cluster with 1 node to keep things simple (and cheap):

  • Choose ‘Kubernetes’ in the left navigation bar.
  • Click ‘Create’ and select ‘Clusters’.
  • Select datacenter AMS3.
  • Under ‘Add node pool’ select from the ‘Standard node pool’ the ‘$5 month plan’ and set the number of nodes to 1.
  • For the name type ‘k8s-test’.
  • Click ‘Create cluster’ and a new cluster will be provisioned.
  • When the cluster is provisioned go to the cluster control panel and click ‘Download’.
  • Copy this file to your ~/.kube directory and name the file ‘config’. This makes kubectl point to the just created cluster.
  • We need to create a default service account with full rights (needed for the Helm package manager and for the Traefik package be able to deploy its stuff to the cluster,
    you probably don’t want to do this in production (at least have a dedicated user). )
  • Paste the following to a file and execute ‘kubectl apply -f filename’ to deploy it to the cluster.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-system

Traefik

To install the Traefik ingress controller:
  • Install Helm, this is a package manager for Kubernetes which makes installation of Traefik very easy.
  • Run ‘helm init’ to initialize Helm on the client and on the cluster.
  • Run ‘helm install –name traefik-thingie -f values.yaml stable/traefik –namespace kube-system’.
    The values.yaml file contain the configuration for Traefik. A more detailed explanation is provided below.
  • Wait until Traefik is deployed. (it creates also a L4 load balancer in Digital Ocean)
  • In the DO control panel look up the IP address of the load balancer just created and create an A record in the Domain section with the whoami.mydomain.com domain pointing to this IP address.

Configuration

Below is the configuration of the Traefik helm chart which is used in this example. For more information about all options see here.
ssl:
  enabled: true        # Enables SSL
  enforced: true       # Redirects HTTP to HTTPS
acme:
  enabled: true             # Enables Let's Encrypt certificates
  staging: true             # Use Lets Encrypt staging area for this example. For production purposes set this to false
  email: fill.in@your.email # Email address that Let's Encrypt uses to notify about certificate expiry etc.
  challengeType: "dns-01"   
  dnsProvider:              
    name:  digitalocean     # This is why you need your domain to be under Digital Ocean control
    digitalocean:
      DO_AUTH_TOKEN: "INSERT_API_TOKEN_HERE"
  domains:
    enabled: true
    domainsList:
      - main: "example.mydomain.com" # Name of the domain that belongs to this certificate
Save this file as ‘values.yaml’ and use it during installation of Traefik. (see above)

Application

The application is deployed on the cluster using a simple service and deployment.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: whoami-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami-container
        image: containous/whoami
----
apiVersion: v1
kind: Service
metadata:
  name: whoami-service
spec:
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoami
Paste this to a file and use (again) ‘kubectl apply -f filename.yaml’ to deploy it to the cluster.

Ingress

The important part however is the definition of the ingress.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: whoami-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: whoami.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: whoami-service
          servicePort: http
The interesting bits are here:

  • The annotation kubernetes.io/ingress.class should be set to ‘traefik’, so that the Traefik ingress controller is aware of our ingress (remember, an ingress is just a routing rule which the ingress controller should enforce)
  • The ‘host’ field should be set to our domain, and the ‘backend’ should be set to our service defined above.
Paste this to a file and use (again) ‘kubectl apply -f filename.yaml’ to deploy it to the cluster.

Showtime!

If all of the above went well, go to https://whoami.example.com and you should see a page with the content like below (note the X-Forwarded-Port and X-Forwarded-Proto which indicates that the request was coming through https. )
Hostname: whoami-deployment-84dfcf599c-5s9ll
IP: 127.0.0.1
IP: 10.244.16.3
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.244.16.1
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-thingie-74ff44595f-lflrm
X-Real-Ip: 10.244.16.1

New screenshots of the FruitHAP Android App

With the upcoming FruitHAP Android App you can control your FruitHAP setup from anywhere in your network.
Here are some screenshots:

 

In the dashboard you can control all switches etc. and see all sensor values in one screen.

 

In the Alerts screen you can see an overview of alerts which are defined in FruitHAP (for example when the temperature in a room is too high, or the doorbell has rang)

Stay tuned for more …

Upcoming new release of FruitHAP

It has been a while since the last version (see here) , but after some months of work a new version is coming soon!

Some of the new features:

  • The FruitHAP Notifier app, with which you can remote control your FruitHAP installation, receive alerts, turn on the lights etc.
  • Added support for my arduino temperature sensor (see here)
  • Added easy deployment of the engine to a target device with an Ansible script
  • More and better documentation (finally…)
  • Bug fixes, improvements

Keep an eye on my twitter account (@traveller1505) for updates on this..

Temperature sensor with Arduino

I wanted to have a cheap temperature sensor to use with my home automation system (FruitHAP).
This sensor must be able to send its temperature through a 433 Mhz RF link since I have already a 433 Mhz RFXCom receiver.
Also a display would be nice to display the temperature so I could put this in a room of my house to monitor the temperature there.
I always wanted to do something with an Arduino, so i thought to create an Arduino project out of this. So I present you my first Arduino project.

Hardware

Parts list

For this project you need:

Since i am too lazy to solder (and i am more a software than a hardware guy) I used components from the Seeed Grove platform
This platform enables you to very quickly cobble together a circuit, it’s literally plug and play.
It consists of a shield which you just click on top of your Arduino and in this shield you can click in various other Grove components.

caption

Assembling the circuit

  • Plug the Base Shield onto the Arduino (check that the alignment is correct)
  • Plug the LCD with a grove cable into one of the I2C ports of the Base Shield
  • Plug the 433MHz transmitter with a grove cablee in port D2 of the Base Shield
  • Put the LED into the LED Socket kit (long leg is the anode/+)
  • Plug the LED socket kit with a grove cable in port D3 of the Base Shield
  • Plug the temperature sensor with a grove cable in port A0 (analog port 0) of the Base Shield

That’s all! The final result is in the pictures below.

OLYMPUS DIGITAL CAMERA OLYMPUS DIGITAL CAMERA OLYMPUS DIGITAL CAMERA

Software

Getting it

Get the code from here and upload it to the Arduino with the help of the IDE.
You need also some external libraries:

When all is correctly connected and set up, the current temperature should be displayed on the LCD (as a bonus the background color changes when the temperature changes, you can test it by putting you finger on the component labeled NTC on the temperature sensor)
The LED should blink every 10 seconds or so (can be altered, see source code), when this happens the current temperature is transmitted through the 433 Mhz transmitter.

Technical details

I won’t discuss all the code, you can view it yourself and in the comments there is also useful info, but i would like to point out some details.

X10

Like mentioned before, the library used to transmit the measurements is X10 since this can emulate a so called RFXSensor device, which is basically
a protocol based off X10.

This protocol can be understood by the RFXCom RFXtrx433 receiver (which i am using already in combination), but you are free to use another receiver/solution. It’s  described in the docs of the X10 library (look for the file x10_RF_formats.pdf and then the section RFXSensor and you can look in the source code of the lib itself).

However, if you use the RFXtrx433 as receiver, pay attention to the fact that it transforms the received information into its own protocol.
Basically the data will be stored in 2 bytes and the temperature is actually stored as temperature * 100 so the range of temperatures to be send is limited.

Possible improvements

Possible improvents could be:
– Put the thingie in a nice box
– Use a combined temperature/humidity sensor, so that you can monitor humidity too!

Hope you like it! Happy coding!

FruitHAP – Part 1

In this series of posts I will describe my own C# based Home Automation platform FruitHAP

What is FruitHAP

FruitHAP is my home-made home automation system, designed to run on a Raspberry Pi.
The name stands for Fruit Home Automation Platform and is a pun on the Raspberry Pi (since you all know that a raspberry is fruit). Also ‘Fruithap(je)’ in Dutch is the word for a plate of mashed fruit you give to a baby; something like the picture below.

Logo

Why did you make your own system, since there are already lot of other HA solutions

I have a house with an attic and since this my ‘man-cave’ i sit/work there a lot. The only problem is that I often can’t hear the doorbell ring, especially if the washing machine is also turned on (it’s on the attic as well and it’s noisy).

So I came up with an idea to make something that delivers a notification to my cellphone when someone rings the frontdoor bell.
Oh, and maybe it’s also convenient to put an image of the person standing there in the notification, so I can decide if it’s worthwhile to go downstairs and answer the door.

Then I bought a Raspberry PI, a wireless doorbell, a camera and some other stuff. Now I needed some software to make this setup work. I looked first at some existing HA platforms but decided that they didn’t put up to my expectations, so i decided to build my own. (it is a lot more fun too 🙂 )

Requirements

I came up with some requirements for my platform:

  • Extensible
    I heard that automating your house is addicting. It shoulde be easy to add different sensors and sensor types, since there are a lot of standards of sensors out there.
  • Accessibilty of sensor data
    It should be easy to access sensor data from other systems/apps. Maybe you want more than only notifications on your phone, maybe you want to have a dashboard with all your sensor data or log
    your energy consumption to a database to perform some analysis on it (BIG data is hot these days)
  • Should run on a Raspberry Pi
  • Open source

General overview of the platform

A high level overview of the platform is in the diagram below. I will explain it a bit in more detail.

fruithap_overview

Engine

The main part of the system is called the Engine. This is a daemon that runs on the Raspberry Pi and acts as the gateway between the actual physical sensors and the ‘outside’ world.
The engine communicates with the outside world by means of a message queue, so external applications can easily interact with the system by reading/writing into this queue, through a publish/subscribe mechanism or
request/response mechanism. (RPC calls)

For example, when the doorbell rings, the event goes from the doorbell through the engine and finally got published in the message queue. An app running on a phone can listen (subscribe) to this event on the queue and send a
notification when this event is fired.

An example of request/response is when an external application wants to read the temperature of a certain sensor. It sends a request to the message queue which the engine handles by retrieving the value from the sensor concerned
and sends it back (through the queue) to the external application.

The idea is that all the configuration of the engine and the other components will also be exposed in this way to make it easier to make a nice web interface for configuring everything, but that’s another story for another time.

Controllers

The ‘lowest’ layer is the controller layer. This layer handles all nitty-gritty low level hardware protocol stuff.
It communicates directly with the sensor hardware or through another hardware controller, such as a zWave or a 433 mhz controller.

Sensors

The sensor layer is the most important layer and is considered the core of the platform. These contain virtual representations of the physical sensors/actuators and use the controllers to communicate with the actual sensors
so that all hardware/protocol specifics are abstracted away (only the configuration for a sensor exposes some of the specifics).

For example one of the actuators supported is a KaKu button. KaKu stands for KlikAanKlikUit and consists a whole range of actuators/sensors/device with a specific communication protocol. Each device has an unique device ID and some more specific settings.
These have to be set only in the sensor configuration with an unique name and for the rest of the system and the outside world it is known as a button with that particular name, which can be ‘pressed’.

Sounds difficult? No worries, in the next post when I will describe my setup it will all come together.

Also it’s possible to have ‘aggregate’ sensors, a sensor that combines the input 2 or more separate physical sensors in a way and use this as one value. This is comparable with some of the Android ‘logical sensors’.
An example is a ButtonCamera, it combines a Button and a Camera, so when the button is pressed an image from the camera is retrieved and is sent along with the button pressed event as if it was one physical sensor.

Actions

An action is actually the interface between the sensor(s) and the message queue. Its input(s) is/are coming from either the sensors or from the outside (messages from the queue) and its
outputs are requests for sensors or other parts of the system (configuration!) or messages that are bound for the queue. Basically is this a rule engine where you can implement all kinds of scenarios for your home automation needs.
A standard action is for example to send a message to a queue every time a sensor updates. These actions can also have specific configuration.

Phew.. that was a lot of reading

This is just a short description of the system. If you have any remarks or comments about this post please let me know.
Next time I will show you an example of how this thing works. The example is actually my live doorbell setup.
When the doorbell rings FruitHAP is picking this up and a notification with a image from a camera is sent through Pushbullet to my phone.
More details about this in the next post..

See you next time!

Since FruitHap is open source, here is a link to the source.
If you can’t wait to try this thing (and you have the right equipment) you can also find here a quick setup with a complete Raspberry Pi image.

A blinking light with Raphaël.js

For my Freeboard based dashboard for my home automation system (that’s a thing i will talk about in another post 🙂 ) i was looking for a fancy blinking LED style light. Basically i wanted it to have the following requirements:

  • Possibility to turn on/turn off the light.
  • Possibility to start/stop blinking
  • Possibility to carry out some action when you click on the light
  • It has to look cool.

During my search for something like this i came along Raphael.js and i decided to build my own blinking led with this framework, just to get it to know better. I think it is a very cool framework with very cool UI possibilities.
So i created a a small javascript library, which i called Blinkenlight that implements all requirements. You can find the code and an example here.

Example usage

Preparation

Put the downloaded .js files somewhere on your file system, preferably some place from which you can reference to it from your html file.

HTML

In your html file add the following script tags in the head section:
<script type=text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/raphael/2.1.4/raphael-min.js">
<script type="text/javascript" src="http://code.jquery.com/jquery-2.1.4.js"></script>
<script type="text/javascript" src="path_to_js/blinkenlight.js"></script>

The first line adds the Raphaël.js library, the second line adds the JQuery library, which i explain later and the third line adds the Blinkenlight library.

Raphaël.js works with a canvas or paper on which we are going to draw our light. This canvas has to be put in some DOM element on your page.
So let’s add this element to the body section of the html page (you can use your own id here):
<div id="canvas_container"></div>

Initialization function

In a new script tag in the head section of the html file add the following:

<script type="text/javascript">
$(document).ready(function ()
{
});

We will initialize our canvas and draw our light and all other things in this function which comes with JQuery and this is the reason why we needed the JQuery library. It’s possible to use the window.onload() or the document.onload()  functions but i didn’t test it. (according to this it’s better to use the JQuery function because of possible browser incompatibilities)

Initialization of Raphael.js

To initialize Raphael.js add the two following lines in the $(document).ready function:

var element = document.getElementById('canvas_container');
var paper = new Raphael(element, 200, 80);

We retrieve our canvas DOM element (somehow this doesn’t work with a JQuery $("#canvas_container"), the page layout gets fucked up) and pass it to the Raphael.js constructor function. The two other parameters of this function are the width and height of the canvas in pixels, in the above case, 200 px wide and 80 px high.
This function returns a canvas which we use for the initialization of the blinkenlight.

Blinkenlight initialization

To initialize a Blinkenlight use the constructor function:

function BlinkenLight(paper, x, y, radius, colorOn, colorOff, glowColor, tooltipText, clickHandler)

Description of parameters:

paper
The Raphael canvas where the light is to be drawn.

x
X-coordinate on the canvas. (0 is left)

y
Y-coordinate on the canvas. (0 is top)

radius
Radius of the light in pixels

colorOn
Color of the light when it is in the ‘on’ state

colorOff
Color of the light when it is in the ‘off’ state

glowColor
Color of the glow around the light.

tooltipText
(Optional) Tooltip text which is displayed when a click handler is assigned

clickHandler
(Optional) Javascript that is executed when you click on the light

Example 1, without click handler:

var blinker1 = new BlinkenLight(paper, 100, 40, 20, "#00DDDD", "#004444", "#00FFFF");

Example 2, with click handler and tooltip text

var blinker2 = new BlinkenLight(paper, 40, 40, 20, "#00DD00", "#004400", "#00FF00", "Click me", function ()
{
alert('You clicked me!')
});

Initially the lights will be ‘off’, so the result of the above will be look like:

blinkenlights_off

Changing state of the light

To turn on the light, call the turnOn() function:

blinker1.turnOn();
blinker2.turnOn();

blinkenlight_on

To turn off the light, call the turnOff() function:

blinker1.turnOff();
blinker2.turnOff();

blinkenlights_off

Blinking

First call the setInterval() function with a parameters which sets the blinking interval in ms:

blinker1.setInterval(500);

Then call the start() function to start blinking:

blinker1.start();

To stop the blinking call the  stop() function:

blinker1.stop();

Conclusion

I hope you get an idea how this thing works. The example provided with the code shows all the concepts plus an example of a click handler that shows a image in an popup.
If you have any comments, tips or tricks please leave them in the comments below.What’s next

In a following post i will use this library to make an actual widget plugin for Freeboard.

 

Happy coding and until next time!

kanban-warrior sees its first birth!

In a previous post I described how you can implement a Kanban board with Task Warrior.
To make life a bit easier when using the flow, especially moving between the stages, I made a wrapper script around Task Warrior, which I called kanban-warrior
It accepts simple commands (in the style of Task Warrior 🙂 ), to add and move around tasks in the different stages. It also gives an error message when you try to move a task to the wrong stage. It can also generate reports of the different stages.
Since kanban-warrior provides just some convenient shortcuts to implement the flow (if you look at the code, you will see that it actually executes TW commands), you can just use it in combination with other TW commands (and equally fuck up things 🙂 )

I released version 1.0 and you can get it here.
It is one of my first projects in Python, so forgive me if the coding style is not according to Python standards.
Please contact me if you have comments, encounter bugs, have suggestions for new features, or even if you want to contribute.

A basic usage example

Add a task to the backlog

./kanban-warrior addtobacklog "Starship:propulsion" "Add static warpfield converter"
Created task 3.
The project 'Starship:propulsion' has changed.  Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining)

Display the backlog

./kanban-warrior.py list backlog Starship
ID Project                  Pri Added                 Started Due Recur Countdown Age Deps  Tags     Description                  
 3 Starship:propulsion   -  14-3-2013 20:59                 -                              2m           backlog           Add static warpfield converter                                                                                                                                                        
1 task

Move a task from the backlog to the Work In Progress Queue and start it immediately

./kanban-warrior start 3
Loading task list..

Modifying task 3 'Add static warpfield converter'.
Modified 1 task.
Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining).

Starting task 3 'Add static warpfield converter'.
Started 1 task.
Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining).

Set a task on hold

 ./kanban-warrior.py hold 3 "Fucked up"
Loading task list..

Stopping task 3 'Add static warpfield converter'.
Stopped 1 task.
Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining).

Modifying task 3 'Add static warpfield converter'.
Modified 1 task.
Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining).

Annotating task 3 'Add static warpfield converter'.
Annotated 1 task.
Project 'Starship:propulsion' is 0% complete (1 of 1 tasks remaining).

Invalid move

./kanban-warrior.py finish 3
Loading task list..
Error: Task must be in progress

You can’t move a task in the ‘on hold’ stage directly to the ‘done’ stage, it has to go through the ‘in progress’ stage first

Note: The ‘3’ in the commands above is the TW task id, which you can retrieve by just generating a TW report (either with or without kanban-warrior)
More information about the siupported commands can be found in the README file and there is also a short help screen when executing kanban-warrior -h

Kanban Warrior

Yesterday I discovered this awesome command line tool called Task Warrior. It is basically a to-do list on steroids; you can define tasks, group them into projects, generate reports etc. etc. (see the website for more information). When I was fiddling around with the program, the thought occurred to me (since I use mostly Scrum+Kanban methodologies in my work projects): Can Task Warrior be used to implement a Kanban board? (for more information about the Kanban methodology see here).

After some thought and browsing through the Task Warrior forum about this topic with some very good ideas, i’d come up with my own workflow. It is actually based on the solution Nikola Petrov posted in the forum. Note that there is no checking on constraints (max WIP items, illegal moves between columns, etc.), so it’s possible to mess up things, that is all up to you. Then again, when using an “analog” whiteboard you can also do these funky things.

Explanation of stages

  • Backlog: Work items defined to be done in this period/sprint/whatever
  • In progress: A queue of work items that YOU are currently working on. See it as your personal backlog. This should be limited to maybe 1 or 2 items (but that is a matter of convention. Can contain active items, i.e. the things you are really working on at this very moment
  • Done: Work items that are done
  • On hold: Work items that can’t be worked on due to an external cause

You can see an example and screenshots below. I used the terminal application found in Xubuntu and for each stage (Backlog, In progress, Done, On Hold) I defined a tab. I use the tab Control for administration, i.e. adding & modifyng tasks.

Add some tasks to the backlog:

task add project:"Starship.Propulsion" +backlog "Implement subspace distortion detector"
Created task 1
The project 'Starship.Propulsion' has changed.  Project 'Starship.Propulsion' is 0% complete (1 of 1 tasks remaining).

task add project:"Starship.Propulsion" +backlog "Implement warp coil heater"
Created task 2
The project 'Starship.Propulsion' has changed.  Project 'Starship.Propulsion' is 0% complete (2 of 2 tasks remaining).

task add project:"Starship.Propulsion" +backlog "Implement warp containment field modulator"
Created task 3
The project 'Starship.Propulsion' has changed.  Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

task add project:"Starship.Defense" +backlog "Implement tachyon beam detector"
Created task 4
The project 'Starship.Defense' has changed.  Project 'Starship.Defense is 0% complete (1 of 1 tasks remaining).

task add project:"Starship.Weapons" +backlog "Improve phaser array"
Created task 5
The project 'Starship.Weapons' has changed.  Project 'Starship.Weapons is 0% complete (1 of 1 tasks remaining).

Moving items from the backlog to the ‘In progress” queue

task 1-3,5 modify -backlog +inprogress
Modifying task 1 'Implement subspace distortion detector'.
Modifying task 2 'Implement warp coil heater'.
Modifying task 3 'Implement warp containment field modulator'.
Modifying task 5 'Improve phaser array'.
Modified 4 tasks.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

(Re)start working on items in the In progress queue

task 1,5 start
Starting task 1 'Implement subspace distortion detector'.
Starting task 5 'Improve phaser arrayr'.
Started 2 tasks
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).
Project 'Starship.Weapons' is 0% complete (1 of 1 tasks remaining).

Stop working on an item (but it’s not finished yet)

task 1 stop
Stopping task 1 'Implement subspace distortion detector'.
Stopped 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

Start working on an item, but then the item is put on hold for some reason

task 3 start
Starting task 3 'Implement warp containment field modulator'.
Started 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

task 3 stop
Stopping task 3 ''Implement warp containment field modulator''.
Stopped 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

task 3 modify -inprogress +onhold
Modifying task 3 'Implement warp containment field modulator'.
Modified 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

task 3 annotate "We can't set the correct delta-factor compensation"

Putting an “On hold” item back to the “In progress” queue and start working on it again

task 3 modify -onhold +inprogress
Modifying task 3 'Implement warp containment field modulator'.
Modified 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

task 3 start
Starting task 3 'Implement warp containment field modulator'.
Started 1 task.
Project 'Starship.Propulsion' is 0% complete (3 of 3 tasks remaining).

Setting items to Done

task 1,3 modify -inprogress
task 1,3 done
Completed task 1 'Implement subspace distortion detector'.
Completed task 3 'Implement warp containment field modulator'.
Completed 2 tasks.
The project 'Starship.Propulsion' has changed.  Project 'Starship.Propulsion' is 66% complete (1 of 3 tasks remaining).

Notes:

I used the ‘subproject’ feature to indicate my user stories (so ‘Starship’ is the name of the project and ‘Propulsion’ is the name of the story), so that I can relate tasks to a particular story.

Screenshots

[nggallery id=1]

A hardware monitor in C# 4.0 with the Task Parallel Library – Part 2 – Exception handling

Welcome back at part 2 in this series. In the last part I described the basic framework for the hardware monitor. In this part I will add some exception handling to the monitor and also a way to provide the monitor with extra configuration parameters.

The code can be found here, in the “Part 2” folder.

Exception handling

We would like to stop the main task when an exception is thrown and log the information.
Remember the continuation task we defined earlier to handle the main task cancellation. We can extend this that it also handles exceptions.

 this.readerTask.ContinueWith(t =>
 {
     if (t.IsFaulted)
     {
        t.Exception.Handle((x) =>
        {
           Console.WriteLine("Exception in task: {0}", x);
           return true;
        });

        try
        {
           CleanUp();
        }
        catch (Exception ex)
        {
           Console.WriteLine("Cleanup exception: {0}", ex.Message);
        }
    }
   //Notify everyone that we stopped
   OnStatusChanged(MonitorStatus.STOPPED);
   Console.WriteLine("Reader task stopped");
},TaskContinuationOptions.NotOnRanToCompletion);

When an exception in a Task is thrown, the status of the task is set to IsFaulted. So our continuation task should also be executed if the main task is in the faulted state. We accomplish this by changing the TaskContinuationOptions to TaskContinuationOptions.NotOnRanToCompletion, so that it executes when the main task is cancelled or is in the faulted state.

All exceptions thrown in a Task are collected by the TPL and put in one exception of type AggregateException (see here).
The Exception property of a Task contains this AggregateException. It exposes also a Handle method which has a delegate as parameter.
This delegate is executed for every exception in the AggregateException. In this delegate you can put your exception handling code, for example write it to log file or in the snippet above, write it to the console. (the X is the parameter to the delegate and is the current exception being processed) It should return True to indicate that this particular exception is handled, if it returns false the exception is further thrown down the stack.

After handling the exceptions, the CleanUp() method is called to ensure the monitor can be started again in a correct way.

Configuration parameters

It is not unthinkable that you want to pass some configuration parameters to your monitor, like baudrates, communication ports etc.
To add this functionality we have to change the interface a little bit:

public interface IDeviceMonitor<TConfigData, TDeviceData> : IDisposable
{
  void Start(TConfigData configData);
  ...
}

An extra type parameter (TConfigData) is added which represents a class (or even just an int or double or whatever)  that defines the configuration data you want to use.
When calling the Start method you pass an instance of this class.

Our base class also changes a bit, because, obviously, you want to work with the configuration data in your derived class:

public abstract class DeviceMonitorBase<TConfigData, TDeviceData> : IDeviceMonitor<TConfigData, TDeviceData>
{
  ..
  protected TConfigData configData;    
  public void Start(TConfigData configData)    
  {    
      this.configData = configData;
 ..

Example

I improved the example implementation a bit. I added a second timer monitor (TimerMonitorWithException) which throws an exception when the number of seconds is equal to an user specified number. You can pass this number as an argument to the Start() method. It therefore demonstrates the exception handling AND the use of configuration parameters.

I also extended the original timer monitor so that it uses a configuration parameter. Here you can adjust the interval of  printing the current time to the console.

This was part 2 of this series.  I hope you enjoyed reading this as much as you enjoyed the first part.

Happy coding and until next time!

A hardware monitor in C# 4.0 with the Task Parallel Library – Part 1 – The basics

Sometimes you need to continuously and in the background retrieve and process data from an external device connected to your system (for example a GPS receiver or you want to monitor your ethernet connection). The external device has maybe a low level driver/interface which only has the possibility to retrieve its data synchronously. I designed a small framework in C# 4.0 that makes a nice threaded abstraction layer around this. The multithreaded part is implemented with the help of the Task Parallel Library.
In this and following posts I will try to describe it step by step. The complete code can be found here and all code snippets below refer to the Part 1 project in the solution

[Disclaimer] I don’t pretend that this is the ONLY and/or MOST EFFICIENT way to implement this. It just works for me. If you have any remarks or suggestions for improvement or bugs, please leave them in the comments below [/Disclaimer]

Ok, after this introduction, LET’S GET STARTED!

Interface

First we start by defining an interface. Obviously we want to start and stop the retrieval/monitoring and since the monitor will be run on a background thread we define an event that is going to be fired when new data arrives.

public interface IDeviceMonitor<TDeviceData> : IDisposable
{
    void Start();
    void Stop();
    event EventHandler<MonitorStatusEventArgs> MonitorStatusChanged
    event EventHandler<DataReceivedEventArgs<TDeviceData>> DataReceived;
}

The generic parameter TDeviceData defines a class in which you can put the data your hardware/device makes available. An instance of this
class is passed along with the event. Since hardware communication may involve access to unmanaged resources we make it IDisposable as well.
We also define an event that is fired when the state of the monitor changes (e.g. from started to stopped or vice versa).

Base class

Then we define a base class in which we put all the TPL stuff and other plumbing code so that they are hidden away nicely from the actual implementations. (those implementations contain probably a lot of low level hardware interaction, and we don’t want to get all this plumbing code in the way.)

public abstract class DeviceMonitorBase<TDeviceData> : IDeviceMonitor<TDeviceData>

Here we define a couple of life cycle methods which can (or must) be overridden in a derived class.

protected virtual void Initialize() { }
protected abstract TDeviceData ReadData();
protected virtual void CleanUp() { }
public virtual void Dispose() { }

Initialize() is called after the monitor is started before entering it’s main event loop. You can put optional initialization code in here, like opening ports, setting baudrates and other stuff.

ReadData()  is called in the main event loop and is the method in which it all happens, reading the data from a device, and must be (obviously) overridden in a derived class.

CleanUp() is called after the monitor receives the request to stop. Here you can put clean up code like closing ports etc. N.B. It’s not recommended to put here ‘expensive’ operations like freeing memory or freeing device handles, put this code in the Dispose() method.  The purpose of this method is to put the monitor in a sort of ‘sleeping’ state so that it can be started again quickly.
The same principle counts for the Initialize method, don’t put there any memory allocation or other ‘heavy’ operation, since this will be executed every time when the monitor is started. You can put those heavy operations in the constructor of your derived class.

TPL stuff

The whole process is executed in a  Task (see this msdn link for more information) with a continuation task which handles the case when the main task is canceled after a call to the Stop() method.

The main task (readerTask) is created in the Start() method and its operations are defined in a lambda expression. The comments explain what is happening.

void Start()
{
    this.ctSource = new CancellationTokenSource();
    CancellationToken ct = ctSource.Token;

    this.readerTask = new Task(() =>
    {
         //Are we cancelled yet?
         ct.ThrowIfCancellationRequested();

         //Notify everyone that we started
         OnStatusChanged(MonitorStatus.STARTED);

         //Do initialization work
         Initialize();

         //This is our main loop and reads data until this task is cancelled
         while (true)
         {
             if (ct.IsCancellationRequested)
             {
                 //Cancel it
                 CleanUp();
                 ct.ThrowIfCancellationRequested();
             }

             //Read the data from the device
             TDeviceData data = ReadData();

             //Fire event
             OnDataReceived(data);
         }

    }, ct);
...
}

One further remark: In case you don’t already know, but cancellation of a task is accomplished with CancellationTokens that are created from a CancellationTokenSource. In the Stop() method actual cancellation is done by calling the Cancel() method on the CancellationTokenSource from which the CancellationToken is created. In the task an exception will be thrown which actually cancels and stops the task. See here for more info about task cancellation.

The continuation task executes only when its predecessor (our readerTask) has been cancelled. In this case a message is printed to the console, but you can write this message to a log file or event log or whatever. It also notifies all the subscribers that the monitor has stopped

 this.readerTask.ContinueWith(t =>
 {
   Console.WriteLine("Read task stopped");
   OnStatusChanged(MonitorStatus.STOPPED);
 }, TaskContinuationOptions.OnlyOnCanceled);

Example implementation

I made a sample implementation to put the theory from above in practice. You can find it in the TimerMonitor class. It’s just an amazingly unwieldy way to display the current time, but hey I just want to show a simple example.
I created also a sample console application which runs the TimerMonitor.

Well, I hope you have learned from this post and found it useful. Stay tuned for Part 2 of this series in which I add exception handling.
You can have a sneak preview when you look at the Part 2 project in the solution

Happy coding!