Friday, April 11, 2014

Non capturing group in regular expression and it's use

Recently I have been stuck in a regular expression where I had to use grouping in places, values of which were not required. It won't make sense to explain the problem without the text. So here it is

Controller Id Connection Status Connection State Secure Role
------------- ----------------- ---------------- ------ ------
1             Connected         Active           No     Slave
25            Disconnected      Idle             No     Master

My goal was to get the numeric values at the beginning of the line while still matching the whole line for safety (so that any other numeric values in the output don't match). As you see in the output, each column can contain a value from a list of fixed values. e.g. Connection Status can only contain Connected or Disconnected. That demands the following regular expression -


But with the above expression, we are fetching all the columns when all we want is the first column. This is where the non capturing group of regular expression comes into picture. With the help of non capturing groups we can remove them from the matched output while still using the grouping functionality.

Corrected regular expression is as below -


By using the "?:" combination just after the parenthesis, we can instruct regular expression engine to not include that group in the matched output. This can be used anywhere we want to use grouping but the value is not needed in the matched output.

Sunday, February 17, 2013

Simple employee table with OpenStack Horizon DataTable module

I was working on OpenStack dashboard (aka Horizon) few past few months and I will share a simple way to build your data tables based on Horizon.
To give a brief introduction of Horizon, it is the official dashboard project of OpenStack, the cloud operating system. Horizon is a python Django application. It provides pre-defined modules for common web application requirements like tabs, workflow, table etc. From Grizzly release onward Horizon modules are separated from OpenStack specific code. It allows one to use Horizon modules independently.

To build a data table in Horizon you need to simply create a model class for your data. The important thing to remember here is that the class must have an "id" attribute. Horizon will build rows based on this id. It can be alpha numeric also. Though it is not mandatory to display the id column in the table, you can still have it. Below is the data model for my Employee table -

class Employee:
    Employee class represents the employee data

    def __init__(self, emp_id, name, addr, salary): = emp_id = name
        self.addr = addr
        self.salary = salary
The employee class has 4 attributes - id, name, address and the salary. Each of these attributes can be displayed in a column of our Horizon DataTable. I use below code in my

from horizon import tables
from .tables import TabledisplayTable

class IndexView(tables.DataTableView):
    table_class = TabledisplayTable
    template_name = 'project/tabledisplay/index.html'

    def get_data(self):
        strobj = '[{"id": 111, "name": "emp1", "salary": 1000, "addr": "kolkata"}, {"id": 222, "name": "emp2", "salary": 5000, "addr": "bangalore"}]'
        instances = json.loads(strobj)
        ret = []
        for inst in instances:
            ret.append(Employee(inst['id'], inst['name'], inst['addr'], inst['salary']))
        return ret

We need to create a with TabledisplayTable class which allows us to map the columns to model attributes. Like this -

from horizon import tables
from django.utils.translation import ugettext_lazy as _

class TabledisplayTable(tables.DataTable):
    name = tables.Column("name",
                         verbose_name=_("Employee Name"))
    salary = tables.Column("salary", verbose_name=_("Salary in Rs"))
    addr = tables.Column("addr",

    class Meta:
        name = "emptable"
        verbose_name = _("Employee Table")

Finally we need a Django template to render this table. I have used the following code -
{% extends 'base.html' %}
{% load i18n %}
{% block title %}{% trans "Table Display" %}{% endblock %}

{% block page_header %}
  {% include "horizon/common/_page_header.html" with title=_("Employee Table Display") %}
{% endblock page_header %}

{% block main %}
  {{ table.render }}
{% endblock %}

This should create the Employee table with 2 rows using Horizon DataTable module. Try it out.

Saturday, February 25, 2012

Get current UTC timestamp in Python

Converting current time to UTC (GMT) in python is a very common query that is searched in Google but, I did not find a compact answer when I did the search for my work. I figured out the code by some help from Google and some experimentation of my own. I'll share the code, in case it is useful to someone.

First I'll explain the Epoch time (or POSIX/Unix time) and UTC (Coordinated Universal Time) time format.
Epoch time is calculated as number of seconds elapsed since midnight Jan 1, 1970. It is used in all *nix systems and has become a standard when considering date/time from multiple systems. In python time module allows us to get the current epoch time in local time zone.

import time

epoch_time_in_local_timezone = time.time()

UTC time is basically synonymous of GMT. It marks the 0 offset time zone. Time in all other time zones are calculated with positive/negative offset from UTC. It is used in many cases where there are multiple machines, located in different time zones are involved. To get current UTC time in python we use utcnow() function from datetime module. But, if you need the time in seconds you need to do the following -

import calendar, datetime

current_utc_time_in_sec = calendar.timegm(datetime.datetime.utcnow().utctimetuple())

You can match the correctness of generated time in by converting Human date to timestamp in GMT format.

Sunday, January 15, 2012

Upgrade wireshark version in Ubuntu

Wireshark is a great and may be most widely used network protocol analyzer. In last few months I had used it extensively in a network project. Wireshark was a great help in debugging hard to find protocol problems.

During this project I was using Ubuntu 10.04 as my development environment. I installed wireshark using
sudo apt-get install wireshark

But default wireshark version provided by Ubuntu 10.04 repository is 1.2.7 which was not suiting my purpose. I needed the newer versions of wireshark which provides some important features for analyzing protocols.

There are 2 ways to upgrade your wireshark version from the default version provided by Ubuntu -

  • Add a third party repository to your repo conf and you can upgrade the wireshark. But I do not prefer this way as it imposes the restriction of version that can be installed.
  • Install wireshark from source. This is best way as you can install any version of wireshark in your Ubuntu.

Here are the steps to follow -

  1. Download the wireshark source code as per your preferred version.
  2. Extract the source.
  3. Building wireshark from source has multiple dependencies e.g. GTK2+, bison etc. To know the dependencies required by your wireshark version, run below command inside the extracted source directory -    dpkg-checkbuilddeps
  4. The output of dpkg-checkbuilddeps command will list down the required libraries for building wireshark. Use sudo apt-get install to install all dependencies.
  5. Once all libraries are installed configure wireshark (prefix option installs it in specific directory) -   ./configure --prefix=/home/myuser/wireshark-custom
  6. Run    make install.
  7. Once the build is done, you can use wireshark just as out of the box.

Sunday, December 25, 2011

Call an external program from MySQL trigger

MySQL is the choice of many when it comes to database. Its free and quite robust.
During one of our prototype implementations we had a requirement of calling some external processes when there is a change in a MySQL table. MySQL triggers are provided for the same purpose. They get executed when the table is changed in certain ways that is specified by the programmer. Now it is very easy (rather trivial) to do some thing in other MySQL database tables when trigger gets fired. But, our requirement was to call a C program.

Fortunately MySQL provides a way to implement your own functions, its called User Defined Functions (UDF). The "how to" is here. Now that we know, how to define your own functions and call them from MySQL events, we need to write our logic in a C program by following the interface provided by MySQL and we are done.

Wait a minute. That is already done by somebody. They have made a library of UDFs. One of them, LIB_MYSQLUDF_SYS does exactly what we want. It allows you to call an external program from MySQL procedure/trigger. Once you download and untar the files, you'll get an Just need to run it but, make sure you have gcc and make before that. The install script copies the library shared object file to /usr/lib. In some 64 bit systems it may be /usr/lib64. Also you need to check MySQL plugin_dir property which points to the directory where the .so file should be placed.

Once the UDF is installed successfully, we can use 2 functions to call an external program -

1. sys_exec : Returns the exit code of the external program.
2. sys_eval : Returns the output from the external program.

Here is a code snippet which show an example:


 DECLARE cmd CHAR(255);
 DECLARE result int(10);
 SET cmd=CONCAT('sudo /home/sarbac/hello_world ','Sarbajit');
 SET result = sys_exec(cmd);

I have a hello_world program (/home/sarbac/hello_world) which accepts 1 command line argument and writes to a file.
Don't forget to return int from the main function if you are using sys_exec().

Deepti Sharma has thoroughly investigated this approach. I thank her for that.

Friday, November 11, 2011

Read a log file in tail mode from python

When you want to develop a log reader application, you'll need to keep reading the log. There is a Linux utility doing exactly the same. Its the tail command. tail blocks on the file till new entries are appended to it.

I needed the similar functionality in python and there was a small library providing it already. Its called The program performs tail functionality and also handles log rotation.

But in my case, the log file used to change (file name) depending on the date. filetail could not handle it as the file name has changed already. To handle this I have made following changes to the filetail code -

In nextline() method file size should be checked to be less or equal to the cursor position:

                # maybe the log got rotated out from under us?
                if stat(self.path)[ST_SIZE] <= self.pos: # modified by sarbajit
                    # file got truncated and/or re-created

In _reset() method I have added the logic to re-create the log file name from the current date value and replace it with the original file name:

    def _reset(self):
        """Internal method; reopen the internal file handle (probably
        because the log file got rotated/truncated)."""

 # modified by sarbajit
 localtime1 = localtime(time())
 year = str(localtime1[0])
 if localtime1[1] < 9:
  month = "0"+str(localtime1[1])
  month = str(localtime[1])
 if localtime1[2] > 9:
  date = str(localtime1[2])
  date = "0"+str(localtime1[2])
 logfile_name = "applog."+year+"-"+month+"-"+date+".log"
 self.path = "/var/log/" + logfile_name
 # end of modification

        self.f = open(self.path, "r")
        self.pos = self.f.tell()
        self.last_read = time()

I apologize for not following standards of coding here. It served the purpose for me.

Sunday, October 16, 2011

jQuery UI dialog without any close option

We use jQuery UI for designing web pages in out project. Some times back we had a scenario where the user log in to a dashboard was implemented using jQuery UI dialog (modal) component. Requirement was that the dashboard page should not be reached without passing the authentication process. But the dialog component of jQuery UI library provides a default X icon at the top left corner of the element which provides an option to close the dialog.

In our case, we wanted to reload the same page on click of the X icon. So that the user can not escape the login dialog by using the X icon. We achieved this by writing the page reload code inside dialog's close handler.  Everything worked fine until we had to handle the successful log in. Once user login succeeds we need to call dialog widget's close handler which reloads the page in turn. Because of this, even after a successful login, user gets the login dialog again.

The only way to stop this was, to remove the page reload code from the widget's close handler. Then it would require to remove the X icon from the dialog widget so that the close handler can not be invoked by clicking the X icon. But jQuery UI gives no option to build a dialog without close icon.

Thanks to Deepti Sharma for finding out a easy solution to the problem. Here is how it looks after the modification -

Below is the HTML code for creation of the dialog box.
<div id="login_div" title="Login" style="display:none;">
  <form id="login_form" name="login_form" action="LoginServlet?action=login" method="post">
     <td>Username: </td>
     <td><input type="text" name="username" id="username" size="20"></td>
     <td >Password:</td>
     <td><input type="password" name="password" id="password" size="20"></td>

The JavaScript part of code is doing the actual business. First we create the login dialog using jQuery UI API, then we hide the X icon by setting its display property. Here is the code
$( "#login_div" ).dialog({
  autoOpen: false,
  height: 'auto',
  width: 400,
  modal: true,
  buttons: {
  "Submit": function() {
                  // form submission code goes here
  "close": function() {
                  // close handler

var dialogDivId = "login_div";

$('div[aria-labelledby="ui-dialog-title-'+ dialogDivId +'"] .ui-dialog-titlebar-close').css("display","none");

The important part is the aria-labelledby attribute. It will contain the original div id of the dialog. So, we can find out the correct X icon to hide (there are multiple dialog boxes in the page). Though the code looks dirty, it works.
Once the close icon is removed, user can only submit the login page but can't close/cancel the login.