An Auto-Status Generator for Software Projects

As a software project manager, I often have to create status reports about software projects.

In a good software process, status reports should not really be necessary. The on-line tools, from github to JIRA, can already provide a great deal of really useful information on project progress. But I can never get any of my bosses to ever look at them. ;-)

(I also suspect that they rarely or never look at the status reports, beyond the first sentence.)

So why not have a little python program that will automatically generate my status report?

Here's the requirements:

• Takes a date and gets git commit comments for that day.
• Can do whole project or just one user.

Clearing up Python Import Confusion

Today I got the familiar Import not found error from Python:

ImportError: No module named mime.multipart

This is complaining about an import here:

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email import encoders

Here's a good way to figure out what went wrong:

Python 2.7.12 (default, Sep 29 2016, 13:30:34)
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import email
>>> dir(email)
['Charset', 'Encoders', 'Errors', 'FeedParser', 'Generator', 'Header', 'Iterators', 'LazyImporter', 'MIMEAudio', 'MIMEBase', 'MIMEImage', 'MIMEMessage', 'MIMEMultipart', 'MIMENonMultipart', 'MIMEText', 'Message', 'Parser', 'Utils', '_LOWERNAMES', '_MIMENAMES', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_name', 'base64MIME', 'email', 'importer', 'message_from_file', 'message_from_string', 'mime', 'quopriMIME', 'sys']
>>> dir(email.mime)
['Audio', 'Base', 'Image', 'Message', 'Multipart', 'NonMultipart', 'Text', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

I see from this that I should be importing email.mime.Multipart (not the upper-case M).

But when I try that, it also does not work:

>>> from email.mime.Multipart import MIMEMultipart

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: No module named Multipart

Turns out the problem was my module name, email.py. This caused some conflicts in python. I changed it to status_email.py and now everything works.

Handling Passwords in Python Code

One problem with dealing with databases is that we need a user name and password. We don't want to check those into version control, or code them into the program, so how to handle them?

One simple method is to have the user set environment variables with their user name and password, and have the python code read those vars and use their values. That way, your user name and password don't have to be checked into the repo.

So I changed my database connect code to this:

    # Open database connection
    db = MySQLdb.connect("localhost", os.environ['DB_USER'], os.environ['DB_PASSWORD'],
                         "tempdb" )

Now I need to define environment vars DB_USER and DB_PASSWORD before running the code.

Create a Database in MySQL

Now that I've got MySQL set up and running, it's time to use it. But when I try to connect I get an error because there is no database.

So to create a database I connect to the MySQL server:

[ed@localhost ~]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 10.1.17-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database tempdb;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]>

Unfortunately, when I try to connect from Python, I still get this error:

   OperationalError: (1049, "Unknown database 'testdb'")

So I list the databases MySQL knows about:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| tempdb             |
| test               |
+--------------------+
5 rows in set (0.07 sec)

The code I am using looks like this:

    # Open database connection

    db = MySQLdb.connect("localhost","root","xxx","testdb" )

    # prepare a cursor object using cursor() method

    cursor = db.cursor()

    # execute SQL query using execute() method.

    cursor.execute("SELECT VERSION()")

    # Fetch a single row using fetchone() method.

    data = cursor.fetchone()

    print "Database version : %s " % data

    # disconnect from server

    db.close()    

OK, the problem was that the name I created was "tempdb" but the name I was using in Python code was "testdb". I changed it to tempdb, and now the code works!! ;-)

Starting MySQL on Fedora

How do I get MySQL service to start? I don't know!

[root@localhost ed]# service myseld start
Redirecting to /bin/systemctl start  myseld.service
Failed to start myseld.service: Unit myseld.service not found.

I found this web page that has instructions.

Well, first I had to install a package!

 yum install mysql-server

I had a hard time from here, but finally this command worked:

[root@localhost ed]# service mariadb start

Redirecting to /bin/systemctl start  mariadb.service

The I could do:

[root@localhost ed]# mysqladmin -u root password ****

I can also connect to the server:

[root@localhost ed]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.1.17-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

In order to get the database server to start every time, I tried:

 systemctl enable mariadb.service

Using MySQL in a Python Program

Now that I've got MySQL install, and python-MySQL installed in python, it's time to use my database from a python script.

I found a good tutorial here.

When I tried this code, I got an error:

    # Open database connection
    db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )

    # prepare a cursor object using cursor() method
    cursor = db.cursor()

    # execute SQL query using execute() method.
    cursor.execute("SELECT VERSION()")

    # Fetch a single row using fetchone() method.
    data = cursor.fetchone()

    print "Database version : %s " % data

    # disconnect from server
    db.close()  

  OperationalError: (2002, 'Can\'t connect to local MySQL server through socket \'/var/lib/mysql/mysql.sock\' (2 "No such file or directory")')

Getting mySQL to use in Python (on Fedora)

In a project I am working on, we need a database, and we have chosen mySQL.

Python has a standard API for database interaction, the DB-API. It works with many databases including mySQL.

I had a hard time getting this working. I tried a lot of things, but here's what worked.

First I installed the mysql-devel package:

 sudo yum install mysql-devel

The I did:

sudo pip install MySQL-python

Now I can do an import MySQLdb, and it works!

Setting up a Jenkins Feature Build

I want a Jenkins feature build, which will build whenever a new branch is added to the repo in github, or updated.

That way, as I develop on branches, Jenkins will pick up the changes and build them for me.

First I made a copy of my already existing master build job.

To build a feature branch, I just leave the branch line blank in the configuration of the jenkins job. Now it builds any newly modified branch. Nice!

One problem I had was that I accidently had a space in the name of the Jenkins job. Since this is used as a directory name, a space causes problems. I took the space out and now it works fine.

Setting up Python Warnings on Jenkins

Using pylint is a great way to help code quality. And the best way to use pylint is to integrate it with your continuous integration server.

Jenkins is the CI server I use, and it has a pylint plug-in  that I needed to install.

For that I go to "Manage Jenkins" on the menu, and select "Manage Plug-ins". Before I can do anything, Jenkins asks me to install updates for a bunch of plugins I already have installed, and I do that, because why not?

I installed the cobertura plug-in, because I will need it later for code coverage reports. But the plugin I need for warnings is called the violations plugin.

There is a good blog post describing this procedure.

The only tricky part was that my coverage.xml was being generated in the tests directory. It's necessary to put the path of this file into Jenkins configuration.

However, now that it is working, I have nice charts of warnings and code coverage on my Jenkins page!



My code coverage is about 40%. Not good!



I have 18 pylint warnings, which is a lot for the tiny amount of code that I checked in.

Setting Up a Jenkins Master Build

I'm starting on a new project, so of course one of the first things to do is to set up the Jenkins continuous integration server.

First I'm going to set up a build of the master branch. For this I select "new item" on the Jenkins menu, and select "Freestyle Project." For the name of the item, use a short and simple name without spaces, like "project_master".

Under source code management, I put in the git URL, and my github user name and password. By default it wants to build the master branch, so I leave that setting alone.

For build trigger, I select "Build when change is pushed to github."

For build steps, I execute the build script, build.sh. This is a script that exists in my repo and will build and run tests. I have to execute the build script like this:

bash ./build.sh

Now it works and Jenkins shows success!

Mocking S3 Storage for Python Tests

When using Amazon S3 cloud storage, it's helpful to mock the S3 service for testing. That way, test can run without actually needing to contact S3 and move data in and out of it.

Fortunately there is a very helpful S3 mocking library available for python called moto.

To use it:

from moto import mock_s3

    # Mock s3 to create that bucket and path.
    moto = mock_s3()
    moto.start()                # begin 'mock mode'
    conn = boto.connect_s3()    # mock connection
    bucket = conn.create_bucket(s3bucket)

    # Now mock the lockfile.
    key = Key(bucket)
    key.key = s3path
    test_data = base_dir + '/data/client1.json'
    key.set_contents_from_filename(test_data)

    # Now check that (mocked) file is there.
    result = main.lock_exists(s3bucket, s3path)
    assert result == True

    # End mock mode.
    moto.stop()

Pytest Fixtures

When setting up automated testing, I use pytest. It has an interesting feature called fixtures.

The idea with fixtures is that you define a function, labeled as a fixture, and then you can pass that function as an argument to tests. The function will be executed at the beginning of the test.

For example, this function is a fixture:

@pytest.fixture(scope="function")
def testdir():
    # always run in test dir
    os.chdir(os.path.dirname(os.path.realpath(__file__)))

To run this function at the beginning of the test, just pass testdir as a parameter to the test:

 def test_run_no_history(testdir, tmpdir, s3_no_history_file, input_file, expected_output, config_file):
    base_dir = str(tmpdir.chdir())

By passing testdir as the first argument, I cause pytest to run the function at the beginning of the test.

Using Google App Engine to Create a Website

First log on to the google app engine console.

From "Project" menu at top of screen, select "Create New Project." Specify a project name. It takes a minute or two for the project to be created.