How to checkout repo contents as it was on a given date in git

A bit tricky, but this does it:

git checkout `git rev-list -n 1 --before="2009-07-27 13:37" master`

Strage error with gnupghome parameter for pgp

I am using them pgp module for encryption in a python app.

It runs fine on the customers machine, and also on our cloud-based Jenkins server.

But it fails on my new development machine with:

init() got an unexpected keyword argument 'gnupghome'

This happening in this line of code:

        gpg = gnupg.GPG(gnupghome=self.TMP_DIR + '/pgp')

In a previous version of this code, this was homedir. I am going to try that again.

An Auto-Status Generator for Software Projects

As a software project manager, I often have to create status reports about software projects.

In a good software process, status reports should not really be necessary. The on-line tools, from github to JIRA, can already provide a great deal of really useful information on project progress. But I can never get any of my bosses to ever look at them. ;-)

(I also suspect that they rarely or never look at the status reports, beyond the first sentence.)

So why not have a little python program that will automatically generate my status report?

Here's the requirements:

• Takes a date and gets git commit comments for that day.
• Can do whole project or just one user.

Clearing up Python Import Confusion

Today I got the familiar Import not found error from Python:

ImportError: No module named mime.multipart

This is complaining about an import here:

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email import encoders

Here's a good way to figure out what went wrong:

Python 2.7.12 (default, Sep 29 2016, 13:30:34)
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import email
>>> dir(email)
['Charset', 'Encoders', 'Errors', 'FeedParser', 'Generator', 'Header', 'Iterators', 'LazyImporter', 'MIMEAudio', 'MIMEBase', 'MIMEImage', 'MIMEMessage', 'MIMEMultipart', 'MIMENonMultipart', 'MIMEText', 'Message', 'Parser', 'Utils', '_LOWERNAMES', '_MIMENAMES', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_name', 'base64MIME', 'email', 'importer', 'message_from_file', 'message_from_string', 'mime', 'quopriMIME', 'sys']
>>> dir(email.mime)
['Audio', 'Base', 'Image', 'Message', 'Multipart', 'NonMultipart', 'Text', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

I see from this that I should be importing email.mime.Multipart (not the upper-case M).

But when I try that, it also does not work:

>>> from email.mime.Multipart import MIMEMultipart

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: No module named Multipart

Turns out the problem was my module name, email.py. This caused some conflicts in python. I changed it to status_email.py and now everything works.

Handling Passwords in Python Code

One problem with dealing with databases is that we need a user name and password. We don't want to check those into version control, or code them into the program, so how to handle them?

One simple method is to have the user set environment variables with their user name and password, and have the python code read those vars and use their values. That way, your user name and password don't have to be checked into the repo.

So I changed my database connect code to this:

    # Open database connection
    db = MySQLdb.connect("localhost", os.environ['DB_USER'], os.environ['DB_PASSWORD'],
                         "tempdb" )

Now I need to define environment vars DB_USER and DB_PASSWORD before running the code.

Create a Database in MySQL

Now that I've got MySQL set up and running, it's time to use it. But when I try to connect I get an error because there is no database.

So to create a database I connect to the MySQL server:

[ed@localhost ~]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 10.1.17-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database tempdb;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]>

Unfortunately, when I try to connect from Python, I still get this error:

   OperationalError: (1049, "Unknown database 'testdb'")

So I list the databases MySQL knows about:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| tempdb             |
| test               |
+--------------------+
5 rows in set (0.07 sec)

The code I am using looks like this:

    # Open database connection

    db = MySQLdb.connect("localhost","root","xxx","testdb" )

    # prepare a cursor object using cursor() method

    cursor = db.cursor()

    # execute SQL query using execute() method.

    cursor.execute("SELECT VERSION()")

    # Fetch a single row using fetchone() method.

    data = cursor.fetchone()

    print "Database version : %s " % data

    # disconnect from server

    db.close()    

OK, the problem was that the name I created was "tempdb" but the name I was using in Python code was "testdb". I changed it to tempdb, and now the code works!! ;-)

Starting MySQL on Fedora

How do I get MySQL service to start? I don't know!

[root@localhost ed]# service myseld start
Redirecting to /bin/systemctl start  myseld.service
Failed to start myseld.service: Unit myseld.service not found.

I found this web page that has instructions.

Well, first I had to install a package!

 yum install mysql-server

I had a hard time from here, but finally this command worked:

[root@localhost ed]# service mariadb start

Redirecting to /bin/systemctl start  mariadb.service

The I could do:

[root@localhost ed]# mysqladmin -u root password ****

I can also connect to the server:

[root@localhost ed]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.1.17-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

In order to get the database server to start every time, I tried:

 systemctl enable mariadb.service

Using MySQL in a Python Program

Now that I've got MySQL install, and python-MySQL installed in python, it's time to use my database from a python script.

I found a good tutorial here.

When I tried this code, I got an error:

    # Open database connection
    db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )

    # prepare a cursor object using cursor() method
    cursor = db.cursor()

    # execute SQL query using execute() method.
    cursor.execute("SELECT VERSION()")

    # Fetch a single row using fetchone() method.
    data = cursor.fetchone()

    print "Database version : %s " % data

    # disconnect from server
    db.close()  

  OperationalError: (2002, 'Can\'t connect to local MySQL server through socket \'/var/lib/mysql/mysql.sock\' (2 "No such file or directory")')

Getting mySQL to use in Python (on Fedora)

In a project I am working on, we need a database, and we have chosen mySQL.

Python has a standard API for database interaction, the DB-API. It works with many databases including mySQL.

I had a hard time getting this working. I tried a lot of things, but here's what worked.

First I installed the mysql-devel package:

 sudo yum install mysql-devel

The I did:

sudo pip install MySQL-python

Now I can do an import MySQLdb, and it works!

Setting up a Jenkins Feature Build

I want a Jenkins feature build, which will build whenever a new branch is added to the repo in github, or updated.

That way, as I develop on branches, Jenkins will pick up the changes and build them for me.

First I made a copy of my already existing master build job.

To build a feature branch, I just leave the branch line blank in the configuration of the jenkins job. Now it builds any newly modified branch. Nice!

One problem I had was that I accidently had a space in the name of the Jenkins job. Since this is used as a directory name, a space causes problems. I took the space out and now it works fine.

Setting up Python Warnings on Jenkins

Using pylint is a great way to help code quality. And the best way to use pylint is to integrate it with your continuous integration server.

Jenkins is the CI server I use, and it has a pylint plug-in  that I needed to install.

For that I go to "Manage Jenkins" on the menu, and select "Manage Plug-ins". Before I can do anything, Jenkins asks me to install updates for a bunch of plugins I already have installed, and I do that, because why not?

I installed the cobertura plug-in, because I will need it later for code coverage reports. But the plugin I need for warnings is called the violations plugin.

There is a good blog post describing this procedure.

The only tricky part was that my coverage.xml was being generated in the tests directory. It's necessary to put the path of this file into Jenkins configuration.

However, now that it is working, I have nice charts of warnings and code coverage on my Jenkins page!



My code coverage is about 40%. Not good!



I have 18 pylint warnings, which is a lot for the tiny amount of code that I checked in.

Setting Up a Jenkins Master Build

I'm starting on a new project, so of course one of the first things to do is to set up the Jenkins continuous integration server.

First I'm going to set up a build of the master branch. For this I select "new item" on the Jenkins menu, and select "Freestyle Project." For the name of the item, use a short and simple name without spaces, like "project_master".

Under source code management, I put in the git URL, and my github user name and password. By default it wants to build the master branch, so I leave that setting alone.

For build trigger, I select "Build when change is pushed to github."

For build steps, I execute the build script, build.sh. This is a script that exists in my repo and will build and run tests. I have to execute the build script like this:

bash ./build.sh

Now it works and Jenkins shows success!

Mocking S3 Storage for Python Tests

When using Amazon S3 cloud storage, it's helpful to mock the S3 service for testing. That way, test can run without actually needing to contact S3 and move data in and out of it.

Fortunately there is a very helpful S3 mocking library available for python called moto.

To use it:

from moto import mock_s3

    # Mock s3 to create that bucket and path.
    moto = mock_s3()
    moto.start()                # begin 'mock mode'
    conn = boto.connect_s3()    # mock connection
    bucket = conn.create_bucket(s3bucket)

    # Now mock the lockfile.
    key = Key(bucket)
    key.key = s3path
    test_data = base_dir + '/data/client1.json'
    key.set_contents_from_filename(test_data)

    # Now check that (mocked) file is there.
    result = main.lock_exists(s3bucket, s3path)
    assert result == True

    # End mock mode.
    moto.stop()

Pytest Fixtures

When setting up automated testing, I use pytest. It has an interesting feature called fixtures.

The idea with fixtures is that you define a function, labeled as a fixture, and then you can pass that function as an argument to tests. The function will be executed at the beginning of the test.

For example, this function is a fixture:

@pytest.fixture(scope="function")
def testdir():
    # always run in test dir
    os.chdir(os.path.dirname(os.path.realpath(__file__)))

To run this function at the beginning of the test, just pass testdir as a parameter to the test:

 def test_run_no_history(testdir, tmpdir, s3_no_history_file, input_file, expected_output, config_file):
    base_dir = str(tmpdir.chdir())

By passing testdir as the first argument, I cause pytest to run the function at the beginning of the test.

Using Google App Engine to Create a Website

First log on to the google app engine console.

From "Project" menu at top of screen, select "Create New Project." Specify a project name. It takes a minute or two for the project to be created.

How to Install Android Studio on Fedora

I have a Fedora system and I need to get Android Studio installed. Here's what I do (following these instructions):

sudo dnf install devassistant

 da pkg install android-studio

da crt android-studio --name my_test

 This uses devassistant. This is a free software program developed by some Red Hat people to help set up development environments.

Unfortunately, it didn't work for me. I got:

INFO: Resolving RPM dependencies with DNF...
Installing 10 RPM packages with DNF. Is this ok? [y(es)/n(o)/s(how)]: y
Installing dependencies ................ Done.
INFO: Android studio was not found on system.
Downloading android studio takes a time. Do you want to continue?
Select [y/n]
y
INFO: Downloading android studio from https://dl.google.com/dl/android/studio/ide-zips/1.3.0.10/android-studio-ide-141.2117773-linux.zip
INFO: Downloading done
INFO: Downloading android SDK from http://dl.google.com/android/android-sdk_r24.3.3-linux.tgz
INFO: Downloading done
INFO: Copy android template project to project destination
INFO: .
[devassistant]$ cp -r /home/ed/.devassistant/files/crt/android-studio/. "my_test"
ERROR: no such snippet: init_add_commit
[ed@localhost ~]$ 

So that's a bummer.

However, it did unpack the android studio tarball into a directory, and by going there and into the bin directory, and running studio.sh, I get a running android studio.

As for devassistant I think I will leave it aside for now and move on to other things. ;-)

WiFi Problems

Android Studio has it's own update manager, and it is trying to download a new version of the SDK (I believe), but it keeps killing the wifi. The download start, but then the wifi melts down and disconnects. The wifi works fine once I disconnect and reconnect it. (I am using a USB wifi for this machine.)

Starting Over

I decided to start over, without devassistant. I went to the android studio download page and hit the download button.

On the install page it asked me to install some packages, so I did:

sudo yum install zlib.i686 ncurses-libs.i686 bzip2-libs.i686

But when I run studio.sh, I get a bunch of java errors.

I think this is a OpenJDK issue, so I am going to try installing Oracle Java.

Installing Oracle Java

I got the correct file from the Oracle Java download page. Double-click on thed rpm and follow the prompts to install.

But in my case, I found it was already installed. OK, that's cool, but why is android studio using OpenJDK?

I really don't need openJDK. I would prefer to have just one java, and no matter what free software purists might say, I want it to be the Oracle java. So I removed OpenJDK according to some intructions here.

[ed@localhost Downloads]$ rpm -qa | grep java

devassistant-dap-java-0.11.1-3.fc24.noarch

javapackages-tools-4.6.0-14.fc24.noarch

tzdata-java-2016f-1.fc24.noarch

java-1.8.0-openjdk-headless-1.8.0.102-1.b14.fc24.x86_64

python3-javapackages-4.6.0-14.fc24.noarch

java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64

abrt-java-connector-1.1.0-8.fc24.x86_64

java-1.8.0-openjdk-devel-1.8.0.102-1.b14.fc24.x86_64

[ed@localhost Downloads]$ rpm -qa | grep jdk

jdk1.8.0_101-1.8.0_101-fcs.x86_64

java-1.8.0-openjdk-headless-1.8.0.102-1.b14.fc24.x86_64

java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64

copy-jdk-configs-1.2-1.fc24.noarch

java-1.8.0-openjdk-devel-1.8.0.102-1.b14.fc24.x86_64

But when I tried to remove it, it wanted to remove a bunch of other packages that depend on it, like openoffice. I need those packages so I canceled out of that.

[ed@localhost bin]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64/jre/bin/java)
 + 2           /usr/java/jdk1.8.0_101/jre/bin/java

Enter to keep the current selection[+], or type selection number: 2

However, no matter what I do, android studio still seems to end up using openJDK

[ed@localhost bin]$ ./studio.sh
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=350m; support was removed in 8.0
Looking in classpath from com.intellij.util.lang.UrlClassLoader@28c97a5 for /com/sun/jna/linux-x86-64/libjnidispatch.so
Found library resource at jar:file:/home/ed/Downloads/android-studio/lib/jna.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
Trying /home/ed/.AndroidStudio2.2/system/tmp/jna7490874677544946085.tmp
Found jnidispatch at /home/ed/.AndroidStudio2.2/system/tmp/jna7490874677544946085.tmp
[   4012]   WARN - s.RepoProgressIndicatorAdapter - File /home/ed/.android/repositories.cfg could not be loaded.
[   5834]   WARN - dea.updater.SdkComponentSource - File /home/ed/.android/repositories.cfg could not be loaded.

However, I deleted the previous android-studio directory, and started over. Now it is downloading stuff.

Build script for Python project on Jenkins using virtualenv

Here's a project build script to use in Jenkins:

PYENV_HOME=$WORKSPACE/.pyenv/
# Delete previously built virtualenv
if [ -d $PYENV_HOME ]; then
    rm -rf $PYENV_HOME
fi
# Create virtualenv and install necessary packages
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate
pip install --quiet httplib2 --upgrade --force-reinstall
pip install --quiet pytest
pip install --quiet pytest-cov
pip install --quiet pylint
python $WORKSPACE/setup.py install
python $WORKSPACE/setup.py develop
pylint -f parseable $WORKSPACE/ats_processor/ | tee pylint.out
py.test $WORKSPACE --junitxml test_results.xml
py.test --cov=$WORKSPACE/ats_processor --cov-report term-missing --cov-report xml

The point of this script is that it uses virtualenv to create a self-contained python execution environment, including using pip to install all necessary packages. This ensures that you are testing with the latest version of each package.

Stripping empty lines from a CSV file with python

In some processing I have to do, the comma-separated value (csv) files I'm using occasionally have a line of empty fields.

For example, instead of:

a,b,c,d,e

I get:

,,,,

Sometimes I get fewer commas, like:

,,

So how to strip these lines out of the file?

I have a function like this:

    @staticmethod
    def row_has_data(row):
        '''
        Detect whether a row in a csv file has data, or is blank.
        '''
        has_data = False
        for col in range(0, len(row)):
            if row[col] != '':
                has_data = True
        return has_data

Then I copy the csv file like this:

    def copy_to_csv(self, infile, has_header, outfile):
        '''
        Copy a file to a csv output file, skipping header if present.
        '''
        csv_writer = csv.writer(outfile, dialect='ats_global_dialect', encoding='UTF-8')
        infile.seek(0)
        csv_data = csv.reader(infile)
        if has_header:
            csv_data.next()  # skip header row
        for row in csv_data:
            if self.row_has_data(row):
                csv_writer.writerow(row)
        outfile.close()

Python and virtualenv

virtualenv is a tool to create isolated python environments.

I am using virtualenv to package up an installation of a python application for testing in the continuous integration server, Jenkins.

To install virtualenv:

sudo pip install virtualenv

To use virtualenv:

PYENV_HOME=.pyenv/
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate

From then on, everything is in that environment until the deactivate command is issued:

(.pyenv) [ed@localhost ~]$ deactivate

In order to ensure that the application will work on any production machine, the complete set of requirements are loaded by a build script, which launches the python setup.py for the application.

There is a good discussion of virtualenv in the Hitchhikers Guide to Python.

See the vittrualenv docs for more info.

Imports and sys.path

Python finds imported packages by looking at the directories listed in the sys.path.

To check the sys.path:

[ed@localhost ~]$ python -m site
sys.path = [
    '/home/ed',
    '/usr/lib/python27.zip',
    '/usr/lib64/python2.7',
    '/usr/lib64/python2.7/plat-linux2',
    '/usr/lib64/python2.7/lib-tk',
    '/usr/lib64/python2.7/lib-old',
    '/usr/lib64/python2.7/lib-dynload',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages/gtk-2.0',
    '/usr/lib/python2.7/site-packages',
    '/usr/lib/python2.7/site-packages/setuptools-26.1.1-py2.7.egg',
]
USER_BASE: '/home/ed/.local' (exists)
USER_SITE: '/home/ed/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

Or, from within python:

[ed@localhost ~]$ python
Python 2.7.12 (default, Sep  2 2016, 14:46:00)
[GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, pprint
>>> pprint.pprint(sys.path)
['',
 '/usr/lib/python27.zip',
 '/usr/lib64/python2.7',
 '/usr/lib64/python2.7/plat-linux2',
 '/usr/lib64/python2.7/lib-tk',
 '/usr/lib64/python2.7/lib-old',
 '/usr/lib64/python2.7/lib-dynload',
 '/usr/lib64/python2.7/site-packages',
 '/usr/lib64/python2.7/site-packages/gtk-2.0',
 '/usr/lib/python2.7/site-packages',
 '/usr/lib/python2.7/site-packages/setuptools-26.1.1-py2.7.egg']

How python builds the sys.path is surprisingly complicated. A brief description can be found on Lee on Coding blog post. A more detailed description can be found in this Stackoverflow answer. Also take a look at the python sys.path docs.

The sys.path is put together at load-time by the automatically imported python site module.

The site module evaluates any .pth files it can find