Python Useful Commands

Install Anaconda Python

Download anaconda. Get the latest link from https://www.anaconda.com/download/#linux

$ wget https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh

$ bash Anaconda3*_64.sh -b

Add anaconda to $PATH

$ echo "export PATH=$HOME/anaconda3/bin:\$PATH" >> ~/.bashrc

$ source ~/.bashrc


Jupyter

Jupyter comes with anaconda distribution. Configure jupyter

$ jupyter notebook --generate-config

Edit the vi ~/.jupyter/jupyter_notebook_config.py. This set the login password for jupyter to "pass123".

from os.path import expanduser

c.NotebookApp.ip = '0.0.0.0'

c.NotebookApp.notebook_dir = expanduser("~") + '/workspace/'

c.NotebookApp.open_browser = False

c.NotebookApp.password = u'sha1:298cdb938f91:e6462632083df74cb2f68d4b55b0e7a7d7e0e45b'

If you want to set your own password, generate password of your choice and replace the above password string.

$ python3 -c "from notebook.auth import passwd; print(passwd())"


Run Jupyter as service

Create bash file file that starts jupyter. Here I am using /opt/anaconda3/bin/start_jupyter.sh Configure as needed.

#!/usr/bin/env bash

/opt/anaconda3/bin/jupyter notebook

Give executable permission

$chmod +x /opt/anaconda3/bin/start_jupyter.sh

Create a service file

$ sudo vi /etc/systemd/system/jupyter.service.

Content of the file is a below. Update the ExecStart and User as per your requirement.

[Unit]

Description=Jypyter notebook service

[Service]

User=abasar

WorkingDirectory=/home/abasar/workspace

#path to executable.

#executable is a bash script which calls jar file

ExecStart=/opt/anaconda3/bin/start_jupyter.sh

SuccessExitStatus=143

TimeoutStopSec=10

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-user.target


Activate the service and

$ sudo systemctl daemon-reload

$ sudo systemctl enable jupyter.service

$ sudo systemctl start jupyter

$ sudo systemctl status jupyter

Setup log

$ sudo journalctl --unit=jupyter

Tail the logs

$ sudo journalctl -f -u jupyter


Run jupyter notebook from terminal command

$ jupyter nbconvert --execute example.ipynb


Install Xgboost on mac

Install Homebrew

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install gcc

brew install gcc6 --without-multilib

Clone xgboost

git clone --recursive https://github.com/dmlc/xgboost

cd xgboost

Open make/config.mk and uncomment these two lines

export CC = gcc-6

export CXX = g++-6

cp make/config.mk .

make -j4

sudo ./build.sh

cd python-package

sudo /home/<username>/anaconda/bin/python setup.py install


Install Tensorflow

$ pip install tensorflow tensorflow-tensorboard --upgrade --force-reinstall

$ pip install protobuf==3.4.0 --upgrade --force-reinstall


Python packaging

http://python-packaging.readthedocs.io/en/latest/minimal.html


Pipe shell command output to python script

$ curl -s 'https://api.github.com/users/lambda' | python3 -c "import sys, json; print(json.load(sys.stdin)['name'])"



Run a python script as a cron job

Example of a python script - poll_public_ip.py.

#!/opt/anaconda3/bin/python3

from urllib.request import urlopen

from boto3 import resource

from time import time

import json


# Find public ip of the current system

ip_address = json.load(urlopen('http://httpbin.org/ip'))["origin"]

file_name = "ip_address/" + ip_address


# Create a zero byte file in S3 with the public ip as name

s3 = resource("s3")

bucket = s3.Bucket("dev.einext.com")

bucket.put_object(Key = file_name)

Note that schebang (the first line) tells how to execute script as executable. Make changes as required. The above script runs on python3.

Give executable permission to the script

$ chmod +x poll_public_ip.py

Edit crobtab - to update the details

$ crontab -e

Add the following line to the file. This cron job runs every 15 minutes.

*/15 * * * * /home/user01/workspace/python/poll_public_ip.py

Crobjobs by default keep log in unix syslog.

View the crontask logs

$ tail -f /var/log/syslog

Sep 13 22:55:01 einext02 CRON[11419]: (user01) CMD (/home/user01/workspace/python/poll_public_ip.py)

Sep 13 23:00:01 einext02 CRON[11466]: (user01) CMD (/home/user01/workspace/python/poll_public_ip.py)

Sep 13 23:05:01 einext02 CRON[11500]: (user01) CMD (/home/user01/workspace/python/poll_public_ip.py)

Sep 13 23:07:08 einext02 crontab[11515]: (user01) BEGIN EDIT (user01)

Sep 13 23:07:23 einext02 crontab[11515]: (user01) REPLACE (user01)

Sep 13 23:07:23 einext02 crontab[11515]: (user01) END EDIT (user01)

Sep 13 23:08:01 einext02 cron[998]: (user01) RELOAD (crontabs/user01)

Sep 13 23:14:50 einext02 crontab[11566]: (user01) BEGIN EDIT (user01)

Sep 13 23:15:02 einext02 CRON[11579]: (user01) CMD (/home/user01/workspace/python/poll_public_ip.py)

Sep 13 23:16:34 einext02 crontab[11566]: (user01) END EDIT (user01)


Python Web Framework Benchmark

http://klen.github.io/py-frameworks-bench/



Simple Key Value Store using SQLite3


import sqlite3

import os.path


class KVStore(object):

def __init__(self, path:str = None):

if path is None or path is ":memory:":

path = ":memory:"

else:

if os.path.exists(path) and not os.path.isfile(path):

raise Exception("Path must be a valid sqldb file: " + path)

self.filename = path

self.conn = sqlite3.connect(path)

self.cursor = self.conn.cursor()

#cursor.execute('''DROP TABLE kvstore''')

self.cursor.execute('''CREATE TABLE IF NOT EXISTS kvstore

(key TEXT PRIMARY KEY, value BLOB)''')

self.conn.commit()

def get(self, key):

if type(key) is not str:

raise Exception("key must be str")

row = self.cursor.execute(

"select value from kvstore where key = ?", (key,)).fetchone()

if row is not None:

return row[0]


def delete(self, key):

if type(key) is not str:

raise Exception("key must be str")

row = self.cursor.execute(

"delete from kvstore where key = ?", (key,))


def select_all(self, limit:int = None):

statement = "select key, value from kvstore"

if limit is not None:

statement = "select key, value from kvstore limit " + str(limit)

result_set = self.cursor.execute(statement)

return result_set.fetchall()


def key_like(self, key, limit:int = None):

if type(key) is not str:

raise Exception("key must be str")

if limit is None:

statement = "select key, value from kvstore where key LIKE ?"

else:

statement = "select key, value from kvstore where key LIKE ? LIMIT " + str(limit)

return self.cursor.execute(statement, (key, )).fetchall()


def truncate(self):

return self.cursor.execute("delete from kvstore where 1>0")

def commit(self):

self.conn.commit()

def insert(self, pairs):

if type(pairs) is not list:

if type(pairs) is tuple and len(pairs) == 2:

pairs = [pairs]

else:

raise Exception("pairs must be list")


insert_statement = """

INSERT INTO kvstore (key, value) VALUES(?,?)

ON CONFLICT(key) DO UPDATE SET value = excluded.value

"""

self.cursor.executemany(insert_statement, pairs)

def size(self):

row = self.cursor.execute("select count(key) from kvstore").fetchone()

if row is not None:

return row[0]

def close(self):

self.conn.close()



db = KVStore()

db.insert([

("k1", "v1"),

("k2", "v2")

])

print("all records", db.select_all())

db.insert([("k3", "v3")])

db.insert([("k3", "v3")])

db.insert([("k3", "v3")])

print("k3 value", db.get("k3"))

db.delete("k3")

print("all records", db.select_all())

db.insert(("k3", 5))

print("k3", db.get("k3"))

print("k4", db.get("k4"))

print("k", db.key_like("k%", 2))

print("all", db.select_all())