Python - BKJackson/BKJackson_Wiki GitHub Wiki

Python Subpages

Effective Python for Data Science

Efficient Python Tricks and Tools for Data Scientists [Class mixins over inheritance for enhanced modularity] (https://khuyentran1401.github.io/Efficient_Python_tricks_and_tools_for_data_scientists/Chapter1/class.html#use-mixins-over-inheritance-for-enhanced-modularity) - basically a trick for adding data or model IO as a class that is inherited by your main model class

Python Automation

pyautomate - CLI tool to automate any set of tasks. You describe a state machine in a config file and use the CLI to go to a certain state. pyautomate will automatically find out how to get there using the config file you provided.

Python Fire - Make command line interfaces (CLIs) from any Python object

Python Fire - github
Fire Getting Started Guide
Example of calling Fire on a function:

import fire

def hello(name="World"):
  return "Hello %s!" % name

if __name__ == '__main__':
  fire.Fire(hello)

Then, from the command line, you can run:

python hello.py  # Hello World!
python hello.py --name=David  # Hello David!
python hello.py --help  # Shows usage information.

Setting and getting environment variables

Use the dotenv package, which allows you to easily store these variables in a file that is not in source control (as long as you keep the line .env in your .gitignore file!). You can then reference these variables as environment variables in your application with
os.environ.get('VARIABLE_NAME').
View .env with
print(open(os.path.join(PROJ_ROOT, '.env')).read())

Convenience function to get the absolute path to a file in a directory from that same file

def script_path(filename):
    """
    A convenience function to get the absolute path to a file in this
    tutorial's directory. This allows the tutorial to be launched from any
    directory.

    """
    import os

    filepath = os.path.join(os.path.dirname(__file__))
    return os.path.join(filepath, filename)

Accessing Documentation for Python Functions

help(pickle.dump)

The ? only works in ipython, not python3:

pickle.dump?

To view the source code use ??. Does not work with builtin functions or methods, generally because the object is implemented in a language other than Python, such as C.

somefunction??

Access a web API and print the json output to the console

# print_user_agent.py
import requests

json = requests.get('http://httpbin.org/user-agent').json()
print(json['user-agent'])

Lambda functions

id = lambda nmodes, t: int((nmodes * t[0]) + t[1])  

$id(nmodes, t)

Create a sorted list of tuples with sorted() and a lambda function

The returned data value is 2nd in a list. That is data[1] = pair[1].

genrelist = [(genre, median)\
              for genre, median\
              in sorted(data, key=lambda pair: pair[1])])

Broadcasting

a = np.random.rand(10000, 1000)
b = np.arange(10000)

c = a + b

timeit -n 10 c = a + np.tile(b.reshape((-1, 1)), (1, 1000))

timeit -n 10 c = a + b.reshape((-1, 1))

array_equal(a + np.tile(b.reshape((-1, 1)), (1, 1000)), a + b.reshape((-1, 1)))

Fancy indexing with np.take

b = np.take(a, ind, axis=0)

Compress for boolean indexing

b = np.compress(ind, a, axis=0)

Formatting print strings and passing variables

print("Spaces %d. tabs %d. lines %d" % (spaces, tabs, lines))  
  
msg = "{0} loves {1}.".format(name, language)  
  
d = datetime.date(2004, 9, 8)  
f"{d} was a {d:%A}, we started the mailing list back then."  
'2004-09-08 was a Wednesday, we started the mailing list back then.'  

answer = 42  
print(f"The answer is {answer}")

Using finally for exception cleanup

try:  
    fobj = open("hello.txt", "w")  
    res = 12 / 0  
except ZeroDivisionError:  
    print("We have an error in division")  
finally:  
    fobj.close()  
    print("Closing the file object.")  

We have an error in division
Closing the file object.

While Loops

Use while loops when you don't know how many times you will iterate.

def countdown(n):
  """Print a countdown from n to 0."""   
  i = n  
  while i >= 0:  
    print i 
    i -= 1

countdown(5)

Searching for filenames with glob

csv_list = glob.glob('/some/path/**/*.csv', recursive=True)

Reading multiple files in one line

Using Pandas

import glob
import pandas as pd
df = pd.concat([pd.read_csv(f, encoding='latin1') for f in glob.glob('data*.csv'), ignore_index=True])

Using dask

import dask.dataframe as dd
df = dd.read_csv('*.csv')

Using os

import os
files = os.listdir()
# ['data_blah.csv', 'more_data.csv', 'yet_another.csv', 'lots_of_data.csv']

Testing

Improve Your Python: Understanding Unit Testing By Jeff Knupp.
pytest Python testing tool that helps you write better programs.

Python and Twitter

Tweepy

Sphinx for Python (& other code) documentation

Sphinx Home
First Steps with Sphinx Tutorial Includes installation info.
Math Support for Sphinx
Selecting an HTML Theme A theme is a collection of HTML templates, stylesheet(s) and other static files. See this page for built-in theme examples also.
Jinja/Sphinx Templating Sphinx uses the Jinja templating engine for its HTML templates. Jinja is a text-based engine, and inspired by Django templates.
Sampledoc Tutorial For getting started with Sphinx.
Pygments Python syntax highlighter.
Generate PDFs with LaTeXBuilder This builder produces a bunch of LaTeX files in the output directory.
reStructuredText Docs For the reStructuredText plaintext markup syntax and parser system.
Documentation using Sphinx and ReadTheDocs.org
Sphinx for Python Documentation - a detailed tutorial

Python on Windows

Reading and Writing Files

Note to Self: Saving Images with PIL (Python Imaging Library)

scipy.misc.toimage followed by PIL Image.save() is nearly 2x faster than skimage.io.imsave or scipy.misc.imsave. Also, skimage.io.imsave and scipy.misc.imsave do not easily pass parameters to underlying image save methods, including pil plugin. (Added: 3/16/2016)

Saving PIL image to TIFF format:

Following TIFF compression options throw error "encoder error -2 when writing image file": jpeg, tif_jpeg, tiff_ccitt, tiff_raw_16, group3, group4, tiff_adobe_deflate, tiff_thunderscan, tiff_deflate, tiff_sgilog, tiff_sgilog24
Only TIFF compression options that work for input 150 KB image: Default/None (14,792 KB), raw (14,792 KB), tiff_lzw (184 KB, no detail info), packbits (477 KB, no detail info)