Python - ilya-khadykin/notes-outdated GitHub Wiki

TO DO:

Key features of the language

  • interpreted (speeds up development);
  • object-oriented (full support with operators overloading);
  • dynamic typing;
  • built-in high-level data structures (tuples, lists, dictionaries and strings);
  • extensive standard library and many free third party Python modules;
  • Python is extensible: you can link Python programs to libraries that may only be available in binary form or you can link the Python interpreter into an application written in C and use it as an extension or command language for that application;
  • available on all major platforms, installed by default (usually) on Unix systems.

PEPs: Python Enhancement Proposals

Index: https://www.python.org/dev/peps/

Useful links

History

Python was developed by Guido van Rossum in the late eighties and early nineties at the National Research Institute for Mathematics and Computer Science in the Netherlands.

Python is copyrighted. Like Perl, Python source code is now available under the GNU General Public License (GPL).

Python is now maintained by a core development team at the institute, although Guido van Rossum still holds a vital role in directing its progress.

Installation

Python Environment Variables

Here are important environment variables, which can be recognized by Python:

Variable Description
PYTHONPATH It has a role similar to PATH. This variable tells the Python interpreter where to locate the module files imported into a program. It should include the Python source library directory and the directories containing Python source code. PYTHONPATH is sometimes preset by the Python installer.
PYTHONSTARTUP It contains the path of an initialization file containing Python source code. It is executed every time you start the interpreter. It is named as .pythonrc.py in Unix and it contains commands that load utilities or modify PYTHONPATH.
PYTHONCASEOK It is used in Windows to instruct Python to find the first case-insensitive match in an import statement. Set this variable to any value to activate it.
PYTHONHOME It is an alternative module search path. It is usually embedded in the PYTHONSTARTUP or PYTHONPATH directories to make switching module libraries easy.

The Python Interpreter

Implementations

When people speak of Python they often mean not just the language but also the CPython implementation. Python is actually a specification for a language that can be implemented in many different ways.

The different implementations may be for compatibility with other libraries, or maybe for a little speed. Here’s a quick rundown on the most popular implementations:

CPython

CPython is the reference implementation of Python, written in C. It compiles Python code to intermediate bytecode which is then interpreted by a virtual machine. CPython provides the highest level of compatibility with Python packages and C extension modules.

If you are writing open-source Python code and want to reach the widest possible audience, use CPython. To use packages which rely on C extensions to function, CPython is your only implementation option.

All versions of the Python language are implemented in C because CPython is the reference implementation.

PyPy

PyPy is a Python interpreter implemented in a restricted statically-typed subset of the Python language called RPython. The interpreter features a just-in-time compiler and supports multiple back-ends, such as C, CIL (Common Intermediate Language), and JVM (Java Virtual Machine) bytecode.

PyPy aims for maximum compatibility with the reference CPython implementation while improving performance.

If you are looking to increase performance of your Python code, it’s worth giving PyPy a try. On a suite of benchmarks, it’s currently over 5 times faster than CPython

PyPy supports Python 2.7. PyPy3, released in beta, targets Python 3.

Jython

Jython is a Python interpreter implementation that compiles Python code to Java bytecode which is then executed by the JVM. Additionally, it is able to import and use any Java class like a Python module.

If you need to interface with an existing Java codebase or have other reasons to need to write Python code for the JVM, Jython is the best choice.

Jython currently supports up to Python 2.7.

IronPython

IronPython is an implementation of Python for the .NET framework. It can use both Python and .NET framework libraries, and can also expose Python code to other languages in the .NET framework.

Python Tools for Visual Studio integrates IronPython directly into the Visual Studio development environment, making it an ideal choice for Windows developers.

IronPython supports Python2.7.

PythonNet

Python for .NET is a package which provides near seamless integration of a natively installed Python installation with the .NET CLR (Common Language Runtime). This is the inverse approach to that taken by IronPython, meaning PythonNet and IronPython complement rather than compete with each other.

In conjunction with Mono, PythonNet enables native Python installations on non-Windows operating systems, such as OS X and Linux, to operate within the .NET framework. It can be run in addition to IronPython without conflict.

PythonNet supports from Python 2.3 up to Python 2.7.

Skulpt

Skulpt is a javascript implementation of Python. It has not ported all of the CPython standard library; they have math, random, turtle, image, unittest, and parts of time, urllib, DOM, and re. It is intended for use in teaching. There is also a way to add your own modules.

Notable examples of its use are Interactive Python and CodeSkulptor.

Skulpt supports most of Python 2.7 and Python 3.3.

Unix

The Python interpreter is usually installed as /usr/local/bin/python3.5, but there are other possible places (depends on systems configuration), /usr/local/pythonfor example. Use/usr/local/bin` in your Unix shell scripts to run Python interpreter.

Windows

Usually installed into C:\Python35. To add this directory to your system path use the following command set path=%path%;C:\python35

Use Control-D or type quit() to close the interpreter.

The interpreter operates somewhat like the Unix shell: when called with standard input connected to a tty device, it reads and executes commands interactively; when called with a file name argument or with a file as standard input, it reads and executes a script from that file.

Examples:

python -c command [arg] ...
python -m module [arg] ...

Argument Passing

When known to the interpreter, the script name and additional arguments thereafter are turned into a list of strings and assigned to the argv variable in the sys module. You can access this list by executing import sys. The length of the list is at least one; when no script and no arguments are given, sys.argv[0] is an empty string. When the script name is given as '-' (meaning standard input), sys.argv[0] is set to '-'. When -c command is used, sys.argv[0] is set to '-c'. When -m module is used, sys.argv[0] is set to the full name of the located module. Options found after -c command or -m module are not consumed by the Python interpreter’s option processing but left in sys.argv for the command or module to handle.

Source Code Encoding

By default, Python source files are treated as encoded in UTF-8. In that encoding, characters of most languages in the world can be used simultaneously in string literals, identifiers and comments — although the standard library only uses ASCII characters for identifiers, a convention that any portable code should follow. To display all these characters properly, your editor must recognize that the file is UTF-8, and it must use a font that supports all the characters in the file.

It is also possible to specify a different encoding for source files. In order to do this, put one more special comment line right after the #! line to define the source file encoding:

# -*- coding: encoding -*-
# -*- coding: utf-8 -*-

Syntax Overview

  • indention instead of parenthesis: Python uses spaces to distinguish code blocks vs () or {} in other languages
  • # - single line comment

Expressions

Arithmetic Operators

Expression Description Example
+ - * equivalent to mathematical operations (50 - 5*6) / 4
/ division always returns a floating point number 8 / 5
// floor division returns an integer result (discarding any fractional part) 17 // 3 # == 5
% calculates a remainder of division 17 % 3 # == 2
** calculates power 5 ** 2 # 5 squared
= used to assign a value to a variable height = 5 * 9
_ available only in interactive mode of the interpreter and contains result of the last evaluated expression _ + 5

Strings

Expression Description Example
'' or "" String declaration word = 'Python'
""" """ multipleline String declaration print(""" your multipleline string """)
"""\ """ new line characters are not included print("""\ your multipleline string without new line characters"""
+ concatination of two Strings 'un' + 'ium' # == 'unium'
* repeats String multiple times 3 * 'un' # == 'ununun'
[] String indexing word[0]; word[-6]; word[4:]; word[-2:];
\ escape character 'doesn\'t' # use \' to escape the single quote...
\n \r \t etc special characters print('C:\some\name') # \n - new line
r'' raw strings (special characters are ignored) print(r'C:\some\name')
len() the built-in function len() returns the length of a string len(str)

Comparison Operators

These operators compare the values on either sides of them and decide the relation among them. They are also called Relational operators.

Operator Description Example
== If the values of two operands are equal, then the condition becomes true. (a == b) is not true.
!= If values of two operands are not equal, then condition becomes true. (a!= b) is true.
> If the value of left operand is greater than the value of right operand, then condition becomes true. (a > b) is not true.
< If the value of left operand is less than the value of right operand, then condition becomes true. (a < b) is true.
>= If the value of left operand is greater than or equal to the value of right operand, then condition becomes true. (a >= b) is not true.
<= If the value of left operand is less than or equal to the value of right operand, then condition becomes true. (a <= b) is true.

is vs ==

is tests for identity, compares two objects by reference meaning that it check if two object are the same. == tests for equality, compares two objects by value.

>>> a = [1, 2, 3]
>>> b = a
>>> b is a 
True
>>> b == a
True
>>> b = a[:]
>>> b is a
False
>>> b == a
True

>>> 1000 is 10**3
False
>>> 1000 == 10**3
True

>>> "a" is "a"
True
>>> "aa" is "a" * 2
True
>>> x = "a"
>>> "aa" is x * 2
False
>>> "aa" is intern(x*2)
True

Control Flow Statements

if statement

x = int(input("Please enter an integer: "))
if x < 0:
    x = 0
    print('Negative changed to zero')
elseif x == 0:
    print('Zero')
elseif x == 1:
    print('Single')
else:
    print('More')

An if ... elif ... elif ... sequence is a substitute for the switch or case statements found in other languages.

conditional statement

conditional_statement = "true" if (5 > 4) else "false"

while loop

a, b = 0, 1
while b < 10:
    print(b)
    a, b = b, a+b

for loop

# Measure some strings:
words = ['cat', 'window', 'defenestrate']
for w in words:
    print(w, len(w))

If you need to modify the sequence you are iterating over while inside the loop (for example to duplicate selected items), it is recommended that you first make a copy. Iterating over a sequence does not implicitly make a copy. The slice notation makes this especially convenient:

for w in words[:]:  # Loop over a slice copy of the entire list.
    if len(w) > 6:
        words.insert(0, w)

enumerate() function

    days = ["Mon", "Tue", "Wen", "Thu", "Fri", "Sat", "Sun"]
    for i, d in enumerate(days):
        print(i, d)

The range() Function

range() generates arithmetic progressions

for i in range(5):
    print(i)
"""
range(5, 10)
   5 through 9

range(0, 10, 3)
   0, 3, 6, 9

range(-10, -100, -30)
  -10, -40, -70

print(range(10))
  range(0, 10)

list(range(5))
  [0, 1, 2, 3, 4]
"""
a = ['Mary', 'had', 'a', 'little', 'lamb']
for i in range(len(a)):
    print(i, a[i])

break, continue Statements and else Clauses on Loops

  • continue statement continues with the next iteration of the loop
  • break stops the execution of a loop
  • loop’s else clause runs when no break occurs
for n in range(2, 10):
    for x in range(2, n):
        if n % x == 0:
            print(n, 'equals', x, '*', n//x)
            break
    else:
        # loop fell through without finding a factor
        print(n, 'is a prime number')

pass Statements

The pass statement does nothing. It can be used when a statement is required syntactically but the program requires no action

while True:
    pass  # Busy-wait for keyboard interrupt (Ctrl+C)

class MyEmptyClass:
    pass

def initlog(*args):
    pass   # Remember to implement this!

Variables and Types

The data stored in memory can be of many types. Python has five standard data types:

  • Numbers
  • String
  • List
  • Tuple
  • Dictionary

You can also delete the reference to an object or variable using del:

del var1[,var2[,var3[....,varN]]]]

Numbers

Python supports two types of numbers - integers and floating point numbers. (It also supports complex numbers as well as complex numbers. The integer numbers (e.g. 2, 4, 20) have type int, the ones with a fractional part (e.g. 5.0, 1.6) have type float.

In addition to int and float, Python supports other types of numbers, such as Decimal and Fraction. Python also has built-in support for complex numbers, and uses the j or J suffix to indicate the imaginary part (e.g. 3+5j).

Here are some examples of numbers:

int float complex
10 0.0 3.14j
100 15.20 45.j
-786 -21.9 9.322e-36j
080 32.3+e18 .876j
-0490 -90. -.6545+0J
-0x260 -32.54e100 3e+26J
0x69 70.2-E12 4.53e-7j

A complex number consists of an ordered pair of real floating-point numbers denoted by x + yj, where x and y are the real numbers and j is the imaginary unit.

myint = 7

myfloat = 7.0
myfloat = float(7)

Strings

Strings are defined either with a single quote or a double quotes.

Python strings cannot be changed — they are immutable. Therefore, assigning to an indexed position in the string results in an error.

mystring = 'hello'
mystring = "hello"

mystring = "Don't worry about apostrophes"

str = 'Hello World!'
print (str)          # Prints complete string
print (str[0])       # Prints first character of the string
print (str[2:5])     # Prints characters starting from 3rd to 5th
print (str[2:])      # Prints string starting from 3rd character
print (str * 2)      # Prints string two times
print (str + "TEST") # Prints concatenated string

Data Structures

Lists

Lists are a mutable type.

Lists can be written as a list of comma-separated values (items) between square brackets. Lists might contain items of different types, but usually the items all have the same type.

Like strings (and all other built-in sequence type), lists can be indexed and sliced.

squares = [1, 4, 9, 16, 25]

squares[0]  # indexing returns the item
squares[-1]
squares[-3:]  # slicing returns a new list [9, 16, 25]

list = [ 'abcd', 786 , 2.23, 'john', 70.2 ]
tinylist = [123, 'john']
print (list)          # Prints complete list
print (list[0])       # Prints first element of the list
print (list[1:3])     # Prints elements starting from 2nd till 3rd 
print (list[2:])      # Prints elements starting from 3rd element
print (tinylist * 2)  # Prints list two times
print (list + tinylist) # Prints concatenated lists

All slice operations return a new list containing the requested elements. This means that the following slice returns a new (shallow) copy of the list:

squares[:] # [1, 4, 9, 16, 25]

Other operations:

squares + [36, 49, 64, 81, 100] # == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

cubes = [1, 8]
cubes.append(216)  # add the cube of 6
cubes.append(7 ** 3)  # and the cube of 7

# clear the list by replacing all the elements with an empty list
letters[:] = []

letters = ['a', 'b', 'c', 'd']
len(letters) # == 4

Tuples

Tuples are immutable

A tuple is another sequence data type that is similar to the list. The main differences between lists and tuples are: Lists are enclosed in brackets ( [ ] ) and their elements and size can be changed, while tuples are enclosed in parentheses ( ( ) ) and cannot be updated.

Tuples can be thought of as read-only lists.

#!/usr/bin/python3

tuple = ( 'abcd', 786 , 2.23, 'john', 70.2  )
tinytuple = (123, 'john')

print (tuple)           # Prints complete tuple
print (tuple[0])        # Prints first element of the tuple
print (tuple[1:3])      # Prints elements starting from 2nd till 3rd 
print (tuple[2:])       # Prints elements starting from 3rd element
print (tinytuple * 2)   # Prints tuple two times
print (tuple + tinytuple) # Prints concatenated tuple

Dictionary

Python's dictionaries are kind of hash table type. They work like associative arrays or hashes found in Perl and consist of key-value pairs. A dictionary key can be almost any Python type, but are usually numbers or strings. Values, on the other hand, can be any arbitrary Python object.

Dictionaries are enclosed by curly braces ({ }) and values can be assigned and accessed using square braces ([]).

#!/usr/bin/python3

dict = {}
dict['one'] = "This is one"
dict[2]     = "This is two"

tinydict = {'name': 'john','code':6734, 'dept': 'sales'}


print (dict['one'])       # Prints value for 'one' key
print (dict[2])           # Prints value for 2 key
print (tinydict)          # Prints complete dictionary
print (tinydict.keys())   # Prints all the keys
print (tinydict.values()) # Prints all the values

Dictionaries have no concept of order among elements. It is incorrect to say that the elements are "out of order"; they are simply unordered.

Functions

def test_func(*args):
    i = 0
    for arg in args:
        print("Argument #" + str(i) + " = " + str(arg))
        i += 1

Classes

class TestClass():
    def method1(self):
        print("method1 in TestClass")

    def method2(self):
        print("method2 in TestClass")


class AnotherClass(TestClass):
    def method1(self):  # Overrides method from TestClass
        print("method1 in AnotherClass")

    def method2(self):  # Overrides method from TestClass
        print("method2 in AnotherClass")

Data Type Conversion

Sometimes, you may need to perform conversions between the built-in types. To convert between types, you simply use the type name as a function.

There are several built-in functions to perform conversion from one data type to another. These functions return a new object representing the converted value.

Function Description
int(x [,base]) Converts x to an integer. base specifies the base if x is a string.
float(x) Converts x to a floating-point number.
complex(real [,imag]) Creates a complex number.
str(x) Converts object x to a string representation.
repr(x) Converts object x to an expression string.
eval(str) Evaluates a string and returns an object.
tuple(s) Converts s to a tuple.
list(s) Converts s to a list.
set(s) Converts s to a set.
dict(d) Creates a dictionary. d must be a sequence of (key,value) tuples.
frozenset(s) Converts s to a frozen set.
chr(x) Converts an integer to a character.
unichr(x) Converts an integer to a Unicode character.
ord(x) Converts a single character to its integer value.
hex(x) Converts an integer to a hexadecimal string.
oct(x) Converts an integer to an octal string.

if __name__ == "__main__": (Credits)

When the Python interpreter reads a source file, it executes all of the code found in it. Before executing the code, it will define a few special variables. For example, if the python interpreter is running that module (the source file) as the main program, it sets the special __name__ variable to have a value "__main__". If this file is being imported from another module, __name__ will be set to the module's name.

In the case of your script, let's assume that it's executing as the main function, e.g. you said something like

python threading_example.py

on the command line. After setting up the special variables, it will execute the import statement and load those modules. It will then evaluate the def block, creating a function object and creating a variable called myfunction that points to the function object. It will then read the if statement and see that __name__ does equal "__main__", so it will execute the block shown there.

One of the reasons for doing this is that sometimes you write a module (a .py file) where it can be executed directly. Alternatively, it can also be imported and used in another module. By doing the main check, you can have that code only execute when you want to run the module as a program and not have it execute when someone just wants to import your module and call your functions themselves

More details: http://ibiblio.org/g2swap/byteofpython/read/module-name.html

Installing pip for different versions of Python

  1. Go to https://pip.pypa.io/en/latest/installing/ and download get-pip.py
  2. Execute get-pip.py using target interpreter
  3. Install required packages using the following command python3.5 -m pip install pytz

Modules

module comment
jira Python module for interacting with JIRA; docs - https://pythonhosted.org/jira/

jira

from jira import JIRA

def create_issue_in_jira(
        jira_base_url,
        username, password,
        project='SNEIMTSANDBOX',
        summary='',
        issue_type='Task',
        assignee=None):
    jira = JIRA(
        options={
            'server': jira_base_url
        },
        basic_auth=(
            username,
            password
        )
    )

    new_issue = jira.create_issue(
        project=project,
        summary='Test: JIRA issue created using API',
        description='Look into this one',
        issuetype={'name': issue_type},
        assignee={'name': assignee}
    )

create_issue_in_jira('https://jira.example.com', 'ilya.khadykin', 'password')

Examples

How to check if a variable is a function:

import types

isinstance(f, (types.FunctionType, types.BuiltinFunctionType, functools.partial))

In [1]: import types
In [2]: types.FunctionType
Out[2]: <type 'function'>
In [3]: def f(): pass
   ...:
In [4]: isinstance(f, types.FunctionType)
Out[4]: True
In [5]: isinstance(lambda x : None, types.FunctionType)
Out[5]: True

Simple Thread Pool Example

from time import sleep
import types
import threading
import requests  # Not an actual dependency, just used to demonstrate class usage.


class ParallelExecutioner:
    """
    Class is intended for executing code on each element of a list in separate Thread
    which it creates depending on parameters used on object creation.
    """
    def __init__(self, number_of_threads, data_list, payload_function, use_timeout=False, timeout_in_seconds=10):
        """
        Constructor uses the following parameters:
        :param number_of_threads: # Number of parallel threads to be created.
        :param data_list: # List of data on which payload function will be called in separate Thread
        :param payload_function: # Function to execute as a payload
        :param use_timeout: # Flag is used to halt execution for specified time after executing payload function
        :param timeout_in_seconds: # How many seconds to wait after executing payload function
        """
        self._lock = threading.Semaphore(number_of_threads)
        self._data_list = data_list
        self._payload_function = payload_function
        self._use_timeout = use_timeout
        self._timeout_in_seconds = timeout_in_seconds
        # creating a Thread Pool
        self.create_thread_pool()

    def run_payload(self, item):
        """
        Executes payload function passed as a parameter on object creation.
        :param item:
        :return: None
        """
        print("Starting payload...")
        # Executing payload function
        if isinstance(self._payload_function, types.FunctionType):
            self._payload_function(item)
        else:
            # Raise an exception if not a function was passed as payload.
            raise TypeError("Expecting a function but received a " + str(type(self._payload_function)))
        if self._use_timeout:
            print("Waiting: " + str(self._timeout_in_seconds) + " seconds (timeout)")
            sleep(self._timeout_in_seconds)
        self._lock.release()

    def create_thread_pool(self):
        """
        Creates a Thread Poll and uses threading.Semaphore object to control number of concurrent Threads.
        :return: None
        """
        print("Starting creating Thread Pool")
        # check if passed data is actually a list
        if not isinstance(self._data_list, list):
            raise TypeError("Expecting a list of data but received a " + str(type(self._data_list)))
        # List of threads objects
        thread_pool = []
        # passing each item of data list to the payload function and running it in separate Thread
        for item in self._data_list:
            # creating new thread that calls payload function
            thread = threading.Thread(target=self.run_payload, args=(item,))
            thread_pool.append(thread)
            thread.start()
            # adding thread to our lock, so we could wait if needed and not spawn all Threads at the same time.
            self._lock.acquire()
        for thread in thread_pool:
            # force waiting until the thread terminates
            thread.join()
        print("Thread Pool was successfully created")


def test_payload(item):
    """Just an example of payload function."""
    print("Sending HTTP Request to " + str(item))
    r = requests.get(item)
    print("Response status code: " + str(r.status_code) + " for " + str(item))


def main():
    """Simple example of how to use the class."""
    lst = ["https://google.com", "http://mail.ru", "http://ya.ru/", "https://vk.com/", "https://mail.google.com/"]
    pe = ParallelExecutioner(10, lst, test_payload)


if __name__ == "__main__":
    main()

Sending HTTP request and working with JSON response

import json
from urllib.request import urlopen
url = "https://gdata.youtube.com/feeds/api/standardfeeds/top_rated?alt=json"
response = urlopen(url)
contents = response.read()
text = contents.decode('utf8')
data = json.loads(text)
for video in data['feed']['entry'][0:6]:
    print(video['title']['$t'])

or:

import requests
url = "https://gdata.youtube.com/feeds/api/standardfeeds/top_rated?alt=json"
response = requests.get(url)
data = response.json()
for video in data['feed']['entry'][0:6]:
    print(video['title']['$t'])

Get formatted Date and Time

import time
time.strftime("%Y-%m-%dT%H:%M%S")
'2016-08-29T13:4145'

Format strings

The following directives can be embedded in the format string used in time.strftime():

Directive Meaning
%a Weekday name.
%A Full weekday name.
%b Abbreviated month name.
%B Full month name.
%c Appropriate date and time representation.
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%I Hour (12-hour clock) as a decimal number [01,12].
%j Day of the year as a decimal number [001,366].
%m Month as a decimal number [01,12].
%M Minute as a decimal number [00,59].
%p Equivalent of either AM or PM.
%S Second as a decimal number [00,61].
%U Week number of the year (Sunday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Sunday are considered to be in week 0.
%w Weekday as a decimal number [0(Sunday),6].
%W Week number of the year (Monday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Monday are considered to be in week 0.
%x Appropriate date representation.
%X Apropriate time representation.
%y Year without century as a decimal number [00,99].
%Y Year with century as a decimal number.
%Z Time zone name (no characters if no time zone exists).
%% A literal ‘%’ character.

References

  1. The Python Tutorial
  2. The Python Standard Library
  3. The Python Language Reference
  4. tutorialspoint.com/python3
  5. The Hitchhiker's Guide to Python