Python_Third_Party - RicoJia/notes GitHub Wiki

========================================================================

========================================================================

1. pudb for multiple processes
   • esc esc up to close window
   • set break point another file
   • c-p: preferences
   • to quit:
2. last command in command window: ctrl-n, p

========================================================================

========================================================================

2. c-contiguous array: array's rows are contiguous. f-contiguous (fortran) is columns are contiguous.

3. stride: number of bytes per dimension in 3D array, (bytes_per_page, bytes_per_row, bytes_per_item).

4. vectorized functions is to apply the function on each element of the array.

5. numpy axes: by default, 1d is a column vector

   

   • note: rgb like [X,Y,3] cannot do frame multiplication directly, because you can only multiply [3,] array with it.

   

6. performance issues:

   • np.clip() is slow because it returns a copy of an array. Alternative: np.core.umath.clip faster
   • np.zeros(shape=a.shape, dtype = np.int32) is 100 times faster than a=np.zeros_like(a).astype(np.uint32)

7. indexing

   • np.any(axis): returns True any element along the axis has element true.

8. np_random

   1. random.randint(low, high=None, size=None, dtype=int)¶
      • works with negative values as well

7. np.unravel_index(indices, shape): convert index in a linear array, into that in an n-d array, specified in (shape)

   np.unravel_index(6, (3, 4))
   (1, 2)  #(1,2) is the index for 6 in a (3,4) array
   
   np.unravel_index([2,3], (3,4))
   (array([0, 0]), array([2, 3]))    #actually it's (0,2), (0,3)

8. np.where

   1. np allocates memory like C, so it's contiguous.
   2. Math operations can be applied to all elements, which is a lot faster than for loop and use math module:
      • +,-, * /
      • np.sqrt(), np, cos
      • select sub-region and change it
      • broadcast is to apply a row/cln to the array. You can even use a LIST
      • Conditional Assignments using np.where
        • np.where(cond, val1, val2). if arr1 meets cond, output arr1, else output arr2
        • np.where(cond), return x, y idex of elements that meet certain criteria:
   3. np matrix - matrix stuff
      1. solve linalg
      2. eigen values, determinant

9. meshgrid:

   np.mgrid[0:5, 0:1]
   array([[[0],
           [1],
           [2],
           [3],
           [4]],
   
          [[0],
           [0],
           [0],
           [0],
           [0]]])
   
   np.mgrid[0:5, 0:2]
   array([[[0, 0],
           [1, 1],
           [2, 2],
           [3, 3],
           [4, 4]],
   
          [[0, 1],
           [0, 1],
           [0, 1],
           [0, 1],
           [0, 1]]])

   • This works on 1d array np.array.reshape (new_row, new_cln)

10. np.array.squeeze(): remove axis with length 1

 arr = np.array([[1,2,3]])
 arr.squeeze()
 # np.array([1,2,3])
 # will not be in effect if no axis is of length 1.

1. np.argsort sorts along an axis (ascending order), and return the indices

   arr = np.array([4,2,3,1])
   np.argsort(arr) # see array([3, 1, 2, 0])

   • if you want descending order, use np.flip()
   • np.flipud():axis 0; np.fliplr() axis 1

2. np.argmin(arr): returns the index of the smallest element

3. Round numbers:

   • np.round(np_array, decimal_places)
   • round(num, digits_after_decimal_point)

     round(4.567, 2) #4.57

========================================================================

========================================================================

1. AttributeError: module 'zipfile' has no attribute 'open': type in module name: zip_file instead of zipfile

========================================================================

========================================================================

1. sym.collect(J_dot, T), collect all terms together

========================================================================

========================================================================

1. reformatting using: autopep8

   pip3 install --upgrade autopep8
   autopep8 --in-place --aggressive --aggressive

========================================================================

========================================================================

1. from RGB values to JPEG

   pil_img = Image.fromarray(np_image, 'RGB')
   pil_img.thumbnail((800, 800))
   pil_img.save(buf, format="JPEG", quality=35)

========================================================================

========================================================================

1. plt - show in real time

   import matplotlib.pyplot as plt
   plt.clf()
   plt.title("Reid Descriptor")
   plt.plot(flat_descriptor)
   # plt.plot(np.arange(1), descriptor[:, :, 1])
   # plt.plot(np.arange(2), descriptor[:, :, 2])
   plt.pause(0.1)

========================================================================

========================================================================

1. boto3 is for uploading to AWS

========================================================================

========================================================================

1. distance_cost_matrix = pairwise_distances(track_pos, det_pos)

   array([[1., 1.],
      [2., 2.],
      [2., 2.]])
   
   array2([[1., 1.],
          [1., 1.],
          [1., 1.],
          [1., 1.],
          [1., 1.]])
   # number of features should be the same
   # returns the distance b/w each sample in 1 and 2
   
   pairwise_distances(arr1,arr2)
   array([[0.        , 0.        , 0.        , 0.        , 0.        ],
          [1.41421356, 1.41421356, 1.41421356, 1.41421356, 1.41421356],
          [1.41421356, 1.41421356, 1.41421356, 1.41421356, 1.41421356]])

1. dataframe

• track_eval_df.to_csv(OUTPUT_LOCATION, index=False)

========================================================================

========================================================================

1. Distance

   math.hypot(p2[0]-p1[0],p2[1]-p1[1], ....)    #hypotenuse
   np.linalg.norm(p2-p1)

2. random number

   s = np.array([0,0])
   e = np.array([10,10])
   np.random.uniform(s,e)  #float
   np.random.uniform(s,e).astype(int) #int
   
   # multiple times: 
   #Return random floats in the half-open interval [0.0, 1.0).
   5 * np.random.random_sample((3, 2)) - 5
   # array([[-3.99149989, -0.52338984], # random
          [-2.99091858, -0.79479508],
          [-1.23204345, -1.75224494]])

3. np array. see code

   • invert a np array
   • Other examples

     # access 
     arr = np.zeros((2,3,3))
     arr[1,2,3] =1
     # see a whole dimension 
     arr[1,2]    # see array([0., 0., 1.])
     # a tuple can access np array
     ls = [1,2]
     arr[tuple(ls)]  #see array([0., 0., 1.])
     
     #shape 
     print(arr.shape)    #see (4,)
     # ndim a smart way, because arr doesn't necessarily have a row number
     print(arr.ndim)     #see 1
     
     # stacking
     arr1 = np.array([1,2,3])
     # or creating an empty array: 
     arr1 = np.array([]).reshape(0,3)
     np.stack((arr1, arr2), axis=-1) #[1, 1], [2,2], [3,3]
     
     # has two random values
     np.empty([2]) 
     # same shape as arr
     np.zeros_like(arr)
     # copy an array 
     arr2 = np.copy(arr)
     
     # default is float, which DISTORTED COLORS.
     img = np.zeros((row_num, cln_num))
     img = np.zeros((row_num, cln_num), dtype=int)
     # or 
     img = img.astype(np.int)
     
     # Count non zeros: 
     num = np.count_nonzero(arr)
     
     # array indexing using another array
     a = np.array([1,1])
     arr = np.eye(3)
     arr[tuple(a.astype(int))]       #see 1
     
     # masking, np masking (Boolean indexing: shows only indices with True. Not applicable to list)
     arr = np.arange(5) #[0, ... 4]
     arr > 3 # see [F, F, T, T, T]
     arr[arr>3]  # actually applying a mask: showing only True ones
     arr1 == arr2 # gives [True, False, ...]
     np.array_equal(arr1, arr2)  #this gives the right T/F
     
     # reverse array
     np.flip(mat, axis=0)
     
     # Flatten 
     array([[1., 2., 3.],
        [0., 1., 0.],
        [0., 0., 9.]])
     arr.flatten()   # array([1., 2., 3., 0., 1., 0., 0., 0., 9.])
     
     # inf
     foo *= np.NINF
     np.isinf(foo)
     
     # transpose 
     arr.transpose()
     
     # dot product of two vectors
     np.dot(arr1, arr2)
     # applies element-wise multiplication
     arr1*arr2 
     
     # Broadcasting: two different shaped arrays may be added together, if the last dimension matches
     [1,2,3] +
     [
      [0,1,2],
      [2,3,4]
     ]    # each row is added by [1,2,3]
     # or scalar
     [1,2,3] + 5
     
     # max of whole array
     max_dim = np.max([3, img.ndim])
     
     # array comparison
     if (np.all(np.array(p[0]) < np.array(frame.shape[0:2]))): 

4. KD tree

   from sklearn.neighbors import KDTree

5. Open3D

   # add color
   # need every point to have a color
   pcd.colors = o3d.utility.Vector3dVector(np_rgb) # need to be in [0, 1], float

6. click for creating cli tool