download_utils.py - cmikke97/Automatic-Malware-Signature-Generation GitHub Wiki
-
import os- provides a portable way of using operating system dependent functionality - os documentation -
import threading- constructs higher-level threading interfaces on top of the lower level _thread module - threading documentation -
from pathlib import Path- provide path-handling operations which donβt actually access a filesystem - pathlib documentation
-
import boto3- used to create, configure, and manage AWS services (s3 included) - boto3 documentation -
from botocore import UNSIGNED- constant to use to connect to s3 bucket anonymously (botocore is the core of boto 3) - botocore documentation -
from botocore.client import Config- advanced configuration for Botocore clients (botocore is the core of boto 3) - botocore documentation -
from logzero import logger- robust and effective logging for Python - logzero documentation -
from tqdm import tqdm- instantly makes loops show a smart progress meter - tqdm documentation
ProgressPercentage (class) - Class used to display a bar indicating download progress.
-
__init__(self, pbar)(member function) - Init progress bar.-
pbar(arg) - Already initialized tqdm progress bar
-
-
__call__(self, bytes_amount)(member function) - Update progress bar.-
bytes_amount(arg) - Amount of bytes received
-
BucketFileDownloader (class) - Class used to download bucket files from an s3 bucket.
-
__init__(self, destination_dir, bucket_name)(member function) - Init bucket file downloader.-
destination_dir(arg) - Path to the folder where to save the element to -
bucket_name(arg) - Name of the s3 bucket where to find the elements to download
-
-
__call__(self, object_name)(member function) - Download single object from s3 bucket.-
object_name(arg) - Name (relative path wrt the s3 bucket) of the object to download
-
check_files(destination_dir) (function) - Check if the dataset needed files are already present inside the specified directory.
-
destination_dir(arg) - Path to the destination folder where to search for the needed files