AI ML - CameronAuler/python-devops GitHub Wiki

Python is the most widely used language for Artificial Intelligence (AI) and Machine Learning (ML) due to its rich ecosystem of frameworks and libraries.

Table of Contents

Deep Learning Frameworks

Deep learning frameworks provide tools for training, optimizing, and deploying neural networks.

Popular Deep Learning Frameworks

Framework Description
TensorFlow Google's deep learning library with support for production deployment.
PyTorch Facebook's dynamic deep learning framework, widely used in research.
Keras High-level API for TensorFlow, easy to use for building neural networks.

Creating a Simple Neural Network with TensorFlow/Keras

Use cases: Image classification, speech recognition, NLP tasks.

import tensorflow as tf
from tensorflow import keras

# Define model
model = keras.Sequential([
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

# Compile model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train model
model.fit(train_data, train_labels, epochs=5)

Natural Language Processing (NLP)

NLP enables machines to understand, interpret, and generate human language.

Popular NLP Libraries

Library Description
spaCy Optimized for large-scale NLP tasks (named entity recognition, tokenization).
NLTK Classical NLP library for tokenization, stemming, and text processing.
Hugging Face Transformers Pretrained transformer models for text generation, classification, and summarization.

Using spaCy for Named Entity Recognition (NER)

Use cases: Text analysis, sentiment analysis, chatbots.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Elon Musk founded SpaceX in 2002.")

for ent in doc.ents:
    print(ent.text, ent.label_)

Computer Vision

Computer vision enables machines to interpret images and videos.

Popular Computer Vison Libraries

Library Description
OpenCV Image processing, object detection, face recognition.
TensorFlow/Keras Deep learning-based image classification.
YOLO (You Only Look Once) Real-time object detection.

Face Detection with OpenCV

Use cases: Facial recognition, object detection.

import cv2

face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")

img = cv2.imread("face.jpg")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5)
for (x, y, w, h) in faces:
    cv2.rectangle(img, (x, y), (x + w, y + h), (255, 0, 0), 2)

cv2.imshow("Face Detection", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

Reinforcement Learning

RL trains agents to make optimal decisions by interacting with an environment.

Popular Reinforcement Learning Libraries

Library Description
Stable-Baselines3 Prebuilt reinforcement learning algorithms.
Gym Environment for training RL models (e.g., Atari games).
Ray RLlib Scalable reinforcement learning framework.

Using Gym for Reinforcement Learning

Use cases: Robotics, gaming AI.

import gym

env = gym.make("CartPole-v1")
observation = env.reset()

for _ in range(100):
    env.render()
    action = env.action_space.sample()  # Take random action
    observation, reward, done, info = env.step(action)
    if done:
        break

env.close()

Model Deployment

Allows serving trained models for real-world applications.

Popular Model Deployment Tools

Tool Description
Flask Lightweight web framework for serving AI models.
FastAPI High-performance API framework for AI applications.
TensorFlow Serving Deploys TensorFlow models in production.

Deploying am AI Model with FastAPI

Use case: Deploying AI-powered web applications.

from fastapi import FastAPI
import joblib

app = FastAPI()
model = joblib.load("model.pkl")

@app.get("/predict/")
def predict(value: float):
    prediction = model.predict([value](/CameronAuler/python-devops/wiki/value))
    return {"prediction": prediction.tolist()}

Data Engineering

Data engineering prepares and processes large datasets for machine learning.

Popular Data Engineering Tools

Tool Description
Pandas Data manipulation and cleaning.
Dask Scalable data processing.
Apache Spark Distributed computing for large-scale datasets.

Processing Large Datasets with Dask

Use case: Handling big data for AI models.

import dask.dataframe as dd

df = dd.read_csv("large_dataset.csv")
print(df.describe().compute())  # Perform computations on large datasets

Explainability & Bias in AI

Explainability helps understand how AI models make decisions. Bias detection ensures fairness in AI predictions.

Tools for AI Explainability & Bias Detection

Tool Description
SHAP Explains model decisions using SHapley values.
LIME Local model interpretability for black-box models.
Fairlearn Detects and mitigates bias in machine learning models.

Using SHAP to Explain Model Predictions

Use cases: AI transparency in finance, healthcare, and hiring.

import shap

explainer = shap.Explainer(model)
shap_values = explainer(X_test)
shap.summary_plot(shap_values, X_test)

Optimization & Hyperparameter Tuning

Optimizing model parameters to improve performance.

Popular Hyperparameter Tuning Tools

Tool Description
Optuna Automated hyperparameter tuning.
GridSearchCV Scikit-learn's grid search method.
Ray Tune Distributed hyperparameter optimization.

Using Optuna for Tuning

Use case: Boosting AI model performance.

import optuna

def objective(trial):
    lr = trial.suggest_loguniform("lr", 1e-5, 1e-1)
    model = train_model(learning_rate=lr)
    return model.evaluate()

study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=10)