Modeling - liniribeiro/machine_learning GitHub Wiki

Model centric AI development x Data centric AI development

Make sure that you are feeding your model a quality data Its not just collect data, but use tools to improve the data in the best efficient way.

Key challenges

AI systems = Code(algorithm/model) + Data

There are many project that the model does well and we must improve the data to have an accurate prediction

Model development is an iterative process

Screenshot 2024-09-20 at 11 43 44

Is important to make good choices while interacting in the loop of development, in how modify the data, model or the hyperparameters.

When building a model there are 3 key milestones that most projects should try to accomplish:

  1. Doing well on training set (usually measured by average training error)
  2. Doing well on dev/tests sets
  3. Doing well on business metrics/projects goals

Just doing well on the test sets isn't enough for many applications

Establish baseline of performance

Ways to establish baseline:

  • Human Level Performance (HLP)
  • Literature search - See what others Systems reports are to have a baseline
  • Quick and dirty implementation - to give you a sense of whats possible
  • Performance of older system

Baselines helps to indicate what might be possible.

Getting started on modeling

  • Literature search to see what's possible (courses, blogs, open-source projects)
  • Find open -source implementation if available
  • A reasonable algorithm with good data will often outperform a great algorithm with no so good data.

Sanity check for code and algorithm

  • Try to overfit a small training dataset before training on a large one

Error analysis and performance auditing

Examine and tag examples -> propose tags Visual inspection example:

  • specific class label (scratch, dent, etc.)
  • Image properties (blury, dark, background)
  • other meta-data: phone model, factory

Product recommendations example:

  • user demographics
  • product features/category

Useful metrics for tags:

  • What fraction of errors has that tag?
  • Of all the data with that tag, what fraction is misclassified?
  • how fraction of all data has that tag?
  • Hoe much room for improvement is there on data with that tag?

Segment the data into different categories and use questions like the above to try to decide what prioritize to work on.

Prioritizing what to work on

Decide on most important categories to work on based on:

  • how much room for improvements there is
  • how frequently that category appears
  • how easy is to improve accuracy in that category
  • how important is to improve in that category

Adding/improving data for specific categories

For categories that you want to prioritize:

  • Collect more data
  • Use data augmentation to get more data
  • Improve label accuracy/data quality

Skewed datasets (distorcidos)

Combining precision + recall is one metric used for many applications, calculating F1 to know what model has best performance.

F1 also can help you to identify were to work on.

Performance auditing

Performance auditing before push model to production can help to catch problems.

Auditing framework

Check for fairness, accuracy and other problems.

  1. Brainstorm the ways the system might go wrong
    • Performance on subset of data
    • how common are the errors
    • Performance on rare classes
  2. Establish metrics to assess performance against there issues on appropriate slices of data
  3. Get business/product owner buy-in.

ps: tensorflow model analysis