AI/ML And Data Analytics

Introduction

• Define artificial intelligence (AI) and its subset machine learning (ML).

Artificial Intelligence is a broad field focused on the development of intelligent computer systems capable of performing humanlike tasks.

Machine learning is a type of AI for training machines to perform complex tasks without explicit instructions. Machine learning training finds the patterns hidden in vast amounts of historical data to produce an ML model. This ML model can then be applied to new data to make predictions or decisions based on the patterns it's learned.

AI/ML on AWS

• Identify common business use cases for AI/ML.
• Identify the three tiers of AI/ML solutions offered by the AWS AI/ML stack.

ML models power the Amazon.com e-commerce recommendations engine. But, ML can solve for lots of other business use cases, such as the following:

• Predict trends, such as future stock prices.
• Make decisions, like routing callers to the right department.
• Detect anomalies, such as bank fraud.

The AWS AI/ML stack is composed of the following three tiers of solutions:

• AI services - pre-built models that are already trained to perform specific functions
• ML services - a more customized approach with Amazon SageMaker AI where you build, train, and deploy your own ML models with fully managed infrastructure
• ML frameworks and infrastructure - a completely custom approach to building models using purpose-built chips that integrate with popular ML frameworks

AWS AI/ML Solutions

• Describe the benefits and purpose of the services contained in each tier of the AWS AI/ML stack.

You can think of the tiers as a progression from pre-built, easily deployable managed solutions to highly customized solutions that require more skill to implement.

Tier 1: Pre-built AWS AI services

The AWS AI services tier is made up of pre-built models that are already trained to perform specific functions. These ready-to-use, managed services can help you quickly solve for a variety of business use cases. In the next section, you will learn about the following three groups of AWS AI services:

• Language services: AWS AI language services are great for when you need to interpret text or speech and transform it into something meaningful. Let's examine how these services solve for some common use cases.
  • Comprehend: uses natural language processing to extract key insights from documents. It develops these insights by recognizing key phrases, language, sentiment, and other common elements in documents { Content classification, customer sentiment analysis, and compliance monitoring}
  • Polly: converts text into lifelike speech. It supports multiple languages, different genders, and a variety of accents. {Virtual assistants, e-learning applications, and accessibility enhancements for visually impaired users}
  • Transcribe: converts speech into text. It supports multiple languages and offers features such as speaker identification, custom vocabulary, and real-time transcription. {Customer call transcription, automated subtitling, and metadata generation for media content.}
  • Translate: text translation service. This service is ideal for global communication because it supports real-time and batch text translation across multiple languages. { Document translation and multi-language application integrations }
• Computer vision and search services: These services are ideal for answering questions and gathering insights from various types of content sources such as documents, images, videos, and more. Let's look at some examples.
  • Kendra: uses natural language processing to search for answers within large amounts of enterprise content. Because it understands the context of a query, it can return more precise and relevant answers than just a list of documents with matching keywords. { Intelligent search, chatbots, and application search integration }
  • ekognition: video analysis service. It can identify objects, people, text, scenes, and activities within images and videos stored in Amazon Simple Storage Service (Amazon S3). { Content moderation, identity verification, media analysis, and home automation experiences }
  • Textract: detects and extracts typed and handwritten text found in documents, forms, and even tables within documents. { Financial, healthcare, and government form text extraction for quick processing }
• Conversational AI and personalization services: users can interact with your apps through text and voice conversations. You can also present your customers with product recommendations personalized just for them. Let's explore these services.
  • Lex: add voice and text conversational interfaces to your applications. This service uses both natural language understanding (NLU) and automatic speech recognition (ASR) to create lifelike conversations. { Virtual assistants, natural language search for FAQs, and automated application bots }
  • Personalize: you can use historical data to build intelligent applications with personalized recommendations for your customers. { Personalized streaming, product, and trending recommendations }

Tier 2: ML Services

The ML services tier provides a more customized approach for customers who want a bit more control over their ML solutions without having to manage infrastructure. SageMaker AI is a key offering in this tier.

With this fully managed service, you can build, train, and deploy your own ML models without worrying about infrastructure. The SageMaker AI integrated development environment (IDE) provides simplified access control and transparency over your ML projects. You can track model training experiments, visualize data, and debug and monitor your workflows all within one environment. SageMaker AI even offers access to hundreds of pre-trained models that you can deploy in a few quick steps.

Tier 3: ML frameworks and infrastructure

ome organizations have highly specialized needs that require complete control over the ML training process. They can use in-house expertise, ML frameworks, and AWS infrastructure to develop their own ML solutions.

Core components:

• ML frameworks: n ML framework is a software library or tool that provides experienced ML practitioners with pre-built, optimized components for building machine learning models. AWS supports ML frameworks like PyTorch, Apache M-X Net, and TensorFlow.
• AWS ML Infra: AWS ML infrastructure, such as ML-optimized EC2 instances, Amazon EMR, and ECS, can support these custom solutions. These services provide high performance and flexibility for advanced ML workloads.

Generative AI on AWS

• Describe deep learning and its subset, generative AI.
• Identify the types of generative AI solutions offered by AWS.

Deep learning (DL) is a subset of machine learning where models are trained using layers of artificial neurons that mimic the human brain. Each layer of these neural networks summarizes and feeds information to the next layer until a final model is produced.

Generative AI is a type of deep learning powered by extremely large ML models known as foundation models (FMs). FMs are pre-trained on vast collections of data. While traditional ML models are trained to perform singular tasks, FMs can be adapted to perform multiple tasks.

Large language models (LLMs), are a popular type of FM trained to use human language. Foundation models can also be used to create videos, images, music, and more.

Generative AI on AWS AWS offers the following types of generative AI solutions:

• SageMaker JumpStart: An ML hub with FMs and pre-built ML solutions deployable with a few clicks. JumpStart is a machine learning hub within SageMaker AI that accelerates the process of building, training, and deploying ML models. SageMaker JumpStart offers a library of pre-built ML solutions across various domains such as computer vision, NLP, and tabular data. These pre-trained models can be fine-tuned to suit your specific needs and deployed with just a few clicks.
• Bedrock: A fully managed service for adapting and deploying FMs from Amazon and other leading AI companies. It provides access to FMs from Amazon and leading AI startups, such as Claude and Stable Diffusion, all through a single unified API. With Amazon Bedrock, you can quickly experiment with FMs, fine-tune them with your own data, and seamlessly integrate them into your AWS applications.
• Amazon Q: An interactive AI assistant that can be integrated with a company's information repositories
  • Amazon Q Business: can answer pressing questions, help solve problems, and take actions using the data and expertise found in your company's information repositories.
  • Q Developer: provides code recommendations to accelerate development for coding languages i

AWS Generative AI Solutions

• Describe the benefits and purpose of the generative AI services offered by AWS.

This includes Amazon SageMaker JumpStart for accelerating model development and Amazon Bedrock for deploying high-performing FMs through a single API. Amazon Q integrates with your existing information repositories to answer questions and helps generate insights and new content.

Introduction to Data Analytics

• Describe the benefits and purpose of ETL data pipelines.
• Define data analytics.

Both AI/ML and traditional data analytics need clean and accessible data in a format that's usable by analytics tools and AI algorithms. ETL processes are used for this purpose. With ETL, you perform the following steps:

• Extract the data from various sources and store it.
• Transform it into a consistent, usable format for downstream tools to consume.
• Load it into a destination system, like a data warehouse or analytics platform.

Data pipelines are automated assembly lines used to make the ETL process efficient and repeatable. AWS has a suite of integrated services so you can build your own data pipelines. Data analytics is when analysts transform raw historical data to uncover valuable insights and trends.

Data Pipelines on AWS

• Describe the AWS services used in each phase of a typical AWS data pipeline.

Data ingestion services:

• Kinesis Data Streams: real-time ingestion of terabytes of data from applications, streams, and sensors. This serverless service even provides automatic provisioning and scaling in on-demand mode.
• Data Firehose: data ingestion in near real-time. This fully managed service provides automatic provisioning and scaling. It also delivers data within seconds to data lakes, warehouses, and analytics services.

Data storage services

Data can come from many different sources. To gain insights, data is commonly consolidated into a single location. There are two storage options for this. Flexible data lakes store vast amounts of raw data. Alternatively, the more structured data warehouses are optimized for business intelligence.

• S3: popular choice for data lakes. This object storage service can securely house virtually any amount of structured or unstructured data. Amazon S3 is also fully elastic, automatically scaling as you add and remove data.
• Redshift: fully managed data warehouse service that can store petabytes of structured or semistructured data. With the scalability and pay-as-you-go pricing model, organizations can cost-effectively analyze large datasets.

Data Cataloging Services

• Glue Data Catalog: provides a centralized, scalable, and managed metadata repository that enhances data discovery. It improves the overall efficiency of data pipelines by delivering metadata to various data stores and analytics services.

Data processing services

Data processing services clean and transform your data so it's ready to be analyzed.

• Glue: fully managed ETL service that makes data preparation simpler, faster, and cost effective. AWS Glue ETL jobs can use the AWS Glue Data Catalog to access metadata about data sources, which can help inform transformations defined in the ETL script.
• EMR: ideal for large-scale data processing and organizations with existing big data expertise. It automatically handles infrastructure provisioning, cluster management, and scaling. Amazon EMR supports popular big data frameworks like Apache Spark, Apache Hadoop, and Apache Hive.

Data analysis and visualization services

Queries and visualization tools help you to develop important insights about your data.

• Athena: you can run SQL queries to analyze data in relational, nonrelational, object, and custom data sources. This fully managed serverless service can access data hosted on Amazon S3, on premises, or even in multi-cloud environments. It offers a cost-effective solution for data analysis because you only pay for the queries you run.
• Redshift: fully managed data warehouse solution. Its columnar storage and massively parallel processing architecture make it ideal for analyzing large datasets. You can use it to perform complex SQL queries on large datasets for frequent, high-performance analytical workloads.
• QuickSight: both technical and non-technical users can quickly create modern interactive dashboards and reports from various data sources without managing infrastructure. Amazon Q in QuickSight provides natural language queries so business analysts and users can build, discover, and share meaningful insights in seconds.
• OpenSearch: you can search for relevant content through precise keyword matching or natural language queries. Unified dashboards provide real-time data visualization as you analyze and monitor logs, traces, and metrics for various applications.

Data Analytics and AI/ML

1. Recommendations
2. Store app Data
3. Kinesis ingests the data from Dynamo and then Firehose aggregates it.
4. Firehose uses a lambda function to convert json into csv
5. Firehose then delivers the data to S3 data lake where it is available for multiple consumers
6. Glue catalog serves as a metadata repository with tables that describe the schema and location of the S3 Data.
7. Athena can be used to gather insight through queries
8. Sagemaker AI can S3 data to train model.

Resources

Link	Description
Amazon Comprehend	Use natural language processing to extract key insights from documents.
Amazon Polly	Convert text into lifelike speech.
Amazon Transcribe	Convert speech into text.
Amazon Translate	Translate text into multiple languages.
Amazon Kendra	Use natural language processing to intelligently query enterprise content.
Amazon Rekognition	Identify objects and activities in images and videos.
Amazon Textract	Detect and extract typed and handwritten text in documents.
Amazon Lex	Add voice and text conversational interfaces to applications.
Amazon Personalize	Add personalized customer recommendations to applications.
Amazon SageMaker AI	Build, train, and deploy your own ML models without worrying about infrastructure.
Amazon SageMaker JumpStart	Deploy pre-trained ML solutions, like computer vision, NLP, and tabular data, with just a few clicks.
Amazon Bedrock	Fine-tune and seamlessly integrate large FMs from Amazon and leading AI startups into your AWS applications with a single API.
Amazon Q Business	Answer questions and solve problems using the data and expertise found in your company's information repositories.
Amazon Q Developer	Accelerate development with code recommendations.
Amazon Kinesis Data Streams	Ingest terabytes of streaming data in real time.
Amazon Data Firehose	Ingest and deliver data within seconds to multiple consumers, such as data lakes and warehouses.
Amazon S3	Store virtually limitless amounts of all types of data using this popular data lake choice.
Amazon Redshift	Store petabytes of structured or semistructured data in this data warehouse service. Analyze large datasets stored in the warehouse using complex SQL queries.
AWS Glue Data Catalog	Provide metadata to various analytics services with this centralized repository.
AWS Glue	Process data by using the AWS Glue Data Catalog as a reference.
Amazon EMR	Process big data workloads by using popular frameworks like Apache Spark.
Amazon Athena	Analyze various data sources hosted anywhere with a single SQL query.
Amazon QuickSight	Visualize data by creating interactive dashboards with or without expertise.
Amazon OpenSearch Service	Visualize and monitor real-time data analytics with keyword or NLP searches.