This long course develops skills for operational analytics, secure data practices, and governance essential to building trustworthy, auditable agentic systems. You will aggregate and analyze operational metrics, design A/B experiments and statistical tests to validate agent improvements, and craft clear visualizations and alerting rules for stakeholders. The course covers end-to-end data hygiene: cleaning, schema validation, reproducible notebooks with data versioning, and trade-offs between sample size and noise for experimental design. It also addresses security and governance: securing API endpoints per OWASP ASVS, dependency vulnerability analysis, secret-management trade-offs (on-prem vs managed), and threat modeling (STRIDE). Practical tasks include building DBT models for telemetry, configuring alerts, producing reproducible analytic notebooks, and creating STRIDE diagrams with documented mitigations to reduce operational and supply-chain risk.
This module trains data analysts, ML engineers, and developers to optimize AI agents built with frameworks like LangChain and Autogen and learn to prove the effectiveness of the agents. You will transform raw logs into actionable KPIs using SQL and dbt, design and execute A/B tests to compare agent versions, and apply statistical methods like the Chi-square test to validate your results. This course equips you to make objective, evidence-based recommendations for deploying agent enhancements, moving from correlation to causation and ensuring your improvements are statistically significant.
涵盖的内容
5个视频2篇阅读材料4个作业1个非评分实验室
显示有关单元内容的信息
5个视频•总计27分钟
Defining Agent Success: From Vanity Metrics to Actionable KPIs•6分钟
The Modern Data Stack for AI•6分钟
Correlation is not Causation•5分钟
Running a Chi-square Test•5分钟
Non-Parametric Tests•6分钟
2篇阅读材料•总计15分钟
Advanced Time-Series Aggregation: Windows, Bucketing, and Operational Definitions•7分钟
Principles of A/B Testing•8分钟
4个作业•总计70分钟
Agent Performance Analysis Report•30分钟
Build an Agent Performance Data Model•20分钟
Knowledge Check: Data Transformation for Business Intelligence•10分钟
Knowledge Check: Statistical Significance in Agent Experiments•10分钟
1个非评分实验室•总计25分钟
Analyze a Controlled Experiment•25分钟
Visualize and Alert AI Performance KPIs
第 2 单元•小时 后完成
单元详情
This module is for training data analysts, ML engineers, and product managers to monitor the operational health of AI systems by focusing on cost, latency, and impact. You will master data storytelling, transforming complex performance data into clear, compelling visualizations that drive decisions. Through hands-on labs, you will learn to build proactive monitoring systems by defining critical KPIs, setting precise thresholds, and configuring automated alerts. By the end, you can create dashboards that empower leadership and build automated defenses to protect your AI systems from budget overruns and performance degradation, ensuring real-world success.
涵盖的内容
4个视频4篇阅读材料4个作业1个非评分实验室
显示有关单元内容的信息
4个视频•总计23分钟
Dashboard Failure: The Cost of Clutter•6分钟
Choosing the Right Visualization Type•5分钟
The High Cost of Unmonitored AI•8分钟
How to Configure an Alert in a BI Tool•4分钟
4篇阅读材料•总计29分钟
What Makes a Visualization Effective?•10分钟
How to Redesign a Cluttered Chart•7分钟
What is an Effective Alerting System?•6分钟
Best Practices for Alerting•6分钟
4个作业•总计68分钟
Visualizing and Alerting on AI KPIs•30分钟
Knowledge Check: Data Visualization Best Practices•10分钟
Hands-On Learning: Designing a Cost Management Alerting Plan•18分钟
Knowledge Check: Proactive Alerting for AI Cost and Performance Management
•10分钟
1个非评分实验室•总计25分钟
Redesigning a Performance Visualization•25分钟
Clean, Analyze, and Visualize Your Data
第 3 单元•小时 后完成
单元详情
This module, designed for aspiring AI and data professionals, provides hands-on experience in data preparation and exploration. You will learn to build world-class models on high-quality data by implementing systematic cleaning and validation routines with tools like Pandera. In guided Jupyter labs, you will master statistical visualization and dimensionality reduction techniques, such as t-SNE, to transform complex data into clear, interpretable plots. Uncover hidden patterns, diagnose issues, and derive key insights. You'll move beyond just cleaning data to truly understanding it, ensuring your AI development is built on a solid foundation.
涵盖的内容
3个视频2篇阅读材料3个作业2个非评分实验室
显示有关单元内容的信息
3个视频•总计13分钟
How to Build a Validation Schema with Pandera•4分钟
Seeing the Unseen: Finding a Hidden Error Cluster•5分钟
How to Create and Interpret a t-SNE Plot•5分钟
2篇阅读材料•总计18分钟
The Data Wrangler's Toolkit: Core Cleaning Concepts•8分钟
Taming the Dimensions: An Introduction to t-SNE and PCA•10分钟
3个作业•总计55分钟
Report: From Data Cleaning to Visual Insight•30分钟
Data Validation and Imputation: Quiz •15分钟
Analyzing a New Visualization •10分钟
2个非评分实验室•总计40分钟
Cleaning a Raw Customer Dataset•20分钟
Visualizing Message Embeddings to Find Errors•20分钟
Evaluate and Reproduce Data Findings Fast
第 4 单元•小时 后完成
单元详情
This module helps data scientists and analysts deliver efficient, trustworthy results. Tackle critical questions like, "Is our data sufficient?" and "Are our findings replicable?" Learn statistical power analysis to optimize sample sizes, preventing wasted resources. You will master reproducible workflows by parameterizing Jupyter notebooks with Papermill and versioning data with DVC. Move beyond simple scripts to build robust, automated analytical projects that accelerate innovation and foster a culture of trust, ensuring your findings can be validated by peers and stakeholders.
涵盖的内容
3个视频2篇阅读材料4个作业1个非评分实验室
显示有关单元内容的信息
3个视频•总计17分钟
The Trade-Off Triangle: Sample Size, Noise, and Confidence•6分钟
Why Reproducibility Matters?•4分钟
How to Build a Reproducible Notebook?•7分钟
2篇阅读材料•总计14分钟
The Point of Diminishing Returns•7分钟
The Reproducibility Toolkit: Papermill and DVC•7分钟
4个作业•总计85分钟
Reproducible Data Analysis Project•30分钟
Hands-On Learning: Analyzing Sample Size and Diminishing Returns•25分钟
Knowledge Check: Sampling Strategy Concepts•10分钟
Knowledge Check•20分钟
1个非评分实验室•总计60分钟
Creating a Reproducible Workflow•60分钟
Secure AI: API and Dependency Risks
第 5 单元•小时 后完成
单元详情
This module transforms developers into defenders, teaching you to build secure, production-grade AI. Learn to harden API endpoints using OWASP guidelines by implementing JWT authentication, input validation, and rate limiting. Adopt an attacker’s mindset, using DAST tools like OWASP ZAP to verify your defenses. You'll master software supply chain security by analyzing vulnerabilities, prioritizing threats with the CVSS framework, and creating hotfix and rollback plans. Through hands-on labs simulating real security incidents, you will be prepared to build and deploy resilient AI services against modern threats.
涵盖的内容
4个视频4篇阅读材料5个作业
显示有关单元内容的信息
4个视频•总计17分钟
JWT: Authentication and Access Control in AI Services•4分钟
The Tester's Mindset: From Coder to Attacker•4分钟
CVSS Explained: Technical Severity vs. Contextual Risk•5分钟
Hotfix Strategy: Compatibility and Rollback Planning•4分钟
4篇阅读材料•总计27分钟
Securing the Gates: The OWASP API Security Top 10•5分钟
Input Validation: The Primary Defense Against Injection•7分钟
The Log4j Case Study: Anatomy of a Supply Chain Crisis•7分钟
The CVSS Framework: A Deeper Dive•8分钟
5个作业•总计60分钟
Security Portfolio and SecOps Defense•15分钟
Hands-On Learning: Implement Authentication and Validation Guards•10分钟
Hands-On Learning: Verification with Dynamic Security Testing (DAST)•15分钟
Response: Defending Against the Next Attack•10分钟
Hands-On Learning: Scan Report Analysis: Spotting the Critical CVE in urllib3•10分钟
Secure Your AI: Threat Modeling
第 6 单元•小时 后完成
单元详情
This module teaches architects and engineers to build resilience directly into AI system designs. You'll master secret management by comparing self-hosted (Vault) and cloud (AWS Secrets Manager) solutions, using Total Cost of Ownership (TCO) analysis to make a justifiable recommendation. Learn to proactively hunt for vulnerabilities by deconstructing architecture with Data Flow Diagrams and applying the STRIDE framework to mitigate threats. Through hands-on projects, you will draft professional security documents, defend your decisions, and gain the skills to design, build, and maintain secure AI systems from the ground up.
涵盖的内容
4个视频5篇阅读材料6个作业
显示有关单元内容的信息
4个视频•总计19分钟
TCO and Compliance: A Cost-Benefit Deep Dive•5分钟
Architect's Choice: Documenting Your Recommendation•6分钟
DFDs and Trust Boundaries: Decomposing AI Architecture•5分钟
STRIDE in Practice: Identifying Spoofing and Information Disclosure•3分钟
5篇阅读材料•总计30分钟
Cloud vs. On-Prem: The Secret Management Trade-off•7分钟
Integration and Latency: Prototyping Your Connection•6分钟
The Power of Proactivity: Threat Modeling in DevSecOps•6分钟
STRIDE: Your Framework for Systematic Threat Identification•6分钟
Targeted Mitigations: Countering Spoofing and Info Disclosure•5分钟
6个作业•总计76分钟
Architectural Review and Mitigation Proposal•16分钟
Hands-On Learning: Prototype and Compare Solutions•15分钟
Hands-On Learning: Draft the Technical Recommendation•10分钟
Justification of Secret Management Decision•10分钟
Hands-On Learning: Scan Report Analysis: Diagramming the Chat-Agent•10分钟
Hands-On Learning: STRIDE Analysis and Mitigation Plan•15分钟
Governance, Alerts and Analytics
第 7 单元•小时 后完成
单元详情
In this hands-on module, you'll master governance, alerting, and analytics by building a complete, reproducible telemetry-to-alert pipeline. Using automated notebooks, you will construct a workflow that ingests raw system data and generates critical, real-time alerts. To embed security directly into your design, you will apply the industry-standard STRIDE framework to develop a proactive threat model, identifying and mitigating vulnerabilities before they are exploited. This module will equip you with the skills to translate data into actionable intelligence, creating a robust, automated system for maintaining secure and reliable operations in a production environment.
涵盖的内容
2篇阅读材料1个作业
显示有关单元内容的信息
2篇阅读材料•总计30分钟
Why This Project Matters: Building Trust in Automated Systems •5分钟
Your Project Blueprint: Requirement and Evaluation•25分钟
Coursera brings together a diverse network of subject matter experts who have demonstrated their expertise through professional industry experience or strong academic backgrounds. These instructors design and teach courses that make practical, career-relevant skills accessible to learners worldwide.
Is Analyzing and Securing AI System Performance suitable for beginners?
This course assumes practical ML and engineering experience. Beginners should complete foundational ML and data-engineering courses first to gain the necessary background for the labs.
What hands-on projects are included in this course?
Labs include building telemetry-to-alert pipelines, creating DBT models and reproducible notebooks, configuring dashboards and alerts, and producing STRIDE threat models with mitigations suitable for a portfolio artifact.
What tools will I use in the course?
The curriculum references telemetry tooling, DBT, reproducible notebooks, and dependency scanners. Exact tool choices and versions will be confirmed by instructors and may vary by offering.
When will I have access to the lectures and assignments?
To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
What will I get if I subscribe to this Certificate?
When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.