Fundamentals of DataOps

A Continuous Delivery Foundation Initiative

Report a Bug · Request Content · Ask a Question

Table of Contents

• About
• Curriculum
• Getting Started
  • Prerequisites
• Roadmap
• Contributing
• Support
• License
• Acknowledgements

---

About

Organizations face complex challenges in the end-to-end deployment of machine learning applications and models, from initial development to operational maintenance. This process requires seamless integration of CI/CD practices, containerization, data infrastructure, MLOps, and security measures.

This repo contains technical material and guides for users interested in the end-to-end process of deploying machine learning applications and models within their organizations. By developing an inclusive set of DataOps and DevOps best practices for engineers, we can empower developers, architects, and decision-makers to effectively leverage open-source tools and frameworks for streamlined, secure, and scalable ML application deployment.

Curriculum

01. Fundamentals

• DataOps vs DevOps
• DataOps Philosophy
• Organizational DataOps

02. Platform

• Example Architecture
• Team Organization
• Data Architecture

03. Operations I

• Pipeline Orchestration
• CI/CD for Data Pipelines
• Data Quality
• Data Contracts

04. Operations II

• Data Governance
• Observability
• Cloud Native Data
• Securing your Data Pipelines

05. AI/ML

• Realtime ML
• MLOps and Monitoring Models
• Security for AI/ML

Getting Started

Prerequisites

Usage

Roadmap

• Develop a series of high-level blog posts to raise awareness and flesh out the course material, test out ideas, in conjunction with OPEA and the CDF
• Develop the course materials, including practical implementations and code checks, set up environments for developer and user use
• Publish the course on Linux Foundation Training as a certification

Contributing

This initiative is spearheaded by the Continuous Delivery Foundation and is actively looking to collaborate with other members and organizations within and outside of the Linux Foundation as part of this.

Please try to create bug reports that are:

• Reproducible. Include steps to reproduce the problem.
• Specific. Include as much detail as possible: which version, what environment, etc.
• Unique. Do not duplicate existing opened issues.
• Scoped to a Single Bug. One bug per report.

Please adhere to this project's code of conduct.

Contribute

Links

License

This project is licensed under the Apache 2.0 license.

See LICENSE for more information.

Acknowledgements