MLOps: Machine Learning Pipeline

Learn how to set up a complete MLOps pipeline for your ML projects.

Pipeline Setup

1. Project Structure

project/
├── data/
│   ├── raw/
│   └── processed/
├── models/
│   ├── trained/
│   └── deployed/
├── src/
│   ├── data/
│   ├── features/
│   ├── models/
│   └── utils/
├── tests/
├── notebooks/
├── configs/
└── .github/
    └── workflows/

2. DVC Configuration

# dvc.yaml
stages:
  prepare:
    cmd: python src/data/prepare.py
    deps:
      - data/raw
    outs:
      - data/processed
      
  train:
    cmd: python src/models/train.py
    deps:
      - data/processed
    outs:
      - models/trained
      
  evaluate:
    cmd: python src/models/evaluate.py
    deps:
      - models/trained
      - data/processed

Data Pipeline

1. Data Preparation

# src/data/prepare.py
import pandas as pd
from sklearn.model_selection import train_test_split

def prepare_data():
    # Loading
    data = pd.read_csv("data/raw/data.csv")
    
    # Cleaning
    data = clean_data(data)
    
    # Feature engineering
    data = engineer_features(data)
    
    # Split
    train, test = train_test_split(data, test_size=0.2)
    
    # Saving
    train.to_csv("data/processed/train.csv")
    test.to_csv("data/processed/test.csv")

2. Feature Engineering

# src/features/engineering.py
from sklearn.preprocessing import StandardScaler

def engineer_features(data):
    # Feature creation
    data["new_feature"] = data["feature1"] * data["feature2"]
    
    # Normalization
    scaler = StandardScaler()
    data[["feature1", "feature2"]] = scaler.fit_transform(data[["feature1", "feature2"]])
    
    return data

Training Pipeline

1. MLflow Configuration

# src/models/train.py
import mlflow
from mlflow.tracking import MlflowClient

def train_model():
    # MLflow configuration
    mlflow.set_tracking_uri("http://localhost:5000")
    mlflow.set_experiment("model_training")
    
    with mlflow.start_run():
        # Parameters
        params = {
            "learning_rate": 0.1,
            "epochs": 100,
            "batch_size": 32
        }
        
        # Training
        model = train(params)
        
        # Logging
        mlflow.log_params(params)
        mlflow.log_metrics({"accuracy": 0.95})
        mlflow.log_model(model, "model")

2. Model Evaluation

# src/models/evaluate.py
from sklearn.metrics import accuracy_score, precision_score, recall_score

def evaluate_model(model, test_data):
    # Predictions
    predictions = model.predict(test_data)
    
    # Metrics
    metrics = {
        "accuracy": accuracy_score(test_data.target, predictions),
        "precision": precision_score(test_data.target, predictions),
        "recall": recall_score(test_data.target, predictions)
    }
    
    return metrics

Deployment and Monitoring

1. Docker Configuration

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "src/app.py"]

2. Monitoring with Prometheus

# src/monitoring/metrics.py
from prometheus_client import Counter, Histogram

# Metrics
PREDICTIONS = Counter('model_predictions_total', 'Total predictions')
LATENCY = Histogram('model_latency_seconds', 'Prediction latency')

def track_prediction(latency):
    PREDICTIONS.inc()
    LATENCY.observe(latency)

CI/CD with GitHub Actions

1. Training Pipeline

# .github/workflows/train.yml
name: Train Model

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.9'
          
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          
      - name: Train model
        run: dvc repro train
        
      - name: Upload model
        uses: actions/upload-artifact@v2
        with:
          name: model
          path: models/trained

2. Deployment Pipeline

# .github/workflows/deploy.yml
name: Deploy Model

on:
  workflow_run:
    workflows: ["Train Model"]
    types:
      - completed

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Download model
        uses: actions/download-artifact@v2
        with:
          name: model
          
      - name: Build and push Docker image
        run: |
          docker build -t model-app .
          docker push model-app
          
      - name: Deploy to production
        run: |
          kubectl apply -f k8s/

Best Practices

1. Versioning

   • Code
   • Data
   • Models
   • Configurations

2. Monitoring

   • Metrics
   • Logs
   • Alerts
   • Performance

3. Security

   • Secrets
   • Permissions
   • Validation
   • Auditing

Conclusion

Key takeaways:

• Automated pipeline
• Complete versioning
• Effective monitoring
• Secure deployment

Recommendations:

• Automate your processes
• Track your metrics
• Document changes
• Keep learning about MLOps
• Maintain quality