MLOps: Pipeline de Machine Learning

Descubra como implementar um pipeline MLOps completo para seus projetos de ML.

Configuração do Pipeline

1. Estrutura do Projeto

project/
├── data/
│   ├── raw/
│   └── processed/
├── models/
│   ├── trained/
│   └── deployed/
├── src/
│   ├── data/
│   ├── features/
│   ├── models/
│   └── utils/
├── tests/
├── notebooks/
├── configs/
└── .github/
    └── workflows/

2. Configuração do DVC

# dvc.yaml
stages:
  prepare:
    cmd: python src/data/prepare.py
    deps:
      - data/raw
    outs:
      - data/processed
      
  train:
    cmd: python src/models/train.py
    deps:
      - data/processed
    outs:
      - models/trained
      
  evaluate:
    cmd: python src/models/evaluate.py
    deps:
      - models/trained
      - data/processed

Pipeline de Dados

1. Preparação dos Dados

# src/data/prepare.py
import pandas as pd
from sklearn.model_selection import train_test_split

def prepare_data():
    # Carregamento
    data = pd.read_csv("data/raw/data.csv")
    
    # Limpeza
    data = clean_data(data)
    
    # Feature engineering
    data = engineer_features(data)
    
    # Split
    train, test = train_test_split(data, test_size=0.2)
    
    # Salvamento
    train.to_csv("data/processed/train.csv")
    test.to_csv("data/processed/test.csv")

2. Feature Engineering

# src/features/engineering.py
from sklearn.preprocessing import StandardScaler

def engineer_features(data):
    # Criação de features
    data["new_feature"] = data["feature1"] * data["feature2"]
    
    # Normalização
    scaler = StandardScaler()
    data[["feature1", "feature2"]] = scaler.fit_transform(data[["feature1", "feature2"]])
    
    return data

Pipeline de Treinamento

1. Configuração do MLflow

# src/models/train.py
import mlflow
from mlflow.tracking import MlflowClient

def train_model():
    # Configuração do MLflow
    mlflow.set_tracking_uri("http://localhost:5000")
    mlflow.set_experiment("model_training")
    
    with mlflow.start_run():
        # Parâmetros
        params = {
            "learning_rate": 0.1,
            "epochs": 100,
            "batch_size": 32
        }
        
        # Treinamento
        model = train(params)
        
        # Logging
        mlflow.log_params(params)
        mlflow.log_metrics({"accuracy": 0.95})
        mlflow.log_model(model, "model")

2. Avaliação dos Modelos

# src/models/evaluate.py
from sklearn.metrics import accuracy_score, precision_score, recall_score

def evaluate_model(model, test_data):
    # Previsões
    predictions = model.predict(test_data)
    
    # Métricas
    metrics = {
        "accuracy": accuracy_score(test_data.target, predictions),
        "precision": precision_score(test_data.target, predictions),
        "recall": recall_score(test_data.target, predictions)
    }
    
    return metrics

Deploy e Monitoramento

1. Configuração do Docker

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "src/app.py"]

2. Monitoramento com Prometheus

# src/monitoring/metrics.py
from prometheus_client import Counter, Histogram

# Métricas
PREDICTIONS = Counter('model_predictions_total', 'Total de previsões')
LATENCY = Histogram('model_latency_seconds', 'Latência das previsões')

def track_prediction(latency):
    PREDICTIONS.inc()
    LATENCY.observe(latency)

CI/CD com GitHub Actions

1. Pipeline de Treinamento

# .github/workflows/train.yml
name: Train Model

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.9'
          
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          
      - name: Train model
        run: dvc repro train
        
      - name: Upload model
        uses: actions/upload-artifact@v2
        with:
          name: model
          path: models/trained

2. Pipeline de Deploy

# .github/workflows/deploy.yml
name: Deploy Model

on:
  workflow_run:
    workflows: ["Train Model"]
    types:
      - completed

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Download model
        uses: actions/download-artifact@v2
        with:
          name: model
          
      - name: Build and push Docker image
        run: |
          docker build -t model-app .
          docker push model-app
          
      - name: Deploy to production
        run: |
          kubectl apply -f k8s/

Boas Práticas

1. Versionamento

   • Código
   • Dados
   • Modelos
   • Configurações

2. Monitoramento

   • Métricas
   • Logs
   • Alertas
   • Desempenho

3. Segurança

   • Secrets
   • Permissões
   • Validação
   • Auditoria

Conclusão

Pontos-chave a lembrar:

• Pipeline automatizado
• Versionamento completo
• Monitoramento eficiente
• Deploy seguro

Recomendações:

• Automatizar os processos
• Acompanhar as métricas
• Documentar as mudanças
• Estudar MLOps
• Manter a qualidade