MLOps: pipeline de Machine Learning

Descubre cómo implementar un pipeline MLOps completo para tus proyectos de ML.

Configuración del pipeline

1. Estructura del proyecto

project/
├── data/
│   ├── raw/
│   └── processed/
├── models/
│   ├── trained/
│   └── deployed/
├── src/
│   ├── data/
│   ├── features/
│   ├── models/
│   └── utils/
├── tests/
├── notebooks/
├── configs/
└── .github/
    └── workflows/

2. Configuración de DVC

# dvc.yaml
stages:
  prepare:
    cmd: python src/data/prepare.py
    deps:
      - data/raw
    outs:
      - data/processed
      
  train:
    cmd: python src/models/train.py
    deps:
      - data/processed
    outs:
      - models/trained
      
  evaluate:
    cmd: python src/models/evaluate.py
    deps:
      - models/trained
      - data/processed

Pipeline de datos

1. Preparación de los datos

# src/data/prepare.py
import pandas as pd
from sklearn.model_selection import train_test_split

def prepare_data():
    # Carga
    data = pd.read_csv("data/raw/data.csv")
    
    # Limpieza
    data = clean_data(data)
    
    # Feature engineering
    data = engineer_features(data)
    
    # Split
    train, test = train_test_split(data, test_size=0.2)
    
    # Guardado
    train.to_csv("data/processed/train.csv")
    test.to_csv("data/processed/test.csv")

2. Feature engineering

# src/features/engineering.py
from sklearn.preprocessing import StandardScaler

def engineer_features(data):
    # Creación de features
    data["new_feature"] = data["feature1"] * data["feature2"]
    
    # Normalización
    scaler = StandardScaler()
    data[["feature1", "feature2"]] = scaler.fit_transform(data[["feature1", "feature2"]])
    
    return data

Pipeline de entrenamiento

1. Configuración de MLflow

# src/models/train.py
import mlflow
from mlflow.tracking import MlflowClient

def train_model():
    # Configuración de MLflow
    mlflow.set_tracking_uri("http://localhost:5000")
    mlflow.set_experiment("model_training")
    
    with mlflow.start_run():
        # Parámetros
        params = {
            "learning_rate": 0.1,
            "epochs": 100,
            "batch_size": 32
        }
        
        # Entrenamiento
        model = train(params)
        
        # Logging
        mlflow.log_params(params)
        mlflow.log_metrics({"accuracy": 0.95})
        mlflow.log_model(model, "model")

2. Evaluación de los modelos

# src/models/evaluate.py
from sklearn.metrics import accuracy_score, precision_score, recall_score

def evaluate_model(model, test_data):
    # Predicciones
    predictions = model.predict(test_data)
    
    # Métricas
    metrics = {
        "accuracy": accuracy_score(test_data.target, predictions),
        "precision": precision_score(test_data.target, predictions),
        "recall": recall_score(test_data.target, predictions)
    }
    
    return metrics

Despliegue y monitorización

1. Configuración de Docker

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "src/app.py"]

2. Monitorización con Prometheus

# src/monitoring/metrics.py
from prometheus_client import Counter, Histogram

# Métricas
PREDICTIONS = Counter('model_predictions_total', 'Total de predicciones')
LATENCY = Histogram('model_latency_seconds', 'Latencia de las predicciones')

def track_prediction(latency):
    PREDICTIONS.inc()
    LATENCY.observe(latency)

CI/CD con GitHub Actions

1. Pipeline de entrenamiento

# .github/workflows/train.yml
name: Train Model

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.9'
          
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          
      - name: Train model
        run: dvc repro train
        
      - name: Upload model
        uses: actions/upload-artifact@v2
        with:
          name: model
          path: models/trained

2. Pipeline de despliegue

# .github/workflows/deploy.yml
name: Deploy Model

on:
  workflow_run:
    workflows: ["Train Model"]
    types:
      - completed

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Download model
        uses: actions/download-artifact@v2
        with:
          name: model
          
      - name: Build and push Docker image
        run: |
          docker build -t model-app .
          docker push model-app
          
      - name: Deploy to production
        run: |
          kubectl apply -f k8s/

Buenas prácticas

1. Versionado

   • Código
   • Datos
   • Modelos
   • Configuraciones

2. Monitorización

   • Métricas
   • Logs
   • Alertas
   • Rendimiento

3. Seguridad

   • Secretos
   • Permisos
   • Validación
   • Auditoría

Conclusión

Puntos clave que debes recordar:

• Pipeline automatizado
• Versionado completo
• Monitorización eficaz
• Despliegue seguro

Recomendaciones:

• Automatizar los procesos
• Seguir las métricas
• Documentar los cambios
• Formarse en MLOps
• Mantener la calidad