AI - MLOps & Production Systems

Machine Learning Operations (MLOps) encompasses the engineering practices, infrastructure platforms, and automation pipelines required to reliably deploy, scale, and monitor machine learning and AI models in production.

Unlike traditional software engineering, MLOps systems must manage dual versioning: code artifacts and data/model artifacts.

The Production ML Pipeline Architecture

A production ML platform connects raw data systems to real-time serving endpoints:

[Raw Event Streams / Databases]
            │
            ▼
[Data Contract Validation & Schemas]
            │
            ├──────────────────────────┐
            ▼                          ▼
  [Offline Feature Store]    [Online Feature Store]
   (Parquet / BigQuery)       (Redis / Aerospike)
            │                          │
            ▼                          │ (Low-latency lookup)
    [Model Training]                   │
   (PyTorch / XGBoost)                 │
            │                          │
            ▼                          │
 [Model Registry & ONNX Export]        │
            │                          │
            ▼                          ▼
   [Inference Engine (Triton / vLLM / ONNX Runtime)]
            │
            ▼
   [Monitoring: Drift, PSI, Latency, Calibration]

Feature Stores & Data Contracts

A primary failure mode in machine learning systems is training-serving skew—discrepancies between features computed during historical model training and features computed during real-time inference.

1. Dual-Store Architecture

• Offline Store (Batch Training): Large-scale columnar storage (Parquet on S3, Delta Lake, Snowflake, BigQuery) optimized for historical feature extraction across millions of rows with point-in-time correctness ("time travel").
• Online Store (Real-Time Serving): In-memory key-value stores (Redis, Aerospike, DynamoDB) designed for sub-10ms point lookups of user and entity features during live API calls.
• Popular Frameworks: Feast, Hopsworks, Tecton.

2. Data Contracts

Data contracts define strict schema definitions and validation rules between upstream engineering services producing raw event data and downstream ML feature pipelines. They enforce:

• Type safety and schema evolution rules (e.g., Protobuf, Avro, JSON Schema).
• Nullability, range constraints, and distribution bounds to prevent silent model performance degradation.

Model Packaging & Neutral Interchange Formats

Models trained in research frameworks (Python, PyTorch) frequently need to execute in performance-critical production runtimes (C++, Go, Java, mobile edge).

1. ONNX (Open Neural Network Exchange)

• Standardized open format representing neural networks and classical ML models as directed acyclic computation graphs (DAGs).
• ONNX Runtime (ORT): Cross-platform engine optimizing execution across CPU, GPU (CUDA, TensorRT), DirectML, and Apple CoreML with zero Python runtime dependency.
• Eliminates Python GIL limitations and reduces inference memory footprint.

2. Legacy Interchange Standards

• PMML (Predictive Model Markup Language): XML-based standard for classical statistical models (decision trees, linear models, clustering).
• PFA (Portable Format for Analytics): JSON/YAML representation intended as a functional programming successor to PMML.

Serving Strategies

Serving Pattern	Typical Latency	Throughput	Use Cases
Real-Time Microservice	5ms – 50ms	1K – 50K RPS	Fraud detection, click-through rate (CTR) prediction, search ranking
Batch Scoring	Minutes – Hours	Millions of records	Customer churn scoring, credit risk batch updates, daily recommendation caches
Streaming / Event-Driven	50ms – 500ms	Millions/sec	Kafka/Flink event processing, anomaly detection, IoT sensor alerts
Edge / Embedded	1ms – 20ms	Local compute	Autonomous driving, on-device mobile voice processing, embedded robotics

Safe Deployment Strategies

1. Shadow Deployment (Dark Traffic): The new model candidate receives a mirror copy of real-time production traffic. Predictions are logged and evaluated for latency and error distribution, but not returned to users.
2. Canary Rollout: Routing a small percentage (1% to 5%) of live requests to the new model, gradually scaling up to 100% while monitoring error rates and conversion metrics.
3. Model Rollback: Instantaneous routing rollback to the previous model artifact in the registry if degradation is observed.

Continuous Monitoring & Drift Detection

Once a model is live, its predictive performance inevitably decays over time due to changing real-world conditions:

• Data Drift (Covariate Shift): The distribution of input features $P(X)$ changes between training and inference (e.g., user demographics change after a marketing campaign).
• Concept Drift: The statistical relationship between inputs and outputs $P(Y|X)$ changes (e.g., consumer behavior shifts during economic changes).
• Monitoring Metrics:
  • Population Stability Index (PSI): Quantifies shifts between reference and current feature distributions ($PSI < 0.1$ indicates stability, $PSI > 0.2$ indicates significant drift requiring retraining).
  • Two-Sample Tests: Kolmogorov-Smirnov (K-S) test for numerical feature shifts and Chi-square tests for categorical features.
  • Calibration Tracking: Monitoring model calibration ratios ($\text{model_ctr} / \text{true_ctr}$) to detect over- or under-confidence before accuracy degrades.