Veterans Track
Introduction to Data Architectures
Data architecture from a software architect's lens — covering ingestion, storage, modelling, governance, security, analytics, AI/ML, reliability, cloud scalability and DataOps.
Domain 01 · Data Sources & Ingestion
Data Sources
Files, RDBMS, unstructured data, streaming sources, APIs, webhooks and IoT data sources.
Data Integration
ETL, ELT, reverse ETL, CDC and API-based integration pipelines.
Ingestion Patterns & SLAs
Batch, micro-batch and real-time ingestion with latency SLAs, idempotency and dead-letter handling.
Data Contracts
Versioned producer-consumer agreements defining schema, SLAs and quality obligations.
Domain 02 · Storage Paradigms & Management
Warehouse, Lakes, Fabrics & Meshes
Data warehouse, data lake, data fabric, data mesh and lakehouse architecture.
Data Storage & Management
Hot, warm, cold and archive storage tiers with Parquet, ORC, Avro and compression strategies.
Master Data Management
Golden record creation, entity resolution, reference data and stewardship workflows.
Data Lifecycle Management
Retention schedules, archival triggers, legal hold and GDPR-compliant erasure workflows.
Domain 03 · Modelling, Transformation & Flow
Data Models
Conceptual, logical and physical models including relational, dimensional, document, graph and key-value models.
Transformation
Cleansing, enrichment, normalisation, aggregation and business logic using dbt, Spark or Flink.
Data Flow
End-to-end movement from source to consumption across batch, streaming and error paths.
Data Quality Framework
Completeness, accuracy, consistency, timeliness, validity and uniqueness checks.
Semantic & Metrics Layer
Consistent KPI definitions using dbt Metrics, Cube.dev, LookML and business glossary alignment.
Domain 04 · Security, Privacy & Compliance
Encryption
Encryption at rest, in transit, column-level encryption, KMS/HSM and key rotation.
Data Utilities
Masking, obfuscation, tokenisation, pseudonymisation, archival and secure deletion.
Data Governance & Security
Cataloging, metadata management, lineage tracking, access policies and stewardship workflows.
Compliance Standards
GDPR, CCPA, HIPAA, PCI-DSS, SOX and continuous compliance evidence.
Data Access Control
Row-level security, column-level security, ABAC, PBAC and just-in-time access.
Data Audit & Lineage
Column-level lineage, impact analysis and immutable audit logs for data access events.
Domain 05 · Processing, Analytics & AI/ML
Data Processing & Analytics
Batch, stream and real-time analytics using Spark, Flink and Kafka Streams.
Data Science
Exploratory analysis, statistical modelling, notebooks, sandboxed access and experiment tracking.
ML & AI Architecture
Feature store, model registry, model serving and drift detection.
MLOps & Model Lifecycle
Retraining pipelines, model CI/CD, A/B testing, explainability and fairness auditing.
Self-Service Analytics
Governed BI access through Tableau, Looker, Power BI and curated data products.
Domain 06 · Reliability, Observability & Testing
Data Integrity Protection
Referential integrity, constraints, ACID boundaries and checksums against silent corruption.
Data Observability
Freshness SLAs, anomaly detection, schema alerts and pipeline health monitoring.
Pipeline Reliability & SLAs
Retry logic, circuit breakers, checkpointing and automated incident escalation.
Data Disaster Recovery
RTO/RPO targets, warehouse snapshots, lake versioning and cross-region replication.
Data Testing Strategy
Unit tests, integration tests, schema validation and KPI regression tests.
Idempotency & Replayability
Replayable pipeline stages for backfills, incident recovery and audit reproducibility.
Domain 07 · Scalability, Portability & Cloud Architecture
Scalability
Horizontal scaling of compute, storage and query layers with cost guardrails.
Portability
Open formats and APIs to prevent vendor lock-in and support GDPR data export.
Cloud-Native & Hybrid
BigQuery, Redshift, Synapse, hybrid connectivity and unified identity.
FinOps & Data Cost Management
Query attribution, auto-suspend, egress optimisation and reserved capacity planning.
Domain 08 · DataOps, Delivery & Ecosystem
DataOps & DevOps
CI/CD for data pipelines, version-controlled transforms and IaC for data infrastructure.
Data Product Architecture
Data products with owners, SLAs, schema, access policy and quality contracts.
Metadata & Data Catalog
Active metadata, automated discovery, PII tagging, glossary and lineage propagation.
Data Sharing & Monetisation
Secure external sharing, data marketplaces and revenue attribution.
Data Platform Operating Model
Centralised, federated or hub-and-spoke operating model with SLA ownership.