Amazon Web Services (AWS)
AWS is the foundation for modern data platforms. Lambda, S3, Glue, and Athena provide serverless data processing at scale. Use for data lakes, ETL automation, and analytics infrastructure.
Recommended Tools
Data Engineering Tools & Partner Platforms
A curated collection of tools and platforms I recommend for data engineering, cloud platforms, and analytics. These are technologies I've worked with professionally.
Cloud Platforms
Enterprise Cloud Infrastructure
Snowflake
Purpose-built cloud data platform with separation of compute and storage. Excellent for analytics, data sharing, and Snowpark-based data engineering. Industry-leading performance and scalability.
Microsoft Azure
Azure Data Factory and Synapse Analytics provide enterprise-grade data integration. Strong for organizations already invested in Microsoft ecosystem. Seamless Active Directory integration.
Data Processing
ETL, ELT, and Analytics Engineering
Databricks
Unified data platform built on Apache Spark. Best-in-class for large-scale data processing, ML workflows, and Delta Lake. Multi-cloud support with Databricks SQL for analytics.
dbt (Data Build Tool)
Modern data transformation framework enabling software engineering best practices in analytics. Version control, testing, and documentation for SQL-based transformations. Integrates with all major platforms.
Apache Airflow
Open-source orchestration platform for complex data workflows. Programmatic workflow definition, rich UI, and extensive integrations. Industry standard for DAG-based orchestration and scheduling.
Matillion
Enterprise-grade cloud ETL platform. Visual no-code/low-code development environment. Native integrations with Snowflake, Databricks, and major cloud data warehouses. Excellent for rapid deployment.
Kafka
Distributed event streaming platform for real-time data pipelines. Handles millions of events per second with durability and fault tolerance. Critical for event-driven architectures and real-time analytics.
Qlik Replicate
Real-time data replication and CDC (Change Data Capture) solution. Supports heterogeneous environments with low latency. Essential for keeping data platforms in sync across systems.
Data Quality
Governance, Monitoring, and Data Trust
Great Expectations
Open-source data quality framework for Python-based validation. Define expectations for data quality, validate, and document. Integrates with major data platforms and orchestration tools.
Collibra
Enterprise data governance platform. Catalog, lineage, quality, and policy management. Strong for regulated industries and large organizations needing comprehensive data governance.
Alation
Data catalog and governance solution with AI-assisted learning. Captures institutional knowledge about data assets. Strong community features for democratizing data across organizations.
Development Tools
IDE, Version Control, and AI-Assisted Development
GitHub
Industry-standard version control and collaboration platform. GitHub Actions for CI/CD, GitHub Copilot for AI-assisted development. Essential infrastructure for modern teams.
Claude (Anthropic)
Advanced AI models for coding assistance and complex problem-solving. Available via API, web interface, and Claude Code (IDE plugin). Excellent for code review and engineering productivity.
VS Code
Lightweight, powerful code editor with extensive extensions. Supports Python, SQL, YAML, and every modern language. Essential tool for data engineers and developers.
Disclosure
Affiliate Links & Partnerships
Affiliate Disclosure: Some links on this page are affiliate or partner links. If you purchase or sign up through these links, I may earn a commission at no additional cost to you. These recommendations are genuine based on professional experience and proven value in data engineering projects.
Partnership: I work with these platforms professionally and can offer consulting, implementation support, and optimization for organizations looking to deploy or enhance their data infrastructure.