Senior Data Engineer Azure, Databricks & ML Pipelines | Remote

About the Role -------------- We're looking for a Senior Data Engineer to design, build, and maintain scalable data pipelines and ML-ready infrastructure on Azure and Databricks. This is a hands-on engineering role: you'll own the full data pipeline lifecycle ingestion, transformation, orchestration, and deployment while supporting machine learning workflows with clean, reliable data. If you're comfortable owning infrastructure decisions and writing production-quality Python at scale, this role is built for that. What You'll Do -------------- * Design, build, and maintain data pipelines using Databricks and Azure-native data services * Develop and optimize ETL/ELT processes to support analytics and machine learning workloads * Build and maintain CI/CD pipelines for data engineering and ML deployment workflows * Write clean, efficient, production-quality Python for data processing and pipeline automation * Support machine learning teams with well-structured, high-quality datasets and feature pipelines * Design and manage data architecture across Azure services (e.g., Azure Data Factory, Azure Data Lake, Azure Synapse) * Monitor pipeline performance, troubleshoot data quality issues, and implement reliability improvements * Implement data governance, security, and access control best practices * Collaborate with data scientists, analysts, and software engineers to align data infrastructure with business needs * Participate in code reviews, architecture discussions, and technical planning What You Bring -------------- * Strong hands-on experience with Azure cloud data services * Proven experience building and maintaining pipelines on Databricks * Solid experience designing and managing CI/CD pipelines for data or ML workflows * Strong Python skills for data engineering and pipeline development * Working knowledge of machine learning workflows and how data engineering supports them * Experience with SQL and relational/distributed data systems * Understanding of data pipeline orchestration, monitoring, and reliability practices * Strong problem-solving skills and ability to work independently on complex data infrastructure challenges * Solid communication skills for collaborating with data science and engineering teams Nice to Have ------------ * Experience with MLOps practices and tools (MLflow, Azure ML) * Familiarity with Spark internals and performance tuning within Databricks * Experience with infrastructure-as-code (Terraform, Bicep, ARM templates) * Exposure to real-time/streaming data pipelines (Kafka, Event Hubs, Structured Streaming) * Relevant Azure or Databricks certifications Why This Role ------------- * Full pipeline ownership: Own data infrastructure end to end, from ingestion through ML-ready delivery * Modern data stack: Work with Azure and Databricks, leading platforms in enterprise data engineering * Cross-functional impact: Directly enable machine learning and analytics outcomes, not just move data * Flexibility: Remote-friendly engagement structure How to Apply ------------ Ready to bring your data engineering expertise to Azure and Databricks-powered ML infrastructure? Apply through Toptal here: https://www.toptal.com/talent/apply