We are seeking an experienced Data Architect with hands-on expertise in Azure Databricks, Azure Data Lakehouse, and Medallion Architecture. The ideal candidate will have strong implementation experience, including the integration of Delta Lake tables, and a background in working with Customer Data Platforms (CDP). This role requires a deep understanding of cloud-native data solutions and the ability to architect and deliver scalable, high-performing data platforms that can support advanced analytics, machine learning, and business intelligence solutions.
Key Responsibilities:
- Design and Implement Data Architecture: Lead the design and development of scalable, cloud-based data architectures leveraging Azure Data Lakehouse and Medallion Architecture principles.
- Azure Databricks and Delta Lake: Architect and implement data pipelines and ETL processes using Azure Databricks, ensuring seamless integration with Delta Lake to enable ACID transactions, time travel, and optimized data storage.
- Medallion Architecture: Implement the Medallion Architecture pattern, building Bronze, Silver, and Gold layers to ensure efficient data processing, aggregation, and enrichment for analytics and reporting.
- Customer Data Platform (CDP) Integration: Design and manage the data architecture to effectively handle customer data, ensuring the Customer Data Platform (CDP) is integrated with the Azure data ecosystem, enabling personalized and customer-centric analytics.
- Data Lakehouse Management: Lead the implementation and optimization of Azure Data Lake Storage (ADLS) to support structured, semi-structured, and unstructured data storage, ensuring efficient querying and data retrieval.
- Data Governance and Security: Define and implement robust data governance, security, and compliance practices using tools like Azure Data Catalog, Azure Purview, and Azure Key Vault. Ensure data is securely stored and accessed in compliance with data privacy regulations (GDPR, CCPA).
- Performance Optimization: Optimize data pipelines for performance, scalability, and cost-efficiency, ensuring that data is processed efficiently and meets SLAs for downstream analytics and reporting systems.
- Collaboration with Data Engineers and Analysts: Work closely with data engineers, business analysts, and data scientists to define data requirements, ensuring that data pipelines are designed to meet business needs.
- Real-Time Data Integration: Architect real-time data ingestion pipelines, integrating with various data sources (APIs, event hubs, etc.), and streamlining real-time analytics using Azure Databricks and Delta Lake.
- Documentation and Best Practices: Establish best practices for data architecture, provide documentation for data pipelines and architecture, and ensure knowledge sharing across the organization.
Required Qualifications:
- 5+ years of experience in data architecture and engineering, with expertise in Azure Databricks, Azure Data Lakehouse, and Delta Lake.
- Hands-on experience implementing Medallion Architecture, building and managing Bronze, Silver, and Gold data layers.
- Strong experience with Azure Data Lake Storage (ADLS) and integration with Delta Lake for efficient data storage and querying.
- Customer Data Platform (CDP) experience, with the ability to design and integrate customer-centric data architectures that drive personalized analytics.
- Experience in data governance, security, and compliance using Azure tools like Azure Purview, Data Catalog, and Key Vault.
- Proven experience in ETL/ELT development and real-time data ingestion pipelines, working with large-scale datasets.
- Strong knowledge of SQL, Python, and Spark for data processing, analysis, and pipeline development.
- Familiarity with data modeling techniques, big data technologies, and data warehousing in a cloud environment.
- Excellent communication and collaboration skills, with the ability to work cross-functionally and lead data-driven initiatives.
Preferred Qualifications:
- Experience with machine learning and advanced analytics workflows using Azure Databricks.
- Familiarity with API integration and real-time data streaming solutions like Azure Event Hubs or Azure Stream Analytics.
- Knowledge of DevOps for data, including automated CI/CD pipelines for data deployments.
Education:
- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.