Remote - Sr. Data Engineer

Senior Data Engineer

Project Location: Newark-NJ (100% Remote)

Duration: 3-6 Months

Qualification:

• Bachelor's degree in computer science, Software Engineering, MIS, or an equivalent combination of education and experience.
• Experience implementing and supporting data lakes, data warehouses, and data applications on AWS or Microsoft Azure for large enterprises (Fabric is plus)
• Programming experience with Python, Spark and SQL.
• Experience with Microsoft Azure services, specifically Data Factory, ADLS Gen2, Synapse Analytics, and Synapse Database, KQL.
• Experience in system analysis, design, development, and implementation of data ingestion pipelines in AWS or Azure.
• Solid experience implementing solutions on Azure-based data lakes (preferred).
• Knowledge of ETL/ELT processes.
• Experience with end-to-end data solutions (ingest, storage, integration, processing, access) on Azure.
• Architect and implement CI/CD strategies for EDP.
• Implement high-velocity streaming solutions.
• Migrate data from traditional relational database systems, on-prem to Azure technologies (preferably in Onelake using Fabric technologies - (preferred).
• Develop data-based APIs
• Implement POCs on any new technology or tools to be implemented on EDP and onboard for real use-cases.
• Any Cloud Certification preferred (Azure / AWS preferred)
• Knowledge of Microsoft Fabric technology focusing on cost and security considerations.

Requirements:

• 10+ years of experience as a Data Engineer/Datalakes.
• Experience developing business applications using NoSQL/SQL databases.
• Experience working with Object stores and Structured/semi-structured/unstructured data is a must.
• Experience with Microsoft Azure services Data Factory, ADLS Gen2, Synapse Analytics, and Synapse Database.
• Develop CI/CD pipeline using Terraform and GitHub.
• Microsoft Fabric is plus.

Responsibilities:

• Designing, building, and maintaining efficient, reusable, and reliable architecture and code.
• Ensure the best possible performance and quality of high-scale web applications and services.
• Participate in architecture and system design discussions.
• Independently perform hands-on development and unit testing of the applications.
• Collaborate with the development team and build individual components into complex enterprise web systems.
• Work in a team environment with product, frontend design, production operation, QE/QA, and cross-functional teams to deliver a project throughout the whole software development cycle.
• Responsible for identifying and resolving any performance issues.
• Keep up to date with new technology developments and implementations.
• Participate in code reviews to ensure standards and best practices are met.