Kumaran Systems

Responsibilities:

Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure’s Data Factory and data lake
Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
Developing scalable and re-usable frameworks for ingesting data sets
Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
Working with event-based/streaming technologies to ingest and process data
Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution.

Requirements:

Min. 4 years of Experience working with Azure Databricks
Experience in Data Modelling & Source System Analysis
Familiarity with PySpark.
Mastery of SQL.
Knowledge of components: Informatica, Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL.
Experience with Python programming language used for data Engineering purposes
Ability to conduct data profiling, cataloguing, and mapping for technical design and construction of technical data flows.
Experience in data visualization/exploration tools.
Providing Technical leadership and guidance to Teams
Strong problem-solving skills and the ability to translate business requirements into actionable data science solutions
Excellent communication skills, with the ability to effectively convey complex ideas to technical and non-technical stakeholders
Strong team player with excellent interpersonal and collaboration skills
Ability to manage multiple projects simultaneously and deliver high-quality results within specified timelines.

Education:

Source ⇲