Data Engineer - Senior Software Engineer

Job description Data Engineer - Senior Software Engineer Bangalore, India Information Technology 16750 Overview We are seeking a skilled and experienced Data Engineer who has expertise in playing a vital role in supporting, data ingestion/migration, creating data pipelines, creating data marts and managing, monitoring the data using tech stack Azure, SQL, Python, PySpark, Airflow and Snowflake. Responsibilities 1. Data Ingestion/MigrationCollaborate with cross-functional teams to Ingest/migrate data from various sources to staging area. Develop and implement efficient data migration strategies, ensuring data integrity and security throughout the process. 2. Data Pipeline DevelopmentDesign, develop, and maintain robust data pipelines that extract, transform, and load (ETL) data from different sources into GCP. Implement data quality checks and ensure scalability, reliability, and performance of the pipelines. 3. Data ManagementBuild and maintain data models and schemas, ensuring optimal storage, organization, and accessibility of data. Collaborate with requirement team to understand their data requirements and provide solutions by creating data marts to meet their needs. 4. Performance OptimizationIdentify and resolve performance bottlenecks within the data pipelines and data services. Optimize queries, job configurations, and data processing techniques to improve overall system efficiency. 5. Data Governance and SecurityImplement data governance policies, access controls, and data security measures to ensure compliance with regulatory requirements and protect sensitive data. Monitor and troubleshoot data-related issues, ensuring high availability and reliability of data systems. 7. Documentation and CollaborationCreate comprehensive technical documentation, including data flow diagrams, system architecture, and standard operating procedures. Collaborate with cross-functional teams, analysts, and software engineers, to understand their requirements and provide technical expertise. Requirements Qualifications: - Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. - Proven experience as a Data Engineer with a focus on output driven. - Knowledge and hands-on experience with Azure, SQL, Python, PySpark, Airflow, Snowflake and related tools. - Proficiency in data processing and pipeline development. Solid understanding of data modeling, database design, and ETL principles. - Experience with data migration projects, including data extraction, transformation, and loading. - Familiarity with data governance, security, and compliance practices. - Good communication and interpersonal skills, with the ability to articulate technical concepts to non-technical stakeholders. Role: Database Architect / Designer Industry Type: Software Product Department: Engineering - Software & QA Employment Type: Full Time, Permanent Role Category: DBA / Data warehousing Education UG: Any Graduate PG: Any Postgraduate Key Skills Skills highlighted with ‘‘ are preferred keyskills database designdata modelingetldata processingpipeline hiveinformation technologyscalapysparksqldockerjavasparksoftware engineeringmysqlhadoopbig datamongodbsnowflakepythonairflowmicrosoft azuredata engineeringnosqlkafkaaws