Senior Specialist - Big Data Engineering

About Us:

LTIMindtree is a global technology consulting and digital solutions company that enables enterprises across industries to reimagine business models, accelerate innovation, and maximize growth by harnessing digital technologies. As a digital transformation partner to more than 700+ clients, LTIMindtree brings extensive domain and technology expertise to help drive superior competitive differentiation, customer experiences, and business outcomes in a converging world. Powered by nearly 90,000 talented and entrepreneurial professionals across more than 30 countries, LTIMindtree — a Larsen & Toubro Group company — combines the industry-acclaimed strengths of erstwhile Larsen and Toubro Infotech and Mindtree in solving the most complex business challenges and delivering transformation at scale. For more information, please visit .

Job Title <<>> Big Data- Spark Scala Tech Lead

Work Location<<>> CPH, Denmark

Job Description-

We are looking for a Spark Scala Tech Lead with 10 Years experience who knows how to fully exploit the potential of Spark cluster. You will be responsible to clean transform and analyse vast amounts of raw data from various systems using Spark to provide ready to use data to feature developers and business analysts. This involves both adhoc requests as well as data pipelines that are embedded in production environment

Responsibilities

• Create Scala Spark jobs for data transformation and aggregation

• Produce unit tests for Spark transformations and helper methods

• Write Scala doc style documentation with all code

• Design data processing pipelines

Mandatory Skills

• Should have 10 years of experience in testing and test automation with Robot framework and python

• Knowledge of data manipulation using various data formats XML CSV JSON plain text etc and experience with SQL queries preparation

• Experience of Designing Editing effective reusable automated tests

• Collaborating with QA Analysts and Software Developers to develop test automation solutions

• Designing and developing test automation scripts

• Contribute in defining the overall automated test strategy approach for team and following test automation guidelines strategy

• Analyse and comment requirements and technical designs from a test perspective

• Designing implementing and executing test cases by automated test methods

• Gathering and managing test data Monitoring and documenting test results Report debug and verify defects

• Collaboration with the team to prepare setup monitor CICD delivery pipelines

• Good Knowledge on Tools like JIRAALMBIT BucketGIT

• Excellent communication and teamwork skills Great attention to detail

• Coordinates with various functions within the company and client organization and ensure timely delivery Own the deliverable task

What you’ll be doing

• Building distributed and highly parallelized big data processing pipeline which process massive amount of data both structured and unstructured data in near real-time

• Leveraging Spark to enrich and transform corporate data to enable searching data visualization and advanced analytics

• Working closely with analysts and business stakeholders to develop analytics models

• Continuous delivering on Hadoop and other Big Data Platforms

• Automating processes where possible and are repeatable and reliable

• Working closely with QA team

As an expert Data Engineer, you are expected to have

• Excellent Programming language skills Spark Scala BashUnix Scripting SQL Bash

• Excellent understanding of Hadoop and HDFS Architecture

• Excellent understanding of file types their pros and cons

• Excellent understanding of Build Tools especially SBTGradle

• Very good understanding and good experience in implementing CICD eg Building and Maintaining Jenkins Pipelines

• Working experience in SAFeAgile

• As an expert you are also responsible for Juniors writing maintainable code

• Are collaborative and achieve your expectations through communication and teamwork

• Are curious responsive and can understand the needs of others to ensure delivery of the desired results

• Work qualitatively and strive to always do things better

Skills

• Spark Scala Scala with a focus on the programming

• Apache Spark 2x

• Spark query tuning and performance optimization

• Understanding on Hadoop Architecture

• Experience working with Hive SQL and HDFS

• Deep understanding of distributed systems eg CAP theorem partitioning replication consistency and consensus

• Good to have experience in Scala

• Experience working with tools like Jenkins Jira Bitbucket and Git

• Experience in writing shell scripts and working with Linux platforms

• As an advantage familiarity with one or more of the below technologies

• Experience working in AgileSAFe

• Experience certification with AWS Cloud

• ETL Testing