[Close] 

Lead Data Engineer

Responsibilities

Data Ingestion

Experience in data processing operations (ETL / ELT) that is optimized and scalable, encapsulated in workflows, that transform source data, move data between multiple sources, load the processed data into an analytical data store using cloud orchestration technology such as Azure Data Factory or Apache Oozie, Sqoop, AWS Data Pipeline, AWS Glue, Informatica.

Batch Processing

Experience in the Hadoop echo system, reading source files, processing them, and writing the output to new files or data stores using Hive, Pig, or custom Map/Reduce jobs in a Hadoop cluster, or using Java, Scala, or Python programs in an Hadoop Spark cluster.

Stream Processing and Real-Time Message Ingestion

Experience with real-time data sources and message ingestion for processing by filtering, aggregating, and preparing the data for analysis. Technologies such as Spark Streaming and Kafka, AWS Kinesis, Azure IOT or Event Hub etc. Good understanding of Lambda Architecture. Experience building and productionizing micro-services and APIs.

ML and AI

Identify and formulate various challenges in user support domains as Machine Learning (ML) problems.

Design, develop, validate and deploy proposed ML solutions.

Analysis and Reporting and Analytical Data Stores

Knowledge of solutions (complex reports, dashboards, and scorecards) that provide insights into the data through analysis and reporting with Enterprise Data Lake initiatives including Data Model development, Semantic Layer Development is a plus.

Knowledge of solutions for shared data usage using cloud based data store technologies such as AWS (Redshift) or Azure (SQL Data Warehouse), SSAS or NoSQL technology such as HBase, or Hive databases in a distributed data store is a nice to have.

Working knowledge in any one of the ETL tools is a plus.

Knowledge of the reporting tools like PowerBI, Tableau, Qlik Sense, QlikView is a plus.

Other


Don't Be Fooled

The fraudster will send a check to the victim who has accepted a job. The check can be for multiple reasons such as signing bonus, supplies, etc. The victim will be instructed to deposit the check and use the money for any of these reasons and then instructed to send the remaining funds to the fraudster. The check will bounce and the victim is left responsible.

More Jobs

HBase/ Big Data Engineer with multinational in...
Princeton, NJ Marshwinds International Inc.
Experienced Field Engineer - Controls - Aerode...
Trenton, NJ FieldCore
Audiovisual Specialist Lead
Princeton, NJ Educational Testing Service
Experienced Field Engineer - Controls - Aerode...
Jersey City, NJ FieldCore
Associate Brand Lead Pipeline Strategy
Bridgewater, NJ Sanofi