Data Engineer I Big Data Ecosystems Multinational Retailer Services Company Years
Key goals for the position:
* Develop technical solutions for a data platform based on existing and emerging patterns and technologies.
* Enable advanced analytics including AI/ML and data-driven decision making by building pipelines that are event driven, can transform/ cleanse/ manipulate data and facilitate visualization of data integrated from enterprise systems, internet, and many other disparate sources.
* Enable data exchange amongst enterprise systems via real time event streaming platform.
* Champion conceptual and technical aspects of real time data platform development and support, using various design patterns, dynamically scalable technologies, and Devops practices.
Team Member Minimum Requirements
Preferred Formal Education/Qualifications:
* Bachelor of Computer Science
* Other qualifications in IT or related discipline Proven experience:
* Developing, Testing and Supporting data lakes, data warehouses and large-scale data processing systems.
* Working effectively in data platform scrum teams for technical design, development and deployment of solutions.
* Implementing and supporting data platform technologies both on-premise and cloud-based.
* Deep understanding and adoption of Agile delivery techniques, including Continuous Integration and Continuous Delivery (CI/CD).
* Developing solutions using a variety of technologies and tools to marry on-premise and cloud-based systems together.
* Participate in technology/tools/framework evaluation to recommend/influence adoption.
* Follow best practices to ensure high standards of data availability, reliability, completeness, efficiency and quality. Skills/Knowledge/Abilities/Technology Used:
* Advanced SQL working knowledge and experience working with a variety of relational databases, SQL query authoring.
* Expert understanding and hands-on working knowledge of message queueing, real time event streaming architecture for data platforms including Kafka, Kinesis, SNS, SQS etc.
* Deep understanding and ability to build automations for data transformation, processing of data structures, metadata, dependencies and workload management for very high volume, velocity and variety of data.
* Deep working knowledge of AWS technologies such as S3, EC2, EMR or Bigdata/Hadoop, RDS, Lambda, Elasticsearch, Redshift, Cloudformation, Terraform, Cloudwatch etc.
* Sound to advanced level working knowledge of Git and CI/CD pipeline technologies such as Jenkins, Chef, Kubernetes and Docker containers.
* Deep experience with object-oriented/object function scripting languages: Python, Scala, Java, R, C ++, Golang etc.
* Working knowledge of Data warehouses, NoSQL databases and ETL technologies like Informatica, Talend etc.
* Understanding of modern Saas based monitoring and logging tools like New Relic/Sumologic or equivalent.
* Must have experience with Linux, shell scripting.
* Familiarity with BI tools a plus
Area of Accountability Key Responsibilities and Deliverables Performance Measures and Targets.
Development and Operations:
* Collaborate with team members to design and implement data solutions in alignment with the project schedule.
* Code, test, and document new or modified data systems to create robust and scalable applications for data analytics.
* Peer and customer feedback
* Line manager observations
* Line manager observations
* Create data flow diagrams for business systems.
* Implement security as part and parcel of all development.
* Builds automation tools to provide a self-service data platform and enable CICD.
* Creates and maintains a data catalogue.
* Develops standards and processes.
* Champions change management
* Take Care Safety * Accountable for a safe site for everyone, every day by implementing and evaluating safe work practices, improving safety performance and.
Celebrating safety achievements
* Follows a Devops mindset and practice
Communication:
* Conducts product demonstrations, showcases, briefings on trending technologies, processes and solutions relating to the data platform.
* Line manager observations
* Line manager observations
* Communicates effectively with stakeholders, partners, vendors.
* Peer and customer feedback
Other Areas of Accountability Key Responsibilities/Major Activities:
Values and Behaviour
* Live Integrity
* Think Customer
* Grow Together
* Reach Higher