Data engineer – streaming
BBD is looking for a data engineer who is committed o harnessing the power of data and cloud technologies to drive innovation and informed decision-making
We are seeking an experienced AWS engineer with expertise in AWS Elastic and Python to join our collaborative team. As part of our team, you will play a vital role in building a data lake on the AWS platform, facilitating the seamless ingestion of real-time and local market data feeds into Elasticsearch, and enabling the creation of dynamic dashboards using Kibana.
The company:
BBD is an international software firm that solves real-world problems with innovative solutions and modern technology stacks. Our experience spans the education, financial services, gaming, government, insurance and telecoms sectors. We maintain our track record by using our vast business domain knowledge and world-class skills to successfully deliver digital solutions for clients.
The complex problems we solve are balanced out by our flexible working culture and flat management structure. Being a part of BBD means working in dynamic project teams, while pursuing your own career growth through our Continuous Learning Programme.
The role:
As a data engineer with experience in AWS and a focused on data lake construction and Elasticsearch integration, you will be at the forefront of our data-driven initiatives. Your responsibilities will encompass working with a range of AWS services, including Lambda, S3, CodePipeline, and CodeCommit, as well as Elasticsearch, Kibana, Logstash, Beats, Elastic Common Schema (ECS), and Elastic Security. Your role will involve deploying, configuring, testing, and troubleshooting AWS services, ensuring efficient data indexing, querying, aggregations, and mappings. You will also serve as the subject matter expert for ELK (Elasticsearch, Logstash, Kibana) implementation across our shared service platform.
Responsibilities:
- Collaborate with cross-functional teams to construct and manage a data lake on AWS for real-time and local market data feeds
- Implement and maintain Elasticsearch, Logstash, Beats, Kibana, Elastic Common Schema (ECS) and Elastic Security components
- Utilise AWS services such as Lambdas, S3, CodePipeline, and CodeCommit to automate data lake processes
- Develop and maintain data ingestion pipelines using Apache Flink and Apache Beam
- Deploy, configure, and optimise AWS services to support data storage, processing, and analysis
- Index data, design complex queries, create aggregations, and manage mappings within Elasticsearch
- Serve as the go-to expert for ELK implementation and configuration
- Collaborate with teams to integrate Elasticsearch with other operational data platforms and tools, including Kafka, SIEM, and more
Requirements:
- Bachelor’s degree in Computer Science, Information Technology, or a related field
- Extensive experience as a developer with a strong focus on AWS Elastic services and Python
- In-depth knowledge of Elasticsearch, Logstash, Kibana, Beats, Elastic Common Schema (ECS) and Elastic Security
- Familiarity with AWS services, including Lambdas, S3, CodePipeline, and CodeCommit
- Proven expertise in data pipeline development using Apache Flink and Apache Beam
- Proficiency in Python for scripting and automation
- Background in cyber security or data engineering highly advantageous
- Strong problem-solving and troubleshooting skills
- Excellent communication and collaboration abilities
- Ability to excel in a fast-paced, team-oriented environment
If you are an experienced data engineer with a strong background in AWS Elastic, Python, and a passion for data-driven projects, we encourage you to apply.
BBD is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, family, gender identity or expression, genetic information, marital status, political affiliation, race, religion, or any other characteristic protected by applicable laws, regulations or ordinances.
