Looking for a solution that gives a global overview of all current shipments while providing customers with real-time last-mile tracking and tracing. A large chemical logistics company turned to BBD to develop and integrate a custom platform. The resultant solution leverages platform and data engineering at its finest. Seamlessly slotting into the existing technical environment while supporting complex data transformation in a secure and reliable manner.
Objectives
- Allow for a global overview of shipments by implementing a track and trace solution covering various modes of transport. Including ocean, rail and road
- Build a platform and data engineering pipeline on the AWS cloud to support the solution
- Integrate with a third-party track and trace solution while also creating a data product for other systems to consume and display
Benefits
- Solution allows for last-mile tracking and tracing across all shipments, providing customers with arrival estimates and notifications
- Reduces customer service agents’ response time on queries regarding specific shipments
- Reduces unnecessary contact with customer service agents, allowing them to focus on critical queries
- Technical benefits:
▪ Fine-grained access control to data across teams
▪ High availability within each region as required by client business
▪ Using an IaC module and CI templates, the platform supports deployments of complex data transformation logic in a secure and reliable manner. With little effort from engineers
▪ Allows data engineers to create complex orchestration logic for data pipelines
▪ CIA classification and data privacy assessment governs security requirements throughout the platform
▪ Allows reuse of code between batch and streaming pipelines, easily extending to support streaming as business processes evolve
▪ Integration of native AWS services with client’s observability platform allows for complete observability of solution
Approach and project overview
BBD’s approach to this project was to leverage the expertise of our platform, data engineering and cloud teams alongside native development skills. In other words to design, implement and deploy a solution that met the client’s objectives. The goal was to deliver a platform where all team members could have sight of the data at all stages to validate correctness and ensure continuous business value.
The initial step was to invest up-front in delivering a minimum viable product (MVP) platform which would enable the data engineers to move quickly. From there, an iterative approach was taken to constantly improve on the MVP with feedback from the data engineers as they used it. There was additionally a small subset of the BBD team that focussed on leveraging managed services to increase the speed of delivery while reducing operational overhead. Leveraging managed services in this way allowed for the BBD team to stay lean while the project reaped the benefits of the services.
An added benefit of BBD’s approach was that our platform team focussed not only on developing actual platforms for engineering teams to work on, but also on developer advocacy and DevOps. In this way, the team could influence how the client teams operate – ensuring the adoption of the required mindset change
Technology
Alongside the core cloud competencies, data engineering and platform expertise, the technology used in the project included Terraform for IaC (Infrastructure as Code), Pyspark, Gitlab CI, and AWS services including Glue, MWAA, S3, LakeFormation, Athena and supporting services such as IAM, KMS, and Secrets Manager.
- Athena – data discovery purposes. Gives analysts direct access to query data using SQL throughout all stages of the data pipeline
- LakeFormation – fine-grained access control to data
- MWAA – AWS managed version of Apache Airflow. Orchestrates various pieces of the data pipeline
with manual and automated triggers, backfilling, and fault tolerance - BDD testing – prevents regression. Ensures data transformations meet requirements designed by
business - Standardised yet extensible CI templates to enable engineers to build and deploy code in a repeatable manner without needing the understand underlying platform
From a methodology point of view throughout the project, consuming data was extracted from diverse ERP systems through a change data capture solution. The business process currently involves asynchronous events triggered over large windows of time which made streaming an unsuitable solution. As such, a batch processing approach was followed. However, the platform can easily be extended to support streaming as the code is reusable across both types of pipelines. Lastly, within the data aspect of the project, medallion architecture was used to create a clear separation between the various levels of data structure and quality as it flows through each step in the pipeline.
Impact of BBD’s partnership
The bespoke platform and data engineering pipelines developed for this large global chemical logistics company have revolutionised their business. The solution allows for not only a holistic view of all shipments across different transport modes but has additionally improved their customers’ experience.Providing last-mile tracking and tracing including arrival time estimates and notifications.
