SUMMARY:
As a Data Engineer, you will be responsible for designing, developing, maintaining, and optimising a data pipeline infrastructure using a proprietary data platform , which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.
WHAT WILL YOU BE DOING:
-
Data Engineering Ownership: End-to-end creation and upkeep of robust data workflows and components, leveraging tools such as Databricks. This includes designing early-stage prototypes and deploying large-scale data acquisition, handling, and storage strategies.
ETL/ELT Workflow Management: Construct and refine data ingestion pipelines to seamlessly integrate varied data sources into the central platform. Ensure data consistency through validation, cleaning, and enrichment routines.
Stakeholder Collaboration: Partner with commercial teams and data stakeholders to gather and refine specifications for analytical products and visual reporting needs.
Architectural Design & Data Modelling: Engage with analysts and data experts to define efficient data models and architectural plans. Support dashboard and report development, while ensuring optimal data structuring for performance.
Pipeline Performance & Reliability: Continuously assess and fine-tune data workflows, addressing system inefficiencies, integration problems, and data fidelity concerns.
Quality Assurance in Data: Define expectations for data integrity and collaborate with QA to automate validation checks. Utilise monitoring tools to surface and track data quality metrics.
Data Controls & Compliance: Apply internal governance standards and safeguard sensitive data through access rules, encryption protocols, and retention strategies, aligning with organisational security frameworks.
Technical Enablement & Knowledge Sharing: Work closely with interdisciplinary teams to ensure data needs are met, while thoroughly documenting processes and solutions. Convey technical ideas in a way that’s accessible across departments.
Innovation & Process Evolution: Keep informed on developments within the data ecosystem. Advocate for and implement improvements in efficiency, tooling, and automation. Participate in internal knowledge communities.
Agile Delivery Engagement: Contribute to delivery efforts by actively participating in sprint planning, stand-ups, and backlog refinement. Help define and deliver technical tasks, aligning work with CI/CD best practices.
WHAT WE ARE LOOKING FOR:
- Approximately 4 years of experience in data engineering for mid-level roles
- 6 or more years of experience for senior-level data engineering positions
- Between 1 to 4 years of experience building data products using platforms such as Databricks, Spark, Cloudera, or HortonWorks
- Skilled in Python (especially PySpark), Scala, and SQL
- Experienced in designing and implementing scalable data pipelines for high-volume environments
- Solid understanding of ELT/ETL practices and data integration strategies
- Capable of writing robust production code with automated testing
- Familiar with CI/CD tools such as GitHub Actions and Jenkins for deploying code
- Hands-on experience with distributed data processing using Apache Spark
- Proficient with cloud services including AWS, Azure, or Google Cloud and tools like S3, Glue, Lambda, Redshift, and BigQuery
- Good knowledge of data modelling, relational databases, and SQL performance optimisation
- Strong analytical and problem-solving abilities, with attention to troubleshooting details
- Effective communicator with experience working in collaborative, cross-functional teams
- Basic understanding of machine learning concepts such as classification, regression, A/B testing, and experimental design
AWESOME BUT NOT REQUIRED:
- Understanding of the UK media landscape, including over-the-top (OTT) and traditional broadcast advertising
- Familiarity with concepts and trends in digital advertising and marketing analytics
- Experience applying statistical approaches such as regression and classification, as well as designing and analysing A/B and other controlled experiments
- Awareness of modern data architecture methodologies including Data Mesh, enterprise-level data frameworks, and business intelligence architecture
- Solid grasp of data protection practices, regulatory compliance, and governance principles
- Exposure to a variety of data management domains such as metadata handling, data quality enforcement, master data systems, and governance tooling
- Hands-on experience with data visualisation platforms like Tableau, Looker, AWS QuickSight, and ThoughtSpot for building interactive reporting solutions
WHAT’S IN IT FOR YOU?
- Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture
- A culture in which we don’t criticise failure but ensure we learn from our mistakes
- An Agile environment where your ideas are welcome
- The possibility to grow and experience different projects
- Ongoing Training & Mentoring
- The possibility to travel
- ATTENTION! THIS POSITION IS FOR PORTUGAL OR BRAZIL BASED ONLY