About the Role

We are hiring a hands-on Data Platform Monitoring & Observability Engineer to design, build, and maintain end-to-end monitoring solutions for a large-scale enterprise data platform on Microsoft Azure, delivered as a long-term managed service for an international client in Europe.

The role focuses on centralized monitoring, automated alerting, data pipeline observability, incident detection, and platform reliability across cloud infrastructure, databases, streaming systems, and enterprise applications.

We are looking for an engineer with strong technical capabilities who can independently develop monitoring dashboards, automate metric collection, and implement proactive monitoring solutions rather than solely performing routine operational monitoring.

Key Responsibilities

• Monitoring & Observability: Design, develop, and maintain centralized dashboards using Grafana, Splunk, Azure Monitor, or equivalent tools. Implement automated metric collection, logging, alerting, anomaly detection, and end-to-end system health monitoring.

• Data Pipeline Monitoring: Build monitoring solutions for batch, streaming, and event-driven pipelines using Apache NiFi, Apache Airflow, dbt, Kafka, Azure Event Hubs, Azure Service Bus, and Apache Flink. Track pipeline success rates, latency, throughput, processing failures, and resource utilization.

• Cloud & Database Monitoring: Monitor the availability, health, and performance of Microsoft Azure infrastructure, Snowflake, PostgreSQL, and data lakes. Track compute, storage, networking, system scalability, capacity, and service recovery capabilities.

• API & Application Monitoring: Implement monitoring for REST APIs, Azure API Management (APIM), microservices, and downstream applications. Track API availability, response times, error rates, service dependencies, and overall application performance.

• Data Quality & Reliability: Monitor data completeness, freshness, timeliness, and validation results. Identify data processing failures, performance bottlenecks, and upstream/downstream dependencies across data pipelines.

• Incident Management: Develop automated alerts and escalation workflows, integrate with IT Service Management tools such as ServiceNow, investigate technical incidents, perform root cause analysis, and monitor SLA/KPI compliance.

• Automation & Continuous Improvement: Develop custom monitoring scripts and integrations using Python, SQL, and REST APIs. Automate routine health checks, metric extraction, alert configuration, and operational reporting. Support CI/CD and infrastructure-as-code practices using Azure DevOps and Terraform.

• Operational Documentation & Support: Maintain monitoring configurations, operational dashboards, troubleshooting documentation, and incident response runbooks. Collaborate with Data Engineering, Platform Engineering, and DevOps teams to improve system reliability and operational efficiency.

Requirements

• Bachelor's degree in Computer Science, Software Engineering, Information Technology, Data Engineering, or a related field.

• Minimum 3 years of hands-on experience in Monitoring Engineering, Data Platform Operations, DevOps, Site Reliability Engineering (SRE), or related technical roles.

• Practical experience with Grafana, Splunk, Azure Monitor, or similar enterprise observability tools, including dashboard development, logging, and alert configuration.

• Proficiency in Python and SQL for automation, monitoring integrations, metric collection, and troubleshooting.

• Experience working with REST APIs, JSON, and system integration technologies.

• Good understanding of ETL/ELT processes, data ingestion, workflow orchestration, and batch or real-time data processing.

• Familiarity with data engineering technologies such as Apache Airflow, Apache NiFi, Kafka, Azure Event Hubs, Apache Flink, or equivalent platforms.

• Working knowledge of Microsoft Azure or other major cloud platforms, including infrastructure monitoring, scalability, and high availability.

• Strong understanding of system performance, incident troubleshooting, root cause analysis, and operational monitoring best practices.

• Good analytical, problem-solving, communication, and technical documentation skills.

• Fluent English, with flexibility to adjust working hours to accommodate collaboration with European-based clients and teams, including meetings during Central European business hours and agreed support during critical operational periods.

Preferred Qualifications

• Experience with Snowflake, PostgreSQL, data lakes, or other enterprise data platforms.

• Familiarity with Astronomer, dbt, Kafka Connect, Azure Service Bus, or Azure Functions.

• Knowledge of Azure API Management, PostgREST, Power BI monitoring, and microservices architectures.

• Experience with ServiceNow, ITIL processes, incident management, and SLA/KPI reporting.

• Familiarity with Azure DevOps, Terraform, CI/CD pipelines, and infrastructure as code.

• Knowledge of data quality management, data modelling, metadata management, or data lineage tools such as Atlan.

• Understanding of cloud cost monitoring, resource optimization, security logging, audit trails, and access control.

• Exposure to high-availability systems, mission-critical enterprise applications, or real-time event processing.

Tech Stack

• Monitoring & Observability: Grafana, Splunk, Azure Monitor, logging, alerting, dashboards, metric collection, and performance monitoring.

• Data Ingestion & Orchestration: Apache NiFi, Apache Airflow, Astronomer, dbt, Azure Functions, REST APIs.

• Streaming & Event Processing: Apache Kafka, Kafka Connect, Azure Event Hubs, Azure Service Bus, Apache Flink.

• Cloud & Data Platforms: Microsoft Azure, Snowflake, PostgreSQL, Data Lakes.

• API & Analytics: Azure API Management (APIM), PostgREST, Power BI.

• Automation & DevOps: Python, SQL, Azure DevOps, Terraform, CI/CD, REST API integrations.

• Incident & Service Management: ServiceNow, ITIL, SLA/KPI monitoring, root cause analysis, and operational reporting.

Working Arrangements

This role involves working closely with an international client based in Europe. Candidates should be comfortable with flexible working hours to support cross-time-zone collaboration, technical discussions, and operational requirements.

Occasional extended hours or on-call support may be required during critical incidents or major business events, subject to agreed working arrangements.

Sumber: JobStreet. Sila rujuk iklan asal untuk maklumat terkini.