Observability Platform Support Lead

Who we are looking for
The ideal candidate for this role (Observability - Logs & Metrics) is responsible for working closely with the Application and infrastructure teams to ensure that all aspects of the telemetry from applications, business events, appliances and infrastructure are accurately received, tagged, and reported. This role involves leading a global team to maintain observability platform, and ensure it is optimized and operating within SLI and SLO's, ensure proper checks and balances are implemented before new instrumentation is rolled into production. The candidate must be a self-motivated individual with cross functional knowledge in software development, monitoring and maintenance.
Observability engineering team is responsible for implementation of Observability stack for monitoring the enterprise infrastructure footprint of StateStreet. Ideal candidate for this role (Observability Engineer - Logs & Metrics) is responsible for working closely with the Application, Infrastructure and Cyber Security teams to ensure that all aspects of the production infrastructure are accurately monitored and reported (alerting etc...). This role also involves writing custom collectors in Python and Go. The candidate must be a self-motivated individual with cross functional knowledge in software development, monitoring and maintenance

What you will be responsible for
As As Observability Platform Support you will:

• Ensure the observability platform is optimized and operating within SLI and SLO's
• Configures monitoring/alerting/reporting for related to logs and metrics using tools like Splunk, Elasticsearch, workbench etc.
• Performs installation and upgrade of Splunk, and Splunk forwarders
• Works closely with application development teams during release rollout to analyze agent performance impact and troubleshoot issues
• Guide App & QA teams to derive Performance Test Cases and perform Root Cause Analysis during Performance tests
• Makes technical recommendations on monitoring improvements (by creates Technical PoCs to demonstrate performance improvements), and partner with Engineering team on new standards for logs, business events, and tags
• Analyzes Logs and relate it with metrics to identify root-cause of applications and provide tuning recommendations
• Work closely with delivery and operations specialists to ensure proper tuning and operation of the systems/applications
• Collaborate with development and support teams to resolve performance related issues

What we value

• These skills will help you succeed in this role
• Proficient in Log aggregation tools such as Splunk, Elasticsearch, Logstash etc. is required
• Must have working experience in at least in one of the following : ITSI, Splunk dashboards or add-ons, Kibana dashboards or Grafana
• Proven understanding of web technologies and distributed application architecture is required
• Proven understanding of full life cycle software development methodologies is required
• Proven ability to troubleshoot complex application logic flow is required
• Application Server experience with Tomcat, JBOSS required

Education & Preferred Qualifications

• Bachelor's degree
• 8+ years of App/Infra monitoring, support and application performance management & tuning is required
• 5+ years experience in supporting Splunk platform support with global team for enterprise logging, and prior experience of application support of Java/J2EE/Spring technologies is required
• 3+ experience in Oracle, Microsoft SQL Server, and Python Scripting is required