I Spy with My Observability Eye 👁️

What is Observability and Why is it important?

Observability is a concept used in the IT context to describe the ability of (typically) DevOps teams to gain visibility into how well their software systems and applications are working by examining its outputs (i.e. without having to write new code to test them).

Although it’s only really picked up buzz and attention over the past couple of years, ‘Observability’ is a term that has been around for decades.  It originates from Control Theory, which is all about understanding self-regulating systems.  It is now increasingly relevant in the world of IT, where it is used to analyse and improve the performance of distributed systems.

In a nutshell, by utilising metrics, logs and traces, observability allows teams to gain deep visibility into these systems and identify the root case of issues, ultimately leading to better system performance.

In recent years, many companies have adopted cloud-native infrastructure services, such as AWS, which rely on microservice, serverless and container technologies.  However, conventional monitoring techniques and tools have struggled to track the many communication pathways and interdependencies in these complex distributed architectures.

Unlike Monitoring which passively tracks pre-defined metrics in systems, Observability makes actionable use of data by providing an intelligent and comprehensive view of the system and enables proactive identification and resolution of issues.

At its core, there are ‘Three Phases of Observability’:-

1. Know – teams need to know about any issues as quickly as possible, ideally BEFORE a customer is impacted. They need to know what happened and when and what systems were impacted.
2. Triage – teams need to figure out how many customers are impacted and to what degree. From there they can determine which teams need to get involved and what priority level it is.
3. Understand – teams need to understand the root cause of the problem and how many different services were impacted and which weren’t. The aim of this is to understand what needs to happen to ensure it doesn’t happen again.

Cloud Dependency Management platform creators, Metrist neatly summarises the problem that Observability solves: “The average digital business uses more than 100 third-party cloud services to power its products and run the company, and those services cause up to 70% of all software outages with an average cost of $300k per hour.  When downtime happens with services like AWS and Stripe, engineers often address incidents blindly because they don’t have the visibility to know if a cloud dependency is a contributing factor, which then results in confusion, longer mean time to resolution, and higher costs for every incident.”  Read more and find links to research findings in their 2022  press release.

As the adoption of cloud-native approaches to software development increases, and moreover multi-cloud strategies increase, Observability is paramount!  According to research from New Relic, 94% of software leaders say observability is key to developing software.  And 99% of leaders say their culture and observability technology allow developers to make quick decisions, without fear of repercussions.

But, how is Observability achieved?

Achieving observability in a system requires appropriate tools to collect telemetry data.  This can be done by building custom tools, using open-source software or purchasing a s commercial observability solution.  There are four key components in implementing observability: –

1. Instrumentation – This involves using measuring tools to collect telemetry data from various components of the system e.g. containers, services, applications, hosts and other infrastructure elements.  This enables visibility across the entire infrastructure.
2. Data Correlation – The telemetry data collected from the system is processed and correlated to create context, which enables automated or customer data curation for time-series visualisations.
3. Incident Response – Incident management and automation technologies are used to notify the appropriate teams and people about outages based on on-call schedules and technical skills.
4. AIOps – Machine learning models are utilised to aggregate, correlate and prioritise incident data automatically.  This helps filter out alert noise, detect issues that can impact the system, and accelerate incident response when necessary.

So, how do you choose the right set of Observability tools?

As this space picks up pace, the Observability tool market will be swamped, making it even harder for observability leaders to identify the right solutions for their teams.  So whether you are building your own observability tools or using open-source or commercial solutions, it is important to have a clear set of criteria to measure tools against, such as ensuring…

• it can comprehensively monitor across the network, infrastructure, servers, databases, cloud applications, and storage.  And collate, review, sample, and process telemetry data across multiple data sources
• it can integrate seamlessly with current stack and support the frameworks, languages, container platforms, and other critical software in the existing environment.
• it is easy to learn and use to ensure quick and comprehensive uptake
• it provides relevant data visualisation through dashboards, reports, and queries in real-time.  It should also separate valuable signals from the noise and provide enough context for teams to address the issues
• it can monitor data at rest from its current source—without the need to extract it—and in motion through its entire lifecycle
• it provides sufficient context for you to comprehend how your system’s performance has changed over time and its relation to other changes in the system. It should also include the scope of the issue and any dependencies on the affected service or component.
• it can support reasonable levels of growth in data volumes
• it can incorporate embedded AIOps and intelligence alongside data visualisation and analytics
• it requires the minimum possible up-front work to standardize and map data
• it provides value to your organisation by improving areas that are important to you e.g. deployment speed, system stability and customer experience

What tools are out there?

Many companies, large and small, are developing observability tools and frameworks that are gaining popularity in the industry. Some of these tools include (with links):

• Autometrics is a developer-first, open source framework that makes it easy to track the most useful metrics (e.g. error rate, response time, and production usage of any function in the source code) and actually understand the data with automatically generated queries, alerts and dashboards. Autometrics is built on OpenTelemetry and Prometheus and extends those projects to make the experience way more developer-friendly – and hopefully even fun!
• Cisco’s AppDynamics Business Observability Platform provides comprehensive visibility into every component of the infrastructure, enabling the correlation of application performance with customer experience and business outcomes. Appdynamics is compatible with various languages and frameworks, DevOps tools, cloud environments, mobile IoT, and other tools in the DataOps technology stack. It was also recognised as a “Leader in the 2021 APM Magic Quadrant” by Gartner.
• Datadog offers excellent versatility for IT operations, developers, business users, security engineering and other roles. It provides real-time visibility into cloud-scale applications, allowing IT teams to monitor and troubleshoot issues quickly. Datadog supports a wide range of integrations and provides out-of-the-box dashboards and alerts to help teams stay on top of issues.
• Elastic Observability is an open and flexible solution, powered by advanced machine learning, that “accelerates problem resolution, provides end-to-end visibility into hybrid and multi-cloud environments and unifies log, metrics and traces.” By providing insight into system performance in real-time, Elastic Observability allows users to quickly identify problems before they become costly outages.
• Fluent Bit is a “super fast, lightweight and highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments.”  It allows IT teams to collect, process, and forward logs and metrics from various sources. Fluent Bit supports a wide range of input and output plugins, making it a versatile tool for observability.
• Honeycomb is a distributed tracing platform that helps teams understand how their applications are performing in real-time. It ingests rich data from production systems and uses dynamic sampling to make in more manageable. Developers have the ability to log a substantial number of events and can subsequently determine how to segment and correlate them.
• Metrist provides engineers and IT leaders with a single platform for insights into the health of their third-party cloud dependencies, with unmatched visibility from accurate, real-time metrics. Metrist monitors cloud dependencies from multiple points of view, alerting users to issues immediately and tracking reliability over time so that engineers can resolve incidents quickly and their organisations can hold cloud vendors accountable.
• New Relic enables you to visualise, analyse and troubleshoot your software stack in one platform. It also supports auto-instrumentation for eight popular programming languages.  New Relic supports a wide range of integrations and provides out-of-the-box dashboards and alerts of your infrastructure health to provide teams with better insights for quick troubleshooting.
• Splunk Observability Cloud is a popular choice in the space and provides real-time insights and analytics to help IT teams troubleshoot and resolve issues. Splunk also supports the OpenTelemetry open standard.

There is no doubt that the area of Observability is an exciting and booming one!  It is a no-brainer for businesses looking to reduce downtime, improve system reliability, keep customers happy, keep their developer team happy and moreover achieve target business outcomes!  We look forward to seeing the developments in observability over the year ahead.