💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This blog took 3 days and 12 hours to be curated, so make sure to show some love!

A few weeks ago, Leo Blondel stopped opening his SigNoz dashboards.

Not because they were bad, they weren’t. He’d set them up himself, spent a weekend tuning them, and he says he liked them, but he just stopped needing to look. “I have dashboards because I set them up at the beginning,” he told us. “I haven’t opened them except for screenshots for my ISO certification.” This is not a story about dashboards being obsolete; plenty of teams will still want them. It’s a story about what happens when something else starts reading your telemetry patiently, in parallel, while you’re trying to get your fourteen-month-old back to sleep, as Leo puts it.

For a long time, observability tools have been built for one user: a human at a screen, scrolling logs, hovering over a flame graph, deciding what to do next. But there is now a second user in the building, and they are strange. They no longer read dashboards; they read APIs and JSON instead, open a dozen context windows at once and most importantly, they never get tired.

They are, in the way that matters for an SRE on call, infinitely patient.

What follows is a look at what one engineer built for that second user, why it worked, and the parts that still belong to the first.

What Leo built

As CTO of a small startup, Leo is one of just three engineers on the team. Their product is the kind of microservice-heavy data plumbing that demands real observability and absolutely cannot justify a dedicated on-call rotation. He started, like a lot of people do, by trying Datadog. The two-week trial ended with a quote north of two thousand dollars.

Their AI SRE feature (Bits AI SRE), which lets you click a button on an error to have an agent investigate, was quoted at roughly $40 per investigation.

“You wouldn’t do it, right?” he said. “So instead, what that inspired me to do was: wait, I can do this.”

He moved to SigNoz, open source, openTelemetry-native, with a hosted MCP server that he could plug into Claude. Setup took, by his account, an afternoon, and then, instead of paying per-click for someone else’s investigation agent, he built his own and let it run.

The architecture is unromantic and worth describing in detail, because the details are where the difference lives:

• Alerts originate in SigNoz. Rules are tuned by humans, when a rule fires, a webhook hits the agent.

• The agent is Claude Opus, running on a small VM, kept alive by PM2.

• It reads telemetry through the SigNoz MCP server, including logs, traces, metrics, the same data a human would query, surfaced through natural-language tool calls instead of a query bar.

• It has read-only access to Kubernetes through an RBAC-scoped service account. It can describe a pod, look at events, inspect an Argo CD application but cannot change anything.

• It has read access to GitLab. When something breaks, the first question it asks is whether anyone merged anything in the last hour.

• It talks to humans through Slack in socket mode, not just a one-way notification firehose but an actual conversation.

• It writes to a Postgres database as its working memory and diligently records every alert it has seen, every investigation it has done and every conclusion it has reached.

• It writes incident reports to Notion when something is resolved, so the post-mortem isn’t a task someone has to do later. It already exists.

• It restarts itself every day at 4 a.m. so its context window doesn’t fill up, reading a summary of its own memory on the way back up.

One of the first things in Leo’s playbook for the agent is, in his words, a warning to itself: “Your context window is your lifeline. If it fills up, you die and must restart.” The agent is instructed to never investigate directly, but to spawn sub-agents with fresh context windows, smaller and cheaper models that do the dirty work and report a single paragraph back. The main agent stays clean, and the sub-agents remain disposable.

What it actually does

When a noisy alert fires, say, a node briefly reporting memory pressure that will be fine in ten minutes, the agent receives the webhook, queries SigNoz, decides it’s transient, and logs the event to its memory.

When the same alert fires for the eleventh time that week, the agent’s memory tells it this is no longer noise and is more of a pattern. So it pings Leo in Slack and explains itself: here is the alert, here is how often I’ve seen it, here is what I think is going on, here is the merge request from yesterday that I suspect is the cause. And Leo is free to reply after checking it out on his own, based on his availability. At the end of the day, the agent files what amounts to an end-of-shift report, including what they saw.

“It’s almost like I have a full-time person whose job is making sure my stack is always green”, Leo said.

It has been live for about three weeks, and in this time, by his count, it has produced zero false positives. It has had one false negative, which is one real thing it categorised as noise. He has been quietly extending it. Last week, he added Falco for security events, wrote a new rule into the agent’s playbook, and watched it pick up the alert and start reasoning about it.

Context isn’t care

The temptation, reading this, is to assume the agent is doing the human’s job; it isn’t. It’s doing a job that didn’t really exist before, or rather, that existed as the worst part of a different job, the part most folks hated.

There is a useful distinction here, one that SigNoz’s own framing of agent-native observability gets right: agents bring context, humans bring care.

An agent connected to your telemetry has more context than any human ever will. It has read every trace in the last hour, scanned every log line, and can correlate ten metrics across five services in the time it takes you to find the right tab. The classic complaint of observability that I don’t have time to look at all of this is, for the first time, not a problem anymore.

But we have to realise that context isn’t the same as care. The agent does not know which 500 actually matters. It does not know that one customer is on a renewal call this afternoon, or that the checkout service degrading by 80ms is a Series A milestone, or that the noisy memory alert on the staging cluster genuinely doesn’t matter because that cluster is being torn down on Friday. You can tell it some of these things, but you cannot tell it all of them, and the list keeps changing.

This is why every team we’ve watched try to hand alert judgment entirely to an agent has ended up in the same place. The agent produces noisy pages, and the team learns to ignore the pages, flagging them as a waste of time and effort. Within two weeks, the experiment will be uninstalled.

Leo’s setup doesn’t fall into that trap, since he kept the rules, including the actual alert thresholds, under human control and written deterministically in SigNoz. The agent’s job is downstream of that: triage what fires, investigate what looks real, escalate what matters. Human up top, deterministic system in the middle, agent at the bottom where the grunt work lives. None of those layers is doing the job of the one above it.

Or, the way he put it: “I don’t want the AI to replace something. I want the AI to augment.”

What this asks of an observability platform

If you take the second-user idea seriously that AI agents are going to be reading your telemetry the way humans read dashboards, start being product decisions.

The data model has to be unified. When a human is investigating, they can paper over schema mismatches between logs, traces, and metrics by squinting, but an agent can’t. If your logs live in one schema and your traces in another and the correlation has to be reconstructed by hand each time, your agent will spend most of its tokens figuring out where it is instead of figuring out what’s wrong. This is one of the reasons SigNoz built on OpenTelemetry from day one and kept everything in a single store. Although it wasn’t an agent-era decision, it just turned out to be one.

The interface has to speak to agents directly. Dashboards are a human-facing rendering of the underlying data, and agents don’t need that. They need the underlying data, exposed through something they can actually call, which, today, is synonymous with MCP. SigNoz’s hosted MCP server is live for Cloud users, and the self-hosted version is on GitHub. Leo’s whole setup runs through it.

The platform has to be legible to a model. Models have been trained on years of OpenTelemetry tutorials, SigNoz documentation, GitHub issues, and Stack Overflow answers. They already know how to talk to an open, well-documented stack. They have a much harder time with proprietary query languages and gated docs, where the model has to reason from forum posts and Reddit threads.

Powered with skills. A raw MCP server is a bag of capabilities. The next layer up is teaching agents the conventions of your team, which metrics matter, which tracing patterns hold up at scale, and what your debugging tribal knowledge actually looks like (more and more context). Your agent eventually actually works exactly as per your runbook.

Two users, one platform

The shape of agent-native observability, at least for the foreseeable future, is two users sharing the same underlying data and asking different things of it.

The human user wants a clear view of what’s happening, the ability to ask hard questions, and confidence that the system will tell them when something they care about is wrong. The agent user wants raw data, fast tool calls, clean schemas, and enough structure to reason without getting lost. The platform that serves both well is the one that stops treating dashboards as the product and starts treating the underlying telemetry as the product with multiple ways to consume it.

Leo’s setup is a small early example of what this looks like in practice. The rules live in SigNoz, configured by him. The agent picks up the work that scales poorly with human attention, like reading everything, correlating everything, remembering everything and gives back the work that scales well: deciding what matters, choosing what to monitor, knowing when something is actually wrong.

Both users belong on the platform now. We’re building for both.

If you want to plug your own agent into your telemetry, the SigNoz MCP server is live for Cloud users and open source on GitHub for everyone else. The longer picture, including the AI Assistant beta and the skills work coming next, lives at Agent Native Observability in SigNoz.

Until today, when I decided to address the 128-bit elephant in the room.

So, like a normal human these days, I searched on Google, “Why is a trace ID 128 bits long?”

And the answer was, surprisingly, a long one.

The answer touches probability theory, distributed systems constraints and fifteen years of industry migration. Let’s actually dig in.

What is a trace ID for, anyway?

When a request enters your system, say, a user clicks checkout, it might bounce through 20 different services, from your API gateway to auth to cart to inventory, etc. Each service does some work and may call other services. To reconstruct what happened when something goes wrong, you need a way to say “all these log entries and spans belong to the same original request.”

That’s the job of a trace ID. It’s generated once, at the entry point, and propagated through every downstream call via HTTP headers like traceparent(or other means of propagation).

So the trace ID has one job, uniquely identify one request’s journey through the system.

Back in our school days, to uniquely identify the students of a class, we had a system in-place which was incremental roll numbers. Why can’t we adopt something similar here, perhaps a counter system?

Why not just use a counter?

In short, counters need coordination. If Service A and Service B both want to generate trace IDs, they’d need to ask a central server to generate the next number and to remember the previously generated number.

To prevent this overhead, trace IDs are generated randomly, independently, with no coordination. Every service just picks a random number and trusts that it won’t collide with anyone else’s.

This is the entire reason the size matters: with no coordination, collision avoidance is purely a function of how large your random number is.

Now the dilemma is deciding how large this random number must be to effectively prevent collisions.

My first instinct was to think of just making the number big enough so that collisions are impossible. Let’s take 64 bits for now. 64 bits gives you 2⁶⁴; roughly 1.8 × 10¹⁹ possible values. That feels astronomical, and surely picking random numbers from a pool that large means collisions should be basically impossible.

The Birthday Paradox

This is where our intuition fails. Let me explain this with the birthday paradox.

Imagine a classroom of 23 students. What are the odds that two of them share a birthday? Instead of computing the probability that two people share a birthday, it’s easier to compute the probability that no two people share a birthday and then subtract it from 1.

            P(no match)=365/365 × 364/365 × 363/365 × ⋯ × 343/365 ≈ 0.493

So the probability that at least two students share a birthday is:

                                P(match)=1−0.493 ≈ 0.507

Just over 50% with 23 people. This is the birthday paradox.

If we extend this paradox to trace-ids, we realise we are asking whether any two IDs in our entire trace history collide. Let’s understand this mathematically in the context of trace-ids.

The Math Behind The Paradox

Imagine you’ve generated 4 IDs: A, B, C, and D. A collision occurs when any two of them are equal. So let’s first list out all the possible pairs,

• A & B

• A & C

• A & D

• B & C

• B & D

• C & D

That’s 6 pairs. Each pair is one opportunity for a collision.

The formula for counting pairs is,

                           number of pairs = k(k−1)/ 2

where k is the number of IDs. For large k, the difference between k and k-1 barely matters, so we can simplify as,

                            number of pairs ≈ k^2 / 2

We observe that the key term (k^2) is quadratic, implying that if we generate 10× more IDs, you get 100× more pairs.

Counting expected collisions

Now we need to turn that pair count into something meaningful. Here’s the cleanest way to think about it:

Picture a dartboard with N spots on it. You throw k darts blindfolded and each dart lands on a random spot. A collision is when two darts hit the same spot.

• The number of dart-pairs is k²/2

• The chance that any single pair hits the same spot is 1/N

• So the expected number of collisions is pairs × per-pair-chance = k²/2N

The Formula for Probability

Combining all those independent chances with a standard probability trick using the approximation 1 - x ≈ e^(-x) for small x, we land on the famous birthday-paradox formula, where k is the number of IDs, and N is the size (2^number of bits),

                          P(collision) ≈ 1−e^(-k²/2N)

The important part of the formula is what’s inside the exponent: k²/2N.

Reading the formula in Plain English

The formula tells you a story in three acts, depending on how big k²/2N is. Here’s the graph for e^x, which makes understanding the acts easier.

Act 1: Safe.

When k²/2N is much less than 1, you’ve generated far fewer IDs, the exponent is near zero, e^(- near zero) ≈ 1, and collision probability is essentially zero. You’re fine.

Act 2: Danger.

When k²/2N is around 1, collision probability jumps to about 63%. You’re now more likely than not to have a collision somewhere in your set.

Act 3: Inevitable.

When k²/2N is much greater than 1, the exponent becomes huge, e^(-huge) ≈ 0, and collision probability rounds to 100%.

Now we have everything we need to answer the original question. Let’s circle back to it.

So why 128 bits?

The collision risk depends on the ratio k²/2N, and we can’t control k (the number of IDs generated).

What we can control is N (the size of the ID space), and N is determined by how many bits we use.

So the design question becomes: how big does N need to be so that even after years of operation, k²/2N stays comfortably in Act 1**?**

Let’s plug in 64 bits and see what happens. N = 2⁶⁴ ≈ 1.8 × 10¹⁹. Now let’s see how the collision probability evolves as k grows:

A billion IDs gets us to a 2.7% collision risk, and this will probably happen at any reasonably large company. And by 10 billion IDs, collisions are nearly guaranteed.

Now let’s redo the same exercise with 128 bits. N = 2¹²⁸ ≈ 3.4 × 10³⁸:

Even at a quadrillion IDs, k²/2N is still 0.0000000015. To reach a meaningful collision risk at 128 bits, you’d need to generate trace IDs in numbers that exceed every trace ever produced by every observability platform on Earth, combined.

We can’t stop k from growing, but we can choose an N so vast that the cliff in Act 3 sits beyond any horizon we’ll ever care about.

That’s why 128 bits.

Why not 256, then?

If 128 bits is good, isn’t 256 bits even better? Mathematically, yes. Practically, no.

Every trace ID has to be propagated on every HTTP request between services, stored alongside every span, indexed in every backend, and shipped through every log line. At scale, those bytes add up. 128 bits is 16 bytes; 256 bits is 32. Doubling the storage and bandwidth costs of every piece of trace data for a safety margin we already won’t reach in any realistic universe isn’t a trade anyone wants to make.

128 bits is the ideal sweet spot, collision safety effectively forever, and it happens to match the size of a UUID, which means every database, every language, and every protocol already knows how to handle it.

Hey there, it’s Elizabeth from SigNoz!

I wasn’t planning on making this a blog of its own and initially wanted to combine it with this. Later, when I got on a call with Pandey (the brains behind Nightswatch), I realised this deserved a blog of its own.

This is an attempt to do justice to the insane work our platform-pod does daily to keep all our customers’ cloud instances alive and stable. I hope you enjoy reading this as much as I enjoyed learning about this and crafting it for you.

Today, our internal observability system watches 6 regions, ingesting 21 billion metric points, 14 TB of logs, and 10 TB of traces per day without breaking a sweat.

We are people who build an observability tool for a living. We spend our days helping customers monitor their systems, debug their incidents, and make sense of their telemetry. So you’d think we’d have our own house in order.

We do now, but the story of how we got here starts with everything falling apart.

Our Early System

About 4 years ago, on an eerie Wednesday morning, one of our US cluster’s ClickHouse (our database) nodes started running hot, some queries were timing out, and a handful of customers were seeing slow dashboards. We followed the standard operating protocol of opening the internal monitoring tool, checking the metrics, and identifying the bottleneck.

Except the monitoring tool was down too.

SaaSMonitor, the system we’d built to watch our customer deployments, had been ingesting so much telemetry that its own collector had crashed.

This wasn’t a one-off event and kept happening.

Our internal monitoring grew organically over the years, starting with a hand-stitched setup called Testbed for our early manually provisioned customers, then SaaSMonitor was bolted on when we launched self-service sign-ups. This meant that if you were debugging something in the US region, you had to check two separate URLs.

Also, telemetry data went straight from collection to storage without a buffer in between, so any spike in volume would choke the pipeline. Every pod had two containers, an application and a sidecar, but metrics were only exposed at the pod level, implying that when something broke, you couldn’t tell which container was the culprit. There was no opt-in mechanism, and we collected everything from every pod by default, which meant we were drowning in data we didn’t need, making the volume problem even worse.

We tried some quick fixes, like putting an Envoy in front of SaaSMonitor as a load balancer, hoping that spreading traffic across more collector instances would prevent the crashes. It didn’t work because distributing an overwhelming load across more instances just gives you more instances that are overwhelmed. We needed something fundamentally different.

That’s when we started building Nightswatch, named after the Night’s Watch from Game of Thrones 🥷🏻. It was our Project Hail Mary, a single, unified system to observe every cluster, every node, and every container across SigNoz Cloud using SigNoz itself. It took our platform pod over a year, 21 issues, and a complete rethinking of how we observe ourselves.

This is the story of what we built.

What are we observing?

Before we get into the story of how we built systems for observability, it helps to understand what exactly we’re trying to watch.

SigNoz Cloud is a multi-tenant platform, and we run three regional Kubernetes clusters across the US, EU, and India. When a customer signs up, they get their own isolated namespace within the cluster of their chosen region. Inside that namespace, they get their own SigNoz instance, their own ClickHouse for storing telemetry data, their own OTel collector for ingestion, and their own endpoint (something like acme.us.signoz.cloud).

Not everything is isolated, though; some infrastructure is shared across all tenants in a cluster. The Nginx controllers that route incoming traffic, the OpenTelemetry gateway that handles initial ingestion, and Redpanda (a Kafka-compatible streaming platform) that buffers data, are pooled resources that every tenant’s data flows through. But the core components where customer data actually resides and gets queried are fully siloed per tenant.

Here’s a slightly abstracted architecture of our customer’s data plane:

All of this, including the shared pipeline, the per-tenant components, both flows, across three regions and hundreds of customers, is what Nightswatch needs to observe. Monitoring all of this is what Nightswatch was built for, and, like its Game of Thrones namesake, it has a very interesting structure for guarding the realm.

Overview of Nightswatch

Nightswatch is our approach to running SigNoz to monitor SigNoz Cloud. Inspired by the Night’s Watch from Game of Thrones, the system is split into three roles, each named after a branch of the order.

Builders man the Wall, and they run on every node, gathering metrics, logs, and traces from the pods around them. Rangers venture beyond it by actively probing customer endpoints and watching for cluster-level threats like node failures and pod evictions. Stewards keep the supply lines running, and they buffer and forward telemetry from builders and rangers to the Castle, a dedicated SigNoz instance where everything comes together.

Under the hood, all three are OpenTelemetry Collectors, each configured differently for its role, but the magic lies in how they work together.

Builders: Eyes on Every Node

I shall live to collect my house gossip and die at my post.

Builders are the most local members of the Watch, and each one cares only about what’s happening on its own node. They run as OpenTelemetry Collector daemonsets, meaning one builder per node in the cluster. If a node has 15 tenant pods, the builder on that node scrapes all 15. If another node has 3, that builder scrapes 3.

For metrics and logs, the builder uses a pull model, scraping each container’s /metrics endpoint (similar to how Prometheus works) and polling container log files on disk at a configurable interval. For traces, it flips to a push model: the application is instrumented with an OpenTelemetry SDK and sends traces directly to the builder’s OTLP endpoint.

Not everything gets collected by default, and this is an intentional change we brought in. In the old system, we collected everything from every pod with no way to opt out, which generated more data than we needed and contributed to the pipeline crashes. Nightswatch fixes this with a two-tier approach: node-level metrics like CPU, memory, disk, and network are always collected, because you never want a node on fire with no basic metrics for debugging, but container-specific metrics and logs are opt-in, controlled by annotations on the pod.

The annotation system is the heart of how builders work. In SigNoz Cloud, each tenant pod has multiple containers, such as the application and a sidecar OTel collector, each exposing metrics on different ports. Pod-level scraping can’t distinguish between them, which was another blind spot in the old setup.

Nightswatch annotations solve this by targeting a specific container within the pod, using <container> in the annotation name which gets replaced with the actual container name.

For metrics:

<OUR_DOMAIN>/<container>.mdscrape: true
<OUR_DOMAIN>/<container>.mdport: 8888
<OUR_DOMAIN>/<container>.mdpath: /metrics
n<OUR_DOMAIN>/<container>.mdinterval: 10s

For logs:

<OUR_DOMAIN>/<container>.ldscrape: true
<OUR_DOMAIN>/<container>.ldpipeline: json/nginx/...
<OUR_DOMAIN>/<container>.ldinterval: 200ms

The prefixes tell you the signal type:

• md → metrics discovery

• ld → logs discovery

And each annotation controls a specific behaviour: whether to scrape, which port and path to hit, how often to scrape, and, for logs, which parsing pipeline to use.

Builders give you deep visibility into what’s happening inside every node, but they can’t tell you whether the customer’s endpoint is actually reachable from the outside, or whether the cluster itself is healthy. Here’s where rangers step in.

Rangers: Eyes on the Cluster

I shall live to probe and check the stability of my kingdom and die at my post.

Unlike builders, which run as daemonsets (one per node), rangers are deployments; we just need a replica running somewhere on the cluster. In our case, they run on a dedicated node pool reserved for Nightswatch workloads, so they never compete for resources with customer pods.

We run two types of rangers, each watching a different surface:

The ingress ranger acts as a synthetic customer. It periodically hits each customer’s endpoint, something like acme.us.signoz.cloud/healthz and checks whether it gets a healthy response. If it doesn’t, that could mean the customer’s SigNoz pod has crashed, or that the Nginx controller is misconfigured. Either way, the failure flows to the Castle as a data point and can trigger an alert immediately.

Like builders, the ingress ranger uses annotations to know what to probe:

<OUR_DOMAIN>/prdscrape: "true"
<OUR_DOMAIN>/prdpath: "/healthz"
<OUR_DOMAIN>/prdinterval: "10s"

The prd prefix stands for probe discovery, prdscrape enables probing, prdpath specifies which path to hit, and prdinterval controls how frequently it runs.

The k8s ranger watches the cluster’s infrastructure health by talking to the Kubernetes API server. It collects cluster-level metrics like node count, pod states, resource usage, and available capacity, giving us the big-picture view of whether a region is running healthy or approaching its limits.

It also captures Kubernetes events: whenever something notable happens, like a pod getting OOMKilled or a container crash-looping, the API server emits an event. The catch is that these events are ephemeral, meaning Kubernetes discards them after about an hour. The k8s ranger grabs them and ships them to the Castle as permanent records, which turns out to be invaluable for post-incident debugging.

Builders and rangers together give us full visibility of what’s happening inside each node, whether endpoints are reachable, and whether the cluster is healthy. But all that telemetry still needs to get somewhere safely. If the pipeline between collection and storage chokes, which is what kept killing the old system, none of this visibility matters.

This is why stewards are central pieces of Nightwatch.

Stewards: The Supply Line

I shall live to serve the builders and the rangers and die at my post.

Remember the core problem with the old system? Telemetry went straight from collection to storage with nothing in between, and any spike in volume would choke the pipeline. We actually tried fixing this once before, in V1, we put Envoy in front of SaaSMonitor as a load balancer, hoping that distributing traffic across more collector instances would stop the crashes. It didn’t work since distributing an overwhelming load across more instances just gives you more instances that are overwhelmed. A load balancer alone wasn’t the solution; we needed to engineer a complete buffer pipeline.

That’s what the stewards are: three OpenTelemetry Collectors working together to move data reliably from collection to storage.

Envoy sits at the front as the entry point. When builders and rangers from the US, EU, and IN clusters send telemetry, Envoy is the first thing that receives it, distributing incoming connections across however many OTel Gateway instances are running behind it. It’s gRPC-aware; the protocol OTel uses to ship telemetry, so it can distribute connections more intelligently than a basic TCP load balancer. The same component that failed in V1 works perfectly in V2, because this time it has the rest of the pipeline behind it.

OTel Gateway receives telemetry from Envoy, batches it, and forwards it downstream. It runs in gateway mode, meaning it doesn’t collect telemetry from local sources; it just receives, processes, and sends. The gateway is exposed via an internal load balancer without authentication, so builders and rangers in any region can send telemetry to the Nightswatch cluster without special credentials.

Redpanda is where it all comes together. It’s a Kafka-compatible streaming platform that acts as a durable buffer between the gateway and the Castle. The gateway writes data into Redpanda, and the Castle’s SigNoz instance consumes from it at whatever pace it can handle. If the Castle slows down or goes offline temporarily, data doesn’t get lost; instead, it queues up, and the Castle catches up later.

Castle

All telemetry ultimately flows into the Castle, which is a dedicated SigNoz instance. This lives in a separate management cluster (the control plane), completely isolated from the data plane clusters where customer workloads run.

This separation was deliberately implemented to prevent the castle from being affected if one of the region’s clusters crashed. The Castle also monitors itself by running its own builders and rangers.

Nightswatch was our platform-pod’s first-ever milestone spanning over 1 year and 3 months, 21 issues to build something we’re insanely proud of. And the Wall still stands. If building observability at this scale sounds like your kind of challenge, we’re hiring. Come join the Watch! 😉

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This blog took 6 days and 7 hours to be curated, so make sure to show some love!

As an observability enthusiast working at an observability startup and running an observability newsletter, I find this topic wildly fascinating. I know a bunch of lore on how companies thought about and invented (or, more precisely, reinvented) their observability systems to support their growing scale. But two of these stories have stuck with me and are interesting because each broke its observability system at a critical moment of growth and rebuilt it in a completely different and particularly breathtaking way, from which we have a lot to learn!

1. Netflix

Netflix’s observability origin story starts in a place that will make most engineers wince. In May 2011, Netflix was using a home-grown solution called Epic to manage time-series data. Epic was a combination of Perl CGI scripts, RRDTool logging, and MySQL. Alongside Epic, their telemetry was split between this home-grown tool and an IT-provisioned commercial product. Epic’s flexibility letting engineers send in arbitrary time-series data and query it made it popular, and it became the primary system of record.

They were tracking around 2 million distinct time series, and the monitoring system was regularly failing to keep up with the volume of data, and several things were about to make it dramatically worse: Netflix was shifting from rolling pushes to red/black deployments, starting to actually leverage auto-scaling rather than just using fixed-size groups, and expanding internationally into Latin America and Europe.

All these changes required them to scale by at least an order of magnitude from 2 million to 20 million metrics or more. Perl CGI scripts and MySQL were never going to handle what Netflix was becoming, and it was simply beyond what Epic was capable of.

So in early 2012, they started building Atlas, and by late 2012, it was being phased into production, with full deployment completed in early 2013.

The design philosophy behind Atlas is a chapter filled with learnings. Atlas features in-memory data storage, allowing it to gather and report very large numbers of metrics very quickly. It captures operational intelligence whereas business intelligence analyses trends over time, operational intelligence provides a picture of what is currently happening within a system.

Since their focus was primarily on operational insight, the top priority was determining what’s going on right now. This led to the following rules of thumb:

1/ data becomes exponentially less important as it gets older

2/ restoring service is more important than preventing data loss

This is a fundamentally different philosophy from store everything forever. Netflix decided that recent data matters enormously and old data barely matters at all.

The internal Atlas deployment breaks data into multiple time windows. The last 6 hours of data is kept fully in memory, so they can show recent data as long as clients can successfully publish. Everything is sharded across machines in these in-memory clusters. For older data, they compute rollups via Hadoop processing, drastically reducing data volume for historical queries.

One of the things they really wanted to fix from Epic was how dimensions worked. In the old system, everything was mangled into a metric name with different conventions per team, and users had to resort to complex regular expressions to slice and dice data. In Atlas, a metric’s identity is an arbitrary, unique set of key-value pairs. Some keys are set automatically by the client library (server name, AWS zone, ASG, cluster, application, region), with significant flexibility for users to specify whatever keys make sense for their use case.

The growth numbers tell the story of why all this mattered. In 2011, they were monitoring 2 million metrics. By 2014, they were at 1.2 billion metrics, and the numbers continued to rise. They routinely see Atlas fetch and graph many billions of datapoints per second. Today, Atlas processes 17 billion metrics and 700 billion distributed traces per day on 1.5 petabytes of log data, and the system’s architecture has kept observability data processing to less than 5% of Netflix’s infrastructure costs!

But even Atlas hit its limits. A few years ago, Netflix’s SRE team was paged because their alerting system was falling behind, and the critical application health alerts were reaching engineers 45 minutes late. One platform team had programmatically created tens of thousands of new alerts, which overwhelmed Atlas’s query capacity. They were looking at an order-of-magnitude increase in alert queries over the next 6 months, and scaling up Atlas’s storage layer to serve that volume would have been prohibitively expensive, since Atlas was already one of Netflix’s largest services in both size and cost.

Their answer was Atlas Streaming Eval, moving alerting from a cron-based query model to a streaming model. Today, they run 20x as many alert queries as a few years ago, at a fraction of the cost. Multiple platform teams at Netflix programmatically generate and maintain alerts on behalf of their users without affecting others, and streaming evaluation enabled them to relax cardinality restrictions and to alert on queries that were previously rejected.

What’s special here is that instead of throwing more hardware at the problem, they changed the model entirely, and in my opinion, that’s what separates great observability teams from the rest.

Some interesting references!

• Introducing Atlas: Netflix’s Primary Telemetry Platform (Netflix Tech Blog)

• Improved Alerting with Atlas Streaming Eval (Netflix Tech Blog)

• Lessons from Building Observability Tools at Netflix (Netflix Tech Blog)

• Solving Mysteries Faster with Observability (InfoQ / QCon)

• Atlas Documentation (Netflix OSS)

2. Uber

Uber’s observability story starts in 2014 with a Graphite, Carbon, and WhisperDB stack that was held together very loosely. By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack, which stored them in the Whisper file format in a sharded Carbon cluster. They used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts.

The problems were fundamental, like the stack not being horizontally scalable, meaning you couldn’t add capacity just by adding machines. There were no replicas, so a single node dying meant losing an eighth of all Uber’s data and adding capacity required taking the system offline for a week or more. Martin Mao’s first on-call week was spent deleting data from the backend just to keep the observability stack alive.

First, they did a quick fix by swapping in Cassandra for time-series storage and ElasticSearch for the metrics index, all stitched together with Go. They stood up this new system in time for Halloween 2015, which was Uber’s second-largest peak load event. That year was the first time Uber’s observability system didn’t have an outage during the Halloween peak.

But Cassandra was the wrong tool for the job because they were using it as a time-series database even though it was built as a key-value store. As they entered their hyper-growth phase, the firefighting that had plagued the Graphite years resurfaced in a new form.

The team decided to build M3DB, a custom time-series database with an embedded inverted index from scratch. The architecture they landed on is worth understanding in detail.

Applications on hosts emit metrics to a local daemon called “Collector“, which aggregates them at 1-second intervals and then forwards them to the aggregation tier using a shard-aware topology retrieved from etcd. The aggregation tier further aggregates into 10-second and one-minute tiles, and the M3DB ingestor writes them to the storage tier. M3 Coordinator acts as a Prometheus sidecar, providing a global query and storage interface on top of M3DB clusters. It handles downsampling and ad hoc retention using rollup rules stored in etcd, which runs embedded in the binary of an M3DB seed node.

Let’s look at the results (quite phenomenal). Any given second, M3 processes 500 million metrics and persists another 20 million aggregated metrics. Extrapolating to a 24-hour cycle means roughly 45 trillion metrics per day, and the platform also houses over 6.6 billion time series!

The really interesting engineering is in the high-dimensional problem. High-dimensionality metrics; data tracked over time with many different aspects like route, region, and status code are critical to the business but costly at Uber’s scale. A single emission could lead to 100 million unique time series, and because code changes roll out to specific groups of cities over a few hours, they need city-level monitoring granularity. Different cities have different configurations; for example, rider pickups might be blocked on a street due to a parade, or local events can cause traffic changes.

Their alerting ecosystem is equally bespoke; it includes two in-datacenter alerting systems: uMonitor for time-series metrics-based alerting against M3, and Neris for host-level checks. Both feed into a common notification and deduplication pipeline called Origami. uMonitor uses static thresholds for steady-state metrics and anomaly thresholds via Argos, Uber’s anomaly detection platform, which generates dynamic thresholds from historical data.

They also added Jaeger, their open-source distributed tracing system. Jaeger’s distributed tracing follows requests from one service to another, composing a narrative of what happened and what went wrong, making it much easier to pinpoint causation.

The operational improvement after M3 was dramatic. Setting up monitoring in new data centres became 4x faster, and the operational maintenance burden dropped by over 16x, while combined high/low-urgency notifications per week went from 25 with Cassandra to 1.5 with M3DB. 👏🏻

Over a million unique visitors hit their systems every day, and more than half of their engineering team are using these observability tools daily.

Some resources that were my references and really good reads!

• M3: Uber’s Open Source, Large-Scale Metrics Platform for Prometheus (Uber Engineering Blog)

• How Uber Built its Observability Platform (The Pragmatic Engineer)

• Observability at Scale: Building Uber’s Alerting Ecosystem (Uber Engineering Blog)

• Optimizing M3: How Uber Halved Metrics Ingestion Latency by Forking the Go Compiler (Uber Engineering Blog)

• Optimizing Observability with Jaeger, M3, and XYS (Uber Engineering Blog)

But here’s an interesting dilemma. What happens when the product you’re monitoring is the monitoring tool itself? When the observability system that’s supposed to tell you everything is broken... is the same system you need to diagnose the problem?

At Signoz, we have solved this exact problem by building a system called Nightswatch, a Game of Thrones-themed architecture featuring builders, rangers, and stewards to run SigNoz to monitor SigNoz.

That story drops in the next edition. Stay tuned.

Subscribe now

Feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between!
&
This piece took 6 days, 5 hours to be cooked, hope we served. 🌚

Subscribe now

There are two popular prophecies floating around tech circles these days.

The first says SRE is the future of all software engineering, that as AI writes more and more code, the humans who remain will be the ones keeping systems alive. The second says AI will devour every tech job alive, SREs included. Neither is particularly useful if you’re an SRE trying to figure out what your Tuesday will look like in 2027.

Let’s ask a more grounded question by looking at what’s already happening: When AI handles 95% of your incident response, do you get worse at handling the 5% that actually matters?

Most of us already use AI for our daily work (our brains a little fried!), and so do SREs. Today’s discussion agenda is not whether AI replaces SREs, but whether AI is quietly making SREs less capable and whether anyone will notice anything before the next novel outage hits. The foundational framework for this entire debate comes from a 1983 research paper that’s eerily prescient.

The Ironies of Automation: Part-I

Note from author: Below is a brief precursor to History of Automation, which you might enjoy if you are into History and Cultures (like me 😉).

In 1983, a cognitive psychologist named Lisanne Bainbridge published a paper called Ironies of Automation. It became one of the most cited papers in human factors research, and its core argument is almost uncomfortably relevant today.

Bainbridge studied what happened when factories and industrial systems automated the work of blue-collar operators. The findings were paradoxical and revealed that the more you automate a process, the more critical the human operator becomes during the rare moments automation fails and the less practice they get, and the worse they become at exactly those interventions. Automation, which was inherently designed to remove humans from the loop, left them with the worst possible job, i.e., long stretches of passive monitoring punctuated by rare, high-stakes crises they were increasingly unprepared for.

Ring any bells yet? 🙂

Basically, I’m drawing a parallel between the AI revolution and industrial automation. Industrial automation reshaped blue-collar work by taking over routine physical tasks, and the workers who remained had to handle exceptions they’d lost the muscle memory for. AI is doing the same thing to knowledge workers by taking over the routine cognitive tasks, the pattern matching, the triage, the known-issue resolution and leaving humans with the rare, complex, ambiguous problems.

The exact problems that require deep expertise, the exact expertise that atrophies when you stop practising.

Now we’re replaying this pattern with AI agents, and the stakes in software systems are only growing.

Current State of AI in SRE

Let’s take stock of where things stand today in the world of site reliability engineering.

What’s already automated or heavily AI-assisted?

Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.

What’s on the horizon?

Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures, predictive incident detection before users are affected, AI-driven change risk assessment and automated rollbacks.

PagerDuty frames this as a tiered model.

• Tier 1 incidents: Known issues with known fixes get fully automated.

• Tier 2 incidents: Partially understood problems receive AI recommendations with human validation.

• Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.

But here’s the catch.

If human SREs (okay, now we have to use adjectives like human 🥹) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them? Intuition is usually developed from years of hands-on incident response, pattern recognition built through repetition, and the kind of gut-level understanding of a system that only develops from painfully waking up at the odd hour to solve the bug that brought the system down.

The Crisis of Deskilling

This is where the picture starts getting blurry. Let’s look at some emerging research on AI-induced deskilling across multiple fields, which paints a consistent and concerning picture.

In medicine, a recent study found that endoscopists who used AI assistance for polyp detection saw their unassisted detection rates drop from 28% to 22% after a period of AI use. They got worse at the thing they were supposed to be experts in, not because they forgot the theory, but because they stopped exercising the skill.

In aviation, research has shown that long-haul pilots who rely heavily on autopilot systems experience measurable degradation in situational awareness and manual flying ability. The problem got serious enough that the FAA now mandates more manual flying time to counteract the effect.

Somewhere over the past year, AI stopped being a tool I occasionally reached for and became the first thing I reach for, always. My instinct now is to offload as much as possible and apply my own thinking only where it’s absolutely unavoidable. The problem is that these moments are becoming the only exercises my brain gets, and I can feel the rust.

We can draw a pattern here. The more you let the system handle, the worse you get at handling things yourself and here’s the truly dangerous part, you don’t feel it happening. It gets masked as hyper-productivity. Cognitive research suggests that because AI tools make tasks feel easier and enhance visible performance, users are often unable to accurately judge the true status of their own skills. You feel competent, dashboards look green, and then on a Wednesday, a novel incident hits that doesn’t match any pattern the AI has seen, and you realise the muscle has atrophied.

For SREs, this manifests in specific ways, like we stop reading raw log streams because the AI summarises them, we stop forming hypotheses during incidents because the AI suggests root causes, we stop building mental models of system architecture because the AI maps dependencies for us, and each of these individually looks like a productivity win. Collectively, they hollow out the very expertise that makes an SRE effective when things go sideways in ways nobody anticipated.

But there’s something even more concerning than deskilling, and researchers have started calling it never-skilling. Deskilling means you once had a capability but have since lost it. Never-skilling means you never developed it in the first place. For junior SREs entering the field today in an environment where AI handles most of the incident response workflow, the opportunities to build foundational intuition and muscle through hands-on practice are vanishing.

The training pipeline itself is broken and not self-healing.

SREs realise their skills are degrading and lean more on AI to compensate, which further degrades their skills, creating a vicious cycle from which escape is difficult.

What Can We Do About It?

We are definitely not rejecting AI tooling; we are adopting it and integrating it stronger than ever before, because that’s the only way forward.

A few approaches worth considering:

Deliberate inefficiency. Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities. This can be considered as a long-term investment to keep skills fresh, although it might come at the cost of a super-fast solution

Build for human-in-the-loop, not human-on-the-side. There’s a meaningful difference between a system where a human approves an AI’s recommendation and one where a human actively engages with the problem alongside AI. The former keeps humans in a supervisory role that Bainbridge (the lady who wrote that research paper about 40 years ago) showed leads to vigilance decay, and the latter keeps them cognitively engaged.

Let’s zoom out and take a look at the bigger picture.

The Bigger Picture

Everything we’ve discussed here, the ironies of automation, the deskilling risk, the never-skilling problem, collectively applies well beyond SRE. Software engineering as a whole is navigating the same tension. As AI writes more code, reviews more PRs, and handles more debugging, the same questions apply.

We’re talking about SREs specifically because that’s the world we live in at SigNoz. We build an open-source observability platform, the kind of tool that gives SREs the metrics, traces, and logs they need to understand their systems deeply. For us, this deskilling question is not a rhetorical fad; it directly shapes how we’re building AI into our product.

Our approach is to start with an AI assistant that helps SREs leverage the power of LLMs while keeping humans firmly in control. Eventually, we’ll enable more autonomy but within clear guardrails, and only as trust is earned.

One advantage we have in this space is that, as an observability platform, we sit on the data itself, the metrics, traces, and logs that SREs rely on. Most AI SRE products today integrate with observability tools through APIs, which means they’re working with a limited, second-hand view of your systems. Because we own the data layer, we can build much deeper, more context-aware AI capabilities that understand your system the way an experienced SRE would.

And to answer the burning question in your head, our goal isn’t AI that replaces SREs. It’s AI that supercharges SREs. Unlike ongoing lore, we believe humans will remain essential for the decisions that matter most, especially those that impact production infrastructure.

The future of SRE is human with AI intentionally designed to keep humans sharp, engaged, and ready for the 5% that really counts.

Here’s the LinkedIn post our founder posted a few days ago, which inspired me to write this.

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

When I first encountered sampling about a year ago, I knew only about head- and tail-based sampling. Mainly because most mainstream documentation covered primarily about them.

But recently, I realised I’d only been looking at the tip of the iceberg.

I stumbled upon an article that discussed sampling in greater depth. I decided to go down the rabbit hole to find the strategies that don’t get the spotlight and make this edition about the lesser-known types of sampling.

Let’s look at them in greater detail.

#1. Remote Sampling

To put it simply, it’s head-based sampling, but centrally controlled. Each service fetches sampling rules from a central config server. You can specify default and per-endpoint rates in a JSON file, and applications poll for updates periodically. If you are still wondering what the bigger deal is, it is that we can increase or decrease the sampling rate during incidents by changing this file, and within a minute, the applications pick up the new sampling rates.

That is quite powerful. Despite being battle-tested (used in Uber!), there’s surprisingly little documentation in OpenTelemetry. Users often struggle to enable Jaeger-style remote sampling with OTel. Some resort to running a Jaeger agent solely to serve the sampling config. OpenTelemetry supports it, but there is very little documentation. Remote sampling lets you keep a low baseline sample rate (say, 1-5%) most of the time and only ramp up to 50-100% when needed, such as during an incident or a debugging session. Because you don’t need a redeploy, teams are more likely to actually adjust rates to control costs or get details when it matters.

#2. Consistent Reservoir Sampling

It’s essentially head-based sampling that guarantees a fixed sample size. Instead of a simple random percentage, a reservoir sampler maintains a rolling buffer of traces, retaining exactly N traces per time window by using a discrete set of sampling rates and consistency algorithms to ensure fair selection.

Probabilistic sampling yields a variable number of samples, i.e if traffic doubles, so do your sampled traces and costs. Reservoir sampling always uses a fixed sample size. It’s statistically representative because the algorithm rotates items in the reservoir with uniform probability.

This strategy essentially puts a hard ceiling on trace ingestion. It’s ideal for ensuring you don’t exceed your budget, even during traffic spikes. The trade-off is that during very low-traffic periods, you might underutilise capacity, but usually most teams prefer predictable costs to a few extra traces.

#3. Metrics-from-Traces

You can sample traces aggressively, for example, only keep 5%, but still extract metrics from 100% of them before they’re dropped. In practice, this means placing a metrics-generation stage in your telemetry pipeline before the sampling stage. OpenTelemetry makes this possible with components such as Span Metrics and Service Graph.

If we naively sample traces, we also lose information needed for metrics such as request rates, error counts, and latencies. One solution is to tally up metrics before any sampling decisions.

In an OTel Collector, we might chain a spanmetrics connector in the pipeline, then a Sampling processor after it. SpanMetrics will emit metrics (RED metrics such as request rate, error count, latency distributions, service call graphs, etc.) for every span that passes through, so you get complete coverage. Then the sampler (head or tail) drops, say, 95% of spans before storage. The result is that our monitoring dashboards and alerts, which rely on metrics, remain 100% correct, while your trace storage volume is only 5% of raw traffic.

#4. Byte-Rate Limiting (Throttle by Data Volume)

This refers to sampling based on the size of traces, not just the count. This is an often-overlooked but effective strategy, you set a cap, such as ingesting at most 10 MB of trace data per second. The sampler then makes decisions to stay under that throughput. OpenTelemetry recently added a bytes_limiting policy in the tail-sampling processor for this. You can read more about it here.

It uses a token bucket algorithm, which is common for rate limiting, but the tokens represent bytes. The collector actually measures the size of each trace in bytes, using the protobuf serialised size to accurately account for how much data each trace would consume. You configure a sustained bytes-per-second rate and a burst capacity. For example:

policies:
  - name: volume-limit
    type: bytes_limiting
    bytes_limiting:
      bytes_per_second: 10485760  # 10 MB per second
      burst_capacity: 20971520   # allow bursts up to 20 MB

If a few gigantic traces arrive, the processor will quickly use up the token budget and start dropping subsequent traces until the rate falls back under 10 MB/s. Conversely, if traces are small, more can pass through until the aggregate size hits the limit.

This becomes extremely useful when trace sizes vary a lot. For instance, one request might normally produce a 50 KB trace, but a worst-case code path might generate a 5 MB trace. A standard sampler working per-trace might keep both equally, but the latter one trace costs as much as 100 smaller ones.

#5. Adaptive Sampling

Adaptive sampling adjusts trace sampling rates in real-time based on live traffic patterns or performance signals. The goal here is to keep overall data volume within budget while dynamically increasing sampling during anomalous events. For instance, you might normally sample only a small percentage of requests, but automatically raise the sample rate when latency or error rates spike beyond an SLO threshold. One strategy is throughput-based adaptation; setting an upper limit on traces per second and letting the system tune the probability to meet that cap. Another is key-based dynamic sampling, where the collector samples frequent events less and rare events more.

Here’s an interesting GitHub thread.

Adaptive schemes keep observability costs predictable by avoiding oversampling during high-traffic periods, yet they can temporarily boost fidelity when something goes wrong.

Care must be taken to ensure coordination across distributed services so that increasing sampling doesn’t overload the system or skew the data.

In my opinion, the shift from conventional probabilistic sampling to the methods above reflects a change in how we view observability. Ultimately, the right sampling strategy aligns your visibility needs with your infrastructure budget, and as OpenTelemetry matures, it will likely become the new standard for any team operating at scale.

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This post took 5 days, 11 hours to be curated, so make sure to show some love!

OpenTelemetry has become the de facto choice for many organisations’ observability needs today. And with it, auto-instrumentation has turned out to be a powerful means to implement the same.

But with great power comes great responsibility.

While auto-instrumentation provides a strong baseline, its out-of-the-box (or magical?) nature often produces a telemetry surplus and is a double-edged sword. Because auto-instrumentation is designed to be comprehensive by default, it captures everything in case you need it. Without intentional refinement, this can dilute your signal-to-noise ratio, leading to the generation of surplus telemetry that can inflate storage costs while burying actionable insights under a heap of low-value signals.

While certain types of telemetry surplus are tied to specific libraries, such as HTTP or gRPC, most telemetry waste is a byproduct of the language runtime itself. To illustrate this, we will use Java in the context of the blog. That said, the lessons presented here aren’t isolated; the signals we’ll uncover are often common patterns across the broader landscape of modern frameworks.

This blog is an attempt to help you sieve out the diamonds (good telemetry) from the rocks (noisy telemetry)!

Java Agent for Auto-instrumentation

By simply attaching a Java agent at runtime, developers can capture traces, metrics, and logs without modifying a single line of application code. The Java agent runs in the same Java Virtual Machine (JVM) as the application, using bytecode manipulation libraries such as ByteBuddy to rewrite classes as they are loaded.

The Java agent automatically hooks into common frameworks such as Spring Boot, Tomcat, and JDBC drivers to inject span creation and context propagation logic. While effective, this process, as mentioned before, can result in the generation of not-so-useful telemetry data that can later bog down storage and cause issues.

Let’s discuss them in greater detail.

The Defaults You Should Know About (and Might Want to Disable)

I’ve curated a list of commonly seen (and publicly complained of) not-so-useful telemetry data, referred to as telemetry surplus. Let me introduce them one by one.

#1. URL Path and target attributes

— not specific to Java

Another commonly missed issue is that auto-instrumentation for HTTP clients and servers often captures the full http.url or http.target attribute. If an application uses RESTful paths with unique IDs like /api/users/12345, every unique ID creates a new attribute value.

This can be considered a waste because it prevents effective aggregation. Aggregation works by grouping similar data into the same bucket based on shared attributes. Hence, if we use a templated route like /api/users/:id, the system puts every ‘Get User’ request into a single bucket thereby letting you accurately calculate the p99 latency for the entire ‘Get User’ feature.

Hence, make a mental note to always use templated http.route rather than the raw path, which can result in millions of useless data points, aka wasteful telemetry.

#2. Controller spans

In frameworks like Spring MVC, auto-instrumentation by default creates multiple spans for a single web request. Some types of spans are,

• Server Span (SpanKind.Server): The parent span. It tracks the entire process, from when the request reaches your server to when the user receives a response.

• Controller Span (SpanKind.Internal): A child span. It tracks only the time spent inside your @Controller method.

• View Span (SpanKind.Internal): Another child span. It tracks how long it took to turn your data into a JavaServer Page (JSP).

The obvious catch is that in modern micro-services, controllers are often very thin, and they just immediately call a Service or a Database. If your database call is already being tracked, having a separate span that says the controller took 2ms adds very little value. That is, for most cases, you might not need spans that capture controller and/or view execution.

The good news is that you can effectively suppress the generation of these spans by using experimental flags. Here are some flags that let you achieve the desired effect, as mentioned in OpenTelemetry documentation.

#3. Thread name in run-time telemetry

A major source of high-cardinality data occurs in Java runtime metrics, like jvm.network.io or jvm.memory.allocation. Versions 2.10.0, 2.11.0, and 2.13.1 of the agent included the thread.name attribute by default in these metrics. In environments that use large thread pools or virtual threads, this creates an unbounded number of unique time series, potentially leading to a cardinality explosion.

This issue was later corrected; maintainers removed the attribute from default metrics starting with version 2.18.0 (via PR #14061). So, if you are using a previous version make sure you have set proper guardrails or bump up to a later version.

#4. Duplicate Library Instrumentation

This is an interesting dilemma.

Let’s first dissect the problem at hand. So, usually Java agents automatically attach to every supported library they find in our class path (of the application/ project) and end up instrumenting multiple layers of the same outgoing request.

Let me break this down with an example.

In modern Java development, we rarely use a low-level library directly. Instead, we use high-level SDKs. For example:

1. Application Code calls the AWS SDK to upload a file to S3.

2. The AWS SDK (high-level) uses Apache HttpClient (mid-level) to execute the request.

3. Apache HttpClient uses Java Networking (low-level) to send bytes over the wire.

Now, the Java Agent would see all three layers and create three separate spans for the same single logical operation. This results in nested spans that describe the same work, effectively doubling or tripling the telemetry volume for every outbound call.

To prevent this, the OpenTelemetry Java Agent suggests using a Span Suppression Strategy. This logic detects when an instrumentation point is already wrapped by another instrumentation point higher up the call stack.

The behaviour is controlled by the following property: otel.instrumentation.experimental.span-suppression-strategy

There are three primary strategies used to decide which spans to keep and which to discard. You can read more about that here.

#5. Resource attributes

Auto-instrumentation detectors for Kubernetes and host metrics often capture dynamic, unique identifiers by default, such as container.id, k8s.pod.uid, or process.pid. When these are attached to metrics (specifically), they create a new time series for every single container restart or process launch. This tampers with aggregation, and the metrics database is flooded with thousands of dead time series, increasing storage costs and significantly slowing down query performance for long-term trends, adding to telemetry surplus.

#6. JDBC and Kafka Internal Signals

Certain auto-instrumentation modules are inherently chatty, generating high-frequency spans for internal mechanics that carry little diagnostic value.

For example, the jdbc-datasource module (now often disabled by default) creates a span every time a connection is retrieved from a pool via getConnection(), resulting in thousands of entries that merely confirm the pool is functional.

Similarly, Kafka instrumentation can produce excessive spans for background heartbeats and metadata checks.

To mitigate this noise, these specific modules can be disabled upstream by setting -Dotel.instrumentation.jdbc-datasource.enabled=false or -Dotel.instrumentation.kafka.enabled=false, OR filtered downstream in the Collector to drop specific span names like poll or heartbeat, depending on the greater architecture of your application.

#7. Scheduler and Periodic Jobs

— can be broadly applied to schedulers and jobs in different languages

Applications using Spring Scheduling or Quartz for background tasks like polling a database or checking a cache every second generate a span for every single execution. If a job runs once per second but does nothing interesting 99% of the time, it creates 86,400 successful but meaningless spans per day. This qualifies as telemetry waste in most cases.

You can disable the generation of these scheduler spans by using the system properties listed below.

#8. SDK Misalignment

Another massive source of enterprise surplus occurs when a framework like Trino initialises its own internal OpenTelemetry SDK instance instead of joining the global instance provided by the Java agent.

This results in two parallel telemetry pipelines running in one JVM, doubling memory overhead and network traffic. Because the instances are separate, the valuable business spans from the framework, which often misses the agent’s auto-detected resource attributes like Kubernetes namespace, making the data invisible to standard production queries and hence becomes telemetry surplus.

Mitigation Strategies 🚒

Now that we have seen several ways your application could generate telemetry, this section provides a broad overview of how you can mitigate the resulting waste. As they say, prevention is better than cure; generating less telemetry surplus is the best way to eliminate it, but in most cases, it’s almost inevitable, and it’s important to learn how to mitigate it.

Mitigating telemetry waste requires a smart combination of upstream prevention and downstream pruning. For upstream, the most effective defence is selective enablement. By disabling the default capture everything behaviour and re-enabling only critical modules, while specifically suppressing known chatty modules or experimental controller spans as mentioned in the sections above. Downstream, where the telemetry meets the collector, it serves as a powerful filter using the processor to delete redundant resource keys and employing tail sampling to keep 100% of error traces, while sampling only a tiny fraction of successful, low-signal traffic can reduce data volume without sacrificing diagnostic efficacy.

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This blog took 6 days and 7 hours to be curated, so make sure to show some love!

Contributing to open-source can be overwhelming at first, and it’s okay to feel a little lost when trying to navigate your way through it. OpenTelemetry is one such open-source project under the CNCF [the second-largest, to be precise].

With over 200 contributors, it’s doing really well and is growing fast. I’ve been part of the community [and advocating its adoption!] for a while and see many people who wish to contribute to the project asking for tips, guidance, and direction in the Slack channels. There isn’t a lack of resources in this aspect, but it could be a bit scattered across a dozen different repos and docs. This blog is an attempt to bring all the resources you need to get started in one place, in a capsule.

I’ve been following Diana Todea’s journey very closely for a while, and recently she won the OpenTelemetry Community Awards at KubeCon, NA 2025. So my obvious next step was to hop on a call with her and gather as many insights as I could! Between our conversation and her own recent writings, I’ve distilled the best insights on how you can move from a lurker to a contributor.

I am also addressing a problem here: While many folks want to contribute, there is a shortage of folks who actually make it to their first PR, and even fewer who consistently continue to contribute and stay active. I’m writing this to address both hurdles, helping you get started and find a reason to stay.

So, if you’ve been waiting for a sign to start or restart, this is it. 💖 ✨

#1. What’s the first step I should take?

Kudos to you for taking the first leap. You can start by joining the CNCF Slack channel [of which OTel is a part] and come say hi in the #hallway channel. Here are some examples.

You can also follow suit and introduce yourself. Next, since contribution is the primary goal, here’s the official documentation outlining key aspects you should know. The next step you could take is try finding a good first issue from this list.

You can also check out CLOtributor, which helps you find good first issues across a number of Cloud Native projects. Here are some channels you can join initially. [as per Diana’s blog]

#otel-sig-end-user, #otel-devex, #opentelemetry-new-contributors, #otel-contributor-experience, #otel-docs-localization

But now you could run into your first dilemma. Let’s see how to get over it.

#2. I can’t find a good first issue, wtd1 ?

Finding a good first issue is indeed a task in its own. Most of them could already have been picked up by someone, and there could be active discussions around them. Because these issues are beginner-friendly, they are highly competitive and often claimed within hours of posting.

If this is the case, you can shift your strategy. While these issues are great for a quick win, they rarely help you build the rapport/ relationships or architectural understanding necessary for long-term contribution. This is why the Special Interest Group [SIG] model of OpenTelemetry Community is so important.

You can always start small, by being an active part of the community, including SIG calls and discussions in the corresponding channels, and by trying to make yourself useful with ad hoc tasks.

#3. I made a PR, not getting any reviews, wtd?

Give it some time.

Most maintainers have a day job in addition to maintaining the project, so small delays can occur. You can always post a message in the corresponding Slack channel with enough context so that anyone can pick up the review task. Here’s an example.

#4. I want to contribute, but non-technically, wtd?

The good news is you are in high demand!

There’s a lot of work that is not coding that could use a lot more hands on deck. Some means of contribution include documentation and blogs. You can find ways to improvise existing documentation or add new ones. You can also join the End User Working Group [EUWG]. They are constantly looking for people to share implementation stories, conduct user interviews, or improve feedback loops between vendors and users. The Merge Forward is another initiative that focuses on diversity and inclusion, and needs allies to help run mentorship programs and community events.

If you’re the kind of person who likes helping others, you can contribute by simply being active in forums or Slack to answer questions from newer users. Helping troubleshoot issues or explaining concepts in the Slack channels or GitHub discussions is a valuable form of contribution, too. So, by being a friendly helper in the community, you’re contributing to the project’s success, and you might build a reputation for yourself along the way.

If you’re interested in the process side of things, OpenTelemetry, being a pretty huge project, has many SIG meetings, public notes, and release planning. You could volunteer to help with note-taking in a SIG meeting, or assist in organising community events like the OpenTelemetry Community Day at KubeCon. The Contributor Experience SIG focuses on improving the project for contributors; they might have initiatives you can join, even if you’re not contributing code.

Another piece of good news is that you can always switch tracks or do both code and non-code contributions. In our call, Diana emphasised that a contributor’s journey can be very fluid; you might start with documentation because that’s what you’re comfortable with, and later move into code as you learn more, or vice versa. The path you choose initially doesn’t lock you in; all contributions count, and in a project as broad as OpenTelemetry, there is a need for a diverse set of skills, which can be the best launchpad for you [if you utilise them well!].

#5. How to contribute actively and remain consistent?

Here’s something harder than getting your first PR merged. Staying consistent and active in the community. Many, many people drop off after a couple of contributions. Here’s when consistency and discipline come into the picture, much like hitting the gym 😅.

Consistency in open source comes from aligning your contributions with what genuinely excites you. You have options to choose from, ranging from whether it’s a SIG you’re passionate about or a specific skill you want to grow. Set a realistic routine, such as contributing weekly or monthly, and stay connected by attending SIG meetings, tracking GitHub updates, or staying active in Slack.

You can stay in the loop by attending the bi-weekly SIG meetings for your area, even if just as a listener at first, or by joining community calls.

As Diana puts it, when something triggers you and helps you learn, it becomes easier to show up consistently and enjoy the journey. And, like everything else in life, motivation is intrinsic and should come from within. 🧘‍♀️

#6. Ok, but what do I get out of this?

Trick question.

Contributing to OpenTelemetry or any open source project, for that matter, is indeed an investment of your time and effort. The good news is the ROI2 can be huge, both personally and professionally.

Today, OpenTelemetry sits at the forefront of observability. By contributing, you’ll gain a much deeper understanding of how instrumentation, tracing, metrics, and related technologies work under the hood. As you debug issues or implement features, you’ll inevitably learn a ton about distributed systems, telemetry data, and best practices in cloud-native architectures from the industry’s best people! You’ll also be interacting with engineers from many companies [since OpenTelemetry has contributors from Lightstep, Google, SigNoz, and dozens of organisations]. These connections can lead to job opportunities or collaborations in the future. Many contributors find that being active in open source eventually opens multiple doors.

Many people also contribute out of a passion for the technology and the ethos of open source; if you’ve benefited from free software, there’s a gratifying element of paying it forward. That motivation can be very fulfilling in itself.

Some areas that could use more help

OpenTelemetry is a broad project with many moving parts, and naturally, some parts of it have more active contributors than others. If you’re looking to make a real impact and perhaps have an easier time finding issues to tackle, it helps to know which areas are currently under-resourced. Based on community insights and what maintainers have pointed out, here are a few areas in need of more contributors:

• Documentation Localisation: As Diana mentioned, translating docs is a major need. Some language communities, like Japanese and Chinese, have been very active in translating OpenTelemetry docs, but others have barely started. If you are fluent in any language besides English, you can make a big difference by contributing to localisation efforts.

• Language SDKs with smaller teams: OpenTelemetry maintains SDKs for many languages. Some of these, especially the most popular languages, have large contributor teams, but others could use help. For example, newer or less common language implementations might have only a couple of maintainers. If you happen to know a language like PHP, Ruby, Erlang, or Rust, those SDKs might appreciate extra contributors to help fix bugs and implement new features to catch up with the latest spec.

• eBPF Instrumentation [OBI]: One of the newer frontiers in OpenTelemetry is the eBPF auto-instrumentation a.k.a. OBI. This allows automatic telemetry data capture at the kernel level without modifying application code. If you’re interested in low-level programming or Linux kernel tech, the OBI project would love some help!

Being part of the community and taking on responsibilities is as simple as sending an intro message to any SIG or channel you’re particularly interested in and asking if you can help out with anything. It can be as easy as the screenshot below! Thanks to the amazing community made even more welcoming by the great people in it!

So go, and make a change!

Feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

I’ve been an OpenTelemetry advocate for over a year and have written many, many blogs on adopting OpenTelemetry in your systems to achieve deep observability. Yet, I’ve always wondered how and what actually happens behind the scenes, in the context of auto-instrumentation.

So, this is me breaking down what happens under the hood of OpenTelemetry for you.

Refresher on Auto-instrumentation

For those who are new to the space, auto-instrumentation refers to collecting telemetry [traces, metrics, logs] from your application without requiring you to make changes to the application code. You can read more about it from the official docs here.

A helpful way to understand how this works is to separate the OpenTelemetry API from the OpenTelemetry SDK.

• The OTel API is the interface for creating telemetry — “start a span”, “add an event”, “record a metric”, “propagate context”, etc. Both manual instrumentation [your code] and auto-instrumentation [instrumentation libraries/agents] ultimately use these same API calls. But in auto-instrumentation, it’s taken care of for you automatically.

• The OTel SDK is the implementation behind the API — it decides what actually happens to that telemetry [sampling, batching, processing] and where it goes [exporting].

So auto-instrumentation is typically achieved in two parts,

• Instrumentation hooks [libraries/agents] that wrap existing functions and call the OTel API at the right points.

• SDK configuration that ensures those API calls actually record telemetry and can be exported.

In Auto-instrumentation, OpenTelemetry wraps existing function implementations and extracts useful data, such as function parameters, execution duration, and results. It’s important to note that the way this wrapping and hooking is done varies widely across programming languages. Broadly, we can say that there’s a clear difference between how it works in dynamic languages [like JavaScript, Python, and Ruby] versus statically-typed or compiled languages [like Java, Go, and .NET].

Let’s dive into those differences (or similarities!) next.

Dynamic vs. Static Languages

It becomes easier to understand what happens behind the scenes when classifying the languages broadly into dynamic and static. Dynamic languages allow instrumentation to patch or wrap functions at runtime easily, whereas static languages, on the other hand, don’t natively allow such runtime patching, so they require different techniques to insert instrumentation code. That is, most dynamic languages like Python, JavaScript, and Ruby, which are more flexible at run-time, depend on methods like monkey-patching to implement auto-instrumentation. While other static languages or those that run on virtual machines like Go or C rely on techniques like build-time injection for the same.

Some Cool Techniques

OpenTelemetry’s auto-instrumentation toolkit boils down to a couple of clever techniques that make all of this possible. Let’s discuss two of the most common methods used under the hood.

Monkey Patching

The lore behind the term monkey-patching fascinated me. Apparently, the word’s etymology comes from guerrilla-patching, which refers to the sneaky act of changing code at runtime to fix a bug or add a feature without altering the original source code. Because guerrilla and gorilla are near-homophones, the term was intentionally used as a pun, gorilla-patch. Eventually, developers who wrote their patches more carefully began calling them monkey-patches to make the process sound less intimidating than a gorilla.

Okay, now let’s get back to the engineering. In dynamic languages such as Python and Node.js, functions and modules are treated as first-class objects that reside in mutable memory structures. This allows OpenTelemetry to employ monkey patching, a technique where existing functions are replaced with instrumented wrappers at runtime.

The concept is straightforward, at runtime, we replace existing functions with instrumented versions that inject telemetry before and after calling the original function.

This piece of code roughly illustrates what happens in Node.js.

const originalFunction = exports.functionName;

function instrumentedFunction(...args) {
  const startTime = process.hrtime.bigint();
  // invoke the OG function here
  const result = originalFunction.apply(this, args);
  const duration = process.hrtime.bigint() - startTime;
  console.log(`functionName(${args[0]}) took ${duration} nanoseconds`);
  return result;
}

exports.functionName = instrumentedFunction;

OTel JavaScript uses a package called require-in-the-middle to intercept module loading and apply such patches before your code runs.

Let’s see how this could work in Python. Say we are trying to collect data from an HTTP client, like requests. Python’s requests lib, exposes a separate function for each HTTP method [requests.get / requests.post / requests.put, and so on]. But each of these functions eventually calls an internal request method, whose parameters are the method, the URL, and all the kwargs. The function then returns a response object.

Let’s see what this looks like pseudo-code-wise:

def request(method, url, **kwargs):
	# Original implementation

def wrapped_request(method, url, **kwargs):
	before = datetime.now()
	# Call the original implementation
	response = request(method, url, **kwargs)
	# Collect the necessary information
	duration = datetime.now() - before
	collect_data(method, url, response.status_code, duration)
	# Return the value from the original call
	return response

To close the loop, the original function implementation needs to be replaced with the new wrapped_request. For dynamic languages, this is done by simply holding a reference to the original implementation and replacing the function with its name. A pseudocode implementation [which isn’t very, very far from a real life code] looks like this:

original_request_impl = requests.request

def wrapped_request(method, url, **kwargs):
	# Wrapped implementation as appears, has the original call
	# As shown in the previous snippet

requests.request = wrapped_request

Calling these requests won’t result in any observable change, albeit the auto-instrumentation will keep collecting necessary data.

Byte-code Instrumentation

This is the underlying technique for languages that run on a virtual machine. Instead of modifying functions at the language level, this approach modifies the compiled code [bytecode] as it’s being loaded into the runtime. Essentially, the instrumentation injects extra bytecode instructions that call OpenTelemetry APIs around the target method’s original instructions.

In the Jurassic Java world, this is done via a special agent. When you run a Java app with the -javaagent flag pointing to the OpenTelemetry Java Agent JAR, the JVM invokes the agent’s premain() method before anything else.

public static void premain(String args, Instrumentation inst) {
    new AgentBuilder.Default()
        .type(ElementMatchers.nameStartsWith("com.example.TargetApp"))
        .transform((builder, typeDescription, classLoader, module, protectionDomain) ->
            builder.method(ElementMatchers.named("targetMethod"))
                   .intercept(MethodDelegation.to(MethodInterceptor.class))
        ).installOn(inst);
}

In that premain(), OTel registers a class transformer [as seen in the snippet] with the JVM. As each class loads, the transformer can inspect it and, if it matches one of the known libraries or functions we want to instrument [e.g., a Servlet filter, a JDBC call, etc.], the agent will modify the class’s bytecode on the fly to insert the telemetry hooks. The end result is that by the time your application’s code runs those functions, they already have tracing logic woven in.

Bytecode instrumentation is extremely powerful because it works at the Java virtual machine [JVM] level, making it language-agnostic within the JVM ecosystem. It can instrument Java, Kotlin, Scala, and other JVM languages without any modification.

The trade-off is a bit more complexity and setup — you need to run the app with the agent [or enable the profiler], and there is some startup overhead to transform classes. Once running, the performance impact of the injected code is usually minimal. Overall, this technique lets OpenTelemetry achieve deep, broad instrumentation of popular frameworks in Java and .NET with near-zero friction for the developer.

Abstract Syntax Tree Modification

Unlike Python, which is a dynamic language and Java, which is a kind of static language that runs in the VM, Go is a static language that does not use a VM, making it an outlier in this case. In Go, auto-instrumentation works by modifying the Abstract Syntax Trees [ASTs].

It was in the Compiler Design Course of my undergrad degree when I first got introduced to ASTs. It’s primarily a data structure widely used in compilers to represent program code. An AST is usually the result of the syntax analysis phase of a compiler. This is exactly where the auto-instrumentation comes into the picture as well.

The auto-instrumentation process of Go involves parsing the source code into an AST, adding instrumentation code to the tree, and generating the modified source code before compilation. This approach ensures that the instrumentation is incorporated in the final binary, providing zero runtime overhead for the instrumentation mechanism itself. But it does come with trade-offs, including the need for access to source code, which makes it difficult to instrument third-party libraries and plugins, and the need for complex changes to build pipelines.

Final Words

Delving into how OpenTelemetry auto-instrumentation works behind the scenes reveals a lot of clever engineering. The mechanisms that we learnt above allow OTel to hook into your application’s execution, gather context and timing information, and funnel it into the OTel SDK, all without you changing your application code. 😊

As an OpenTelemetry user, you don’t usually need to worry about these details, but understanding them can be helpful when you are instrumenting

In the end, what feels like telemetry appearing out of thin air, aka auto-instrumentation, is actually the result of these well-orchestrated techniques. Knowing this, you can better appreciate the work done by the OTel community and troubleshoot issues with a deeper intuition.

Happy instrumenting!

On another note, SigNoz along with InKeep is hosting a webinar on Debugging AI Agents: Observability Best Practices with Inkeep & SigNoz. Check out if it is something that interests you!

Feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

In honour of the Stranger Things finale, I’ve hidden a few easter eggs throughout this edition for all the fellow fans out there. See if you can spot them all! As this is our final edition of the year, I want to wish all of my readers a very happy holiday season, a joyous Christmas, and a wonderful new year.

Cheers.

When I was building applications, I used to always rely on the DevTools console of my web browser to examine logs in the frontend. But, with UI log messages only being accessible within your browser rather than forwarded to a file somewhere, which is the common pattern with backend services, losing visibility of this resource when triaging user issues was a real dilemma. Since adding any kind of monitoring/ observability solution would blow up the bundle size, I’d try to avoid it as much as possible.

But here’s the thing, neglecting observability for reducing bundle size isn’t a good trade-off. There are several other ways for you to run up that hill, and meanwhile, if you are caught in a scenario where your requests are not being sent, and the site is crashing and everything is turning upside down, you’ll have to inevitably start looking inside.

Inside your traces, spans and contexts.

In this blog, we explore strategies to trim the bundle impact of OTel, focusing on tree-shaking [removing unused code] and lazy-loading [deferring loading until needed] and how to apply these in different frameworks.

Impact of OpenTelemetry on Bundle Size and Performance

Out of the box, adding OpenTelemetry’s web libraries can introduce quite a significant amount of JavaScript. For example, the official browser auto-instrumentation bundle was about 300 KB uncompressed [~60 KB gzipped] after recent optimisations, which is in the same ballpark as many third-party RUM [Real User Monitoring] agents. While 60 KB may seem okay-ish, loading and executing this script during initial page load can delay rendering. A large script can increase blocking time, potentially pushing out LCP [Largest Contentful Paint — the render of the largest element] beyond the optimal 2.5s threshold.

Core Web Vitals are very sensitive to any render-blocking resources. We generally avoid deferring critical content, but telemetry scripts are not user-facing content. In fact, web performance guidelines note you should not lazy-load an LCP image [since that delays visible content]; however, lazy-loading a telemetry script is a good practice precisely because it’s non-essential to the user’s immediate experience. The challenge is finding a balance: we want to collect telemetry [traces of page loads, API calls, user interactions, metrics like Web Vitals, etc.] for observability, but we must prevent the OTel code from slowing down the page. We will look at two proven techniques — tree-shaking and lazy-loading to reduce bundle bloat.

Tree-Shaking 🌳 OpenTelemetry Code

Tree-shaking is a build optimisation that removes dead code, including modules or functions that your application doesn’t actually use. OpenTelemetry’s JavaScript SDK is modular, which means if you import only certain parts [say, the tracing API and one exporter], you should be able to exclude others [like metrics, logging, or unused instrumentations]. Ensuring that tree-shaking works with OTel involves a few considerations:

Use Modern ESM Imports

All OTel packages support ES Modules. Import only the symbols you need, rather than entire libraries. For example, if you only need the web tracer and the OTLP exporter, you might do:

import {WebTracerProvider }from'@opentelemetry/sdk-trace-web';
import {BatchSpanProcessor }from'@opentelemetry/sdk-trace-base';
import {OTLPTraceExporter }from'@opentelemetry/exporter-trace-otlp-http';

This pulls in only tracing-related code and the OTLP trace exporter, leaving out metrics and logging code.

Avoid Catch-all Imports or Meta-Packages

OpenTelemetry offers auto-instrumentation packages that conveniently bundle many instrumentations. For example, @opentelemetry/auto-instrumentations-web will include document load, fetch/XHR, user interaction, and more. If you use it, your bundle will include all those instrumentations. To keep things slim, only import the instrumentations you actually want individually, instead of a blanket import. This way, unused ones can be dropped.

In code, that means doing something like:

import {DocumentLoadInstrumentation }from'@opentelemetry/instrumentation-document-load';
import {FetchInstrumentation }from'@opentelemetry/instrumentation-fetch';
// ... then use these in registerInstrumentations ...

If you don’t need, say, user interaction tracking or certain network instrumentation, not importing them will ensure they don’t appear in the bundle.

Mark OTel Packages as Side-Effect-Free

Tree-shaking works best when libraries declare that they have no side effects on import. Many OTel packages now include sideEffects: false in their package.json, which helps Web-pack/Rollup know it can safely drop unused exports.

This was more of an issue in the previous versions. A user noted that manually adding sideEffects: false to OTel packages reduced bundle size by ~40KB, and the OTel maintainers addressed this in later releases. You can view the Github discussions here. Using OpenTelemetry JS v1.2+ or v2.x is recommended, as newer versions have improved in this area. In fact, the OTel JS SDK 2.0 [released in 2025] explicitly removed certain patterns [like extensive classes or namespaces] to improve tree-shakability and minification.

Upgrading to the latest version can yield a smaller bundle thanks to these optimisations!

Consistent Versioning to Avoid Duplicates

One subtle cause of bundle bloat that often goes missed, is version mismatches. If you depend on multiple OTel packages that internally bring different versions of the core API, you might accidentally bundle two copies. Ensure all your OTel packages are on the same version so the bundler can deduplicate them. For instance, if everything is on version 1.5.0 except one package on 0.26.0, you may get two sets of code.

Aligning package versions will help prevent that scenario.

In summary, tree-shake aggressively. That means prune everything optional — disable features that aren’t useful anymore, drop instrumentations you don’t need, and let your bundler eliminate the dead code. By doing so, you minimise the impact on bundle size to a great extent.

Lazy-Loading the OpenTelemetry SDK

This is the next concept you can explore. Lazy-load the OTel code, so it isn’t even downloaded or executed until after the critical page content is loaded. This strategy has perhaps the biggest positive impact on LCP and initial load performance. The idea is to defer the initialisation of OpenTelemetry modules to a non-critical moment [for example, after the page’s main content is on screen or when the user interacts], rather than blocking the main thread early.

Dynamic import() in Single-Page Apps

In a React or other Single Page Application [SPA], you can use the dynamic import() function to load your telemetry setup code asynchronously.

For example, you might create a module otel-init.js that configures the OTel SDK, and then instead of importing it at the top of your app, you load it on demand. For instance:

// In your main App component
useEffect(() => {
import('./otel-init').then(module => {
module.initTelemetry(); // call the initialization function exported here
  });
}, []);

This ensures that the OTel code [everything inside otel-init and its imports] is pulled in only after the first render. The UI can render, LCP can happen, and only then does the telemetry code load in the background. From the user’s perspective, the page appears quickly; from the app’s perspective, OTel starts slightly later.

Code-Splitting with Bundler Config

If you’re using Webpack, you can explicitly split OTel into its own chunk. For example, in an Angular app using Webpack, you can configure a separate cache group for @opentelemetry modules.

This means your build will produce something like main.js and opentelemetry.js. However, to truly lazy-load that chunk, you should ensure it’s not required immediately. In practice, that again means using dynamic import or a similar mechanism to load that chunk at a later time. The Webpack config might look like:

// webpack.config excerpt
optimization: {
splitChunks: {
chunks:'all',
cacheGroups: {
opentelemetry: {
test:/[\\\\/]node_modules[\\\\/](@opentelemetry)[\\\\/]/,
name:'opentelemetry',
priority:10,
reuseExistingChunk:true,
      },
    },
  },
}

There’s a small trade-off here. Delaying the loading of OTel modules would also inevitably result in the loss of some early telemetry data. For example, if you want to capture any errors or events during the first few seconds, a delayed start misses them. If those are crucial, you might decide to load a minimal part of OTel early [or use a buffered logging approach] and load the rest later. It’s a balancing act.

Both of the above are proven techniques for bringing down bundle size. Apart from these, there are some more optimisations for how we send telemetry data from the browser and framework-specific techniques, which I’ll cover in another edition. Till then, adieu!

Feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This piece is written by Piysuh, Software Engineer at SigNoz, who was also one of the key contributors for making this engineering overhaul happen.

Cheers.

Customer logs data is always messy.

Being (and building!) an observability platform, we get to see all the beautiful, creative ways it can be messy, every single day. And yet, our customers expect, quite fairly, I might add, perfect query results and peak performance.

SigNoz is an open-source observability platform that can be your one-stop solution for logs, metrics and traces. Using ClickHouse as a single datastore and built to support OpenTelemetry natively, SigNoz can help you troubleshoot issues faster with powerful querying capabilities on your observability data.

We recently overhauled how we store JSON logs in ClickHouse [our datastore] to improve query performance and enable filtering of nested dot-notation keys, which was previously not possible. What started as an investigation into filtering inconsistent dot-key notation in JSON logs ended up optimising our query performance by 30%.

In the process, we developed a two-tier JSON storage model that helped us overcome the limitations of ClickHouse’s native JSON data type while paving the way for superior query and aggregation performance for any key in customers’ logs.

How We Used to Store and Query Log Data Earlier

Before this overhaul, we stored the raw log body as a simple string data type. While this was easy to ingest, it created some bottlenecks when developers tried to interact with the data.

Slow Run-Time Parsing and the Impossible GROUP BY

Storing the log body as a string meant the database had no way to instantly look up values inside the JSON whenever a filter is applied to log data.

Whenever a user entered a filter query like the one above, the entire log body [stored as a string] is dynamically converted to JSON at runtime. This led to slow query performance, especially when the selected time range was long or there was too much data to scan.

Currently, we address slower query performance with the help of log pipelines. With log pipelines, users can transform their logs to suit their querying and aggregation needs before they are stored in the database.

Though log pipelines are helpful, they are not straightforward. To achieve the necessary performance, users had to manually implement a log pipeline to extract key-value pairs from the JSON string and store the extracted fields as separate attributes.

This is not a seamless out-of-the-box experience for users sending JSON logs.

The Ambiguity of Dot Notation

The final breaking point that spurred our full investigation was the ambiguity created by dot notation. Our query builder could not differentiate between logically different JSON structures when developers used dots for querying:

Scenario 1: Key with Dot in Name:

{
  “user”: {
    “session.id”: “abc-1234”
  }
}

Scenario 2: Nested JSON Structure:

{
  “user”: {
    “session”: {
      “id”: “xyz-5678”
    }
  }
}

Although both of these logs record the same piece of information, it is difficult to differentiate between the two when the user wants to run a query for them. The query needed to find the data in the first example will not work on the data from the second example, and vice versa.

And we needed something that works on both.

This means that when a user performs a search, they might get incomplete results, not realising that some data is being missed simply because of a formatting difference. We can’t expect our users to write separate, complex queries to find both formats or even perform a union to get the necessary data.

This was a really big pain point for us, and the ultimate trigger.

Normalising JSON logs in the Collector

Our first and most direct approach was to solve the problem before the data reached the database. The idea was to intercept incoming logs in our OpenTelemetry collector and transform the JSON structure in-flight.

The proposed solution was to have the collector inspect the keys of every incoming JSON object.

If a key contained a dot, e.g., {”a.b”: “c”}, our code would parse the string, create a nested JSON structure, e.g., {”a”: {”b”: “c}}and replace the original flattened key.

But this involved modifying the actual data the user was sending, and the performance issue was still unresolved. Given these drawbacks, we concluded that modifying the data shape within the OpenTelemetry collector was not a viable path forward.

And at the same time, ClickHouse announced a stable version of the JSON data type.

Using ClickHouse’s native JSON data type

With the introduction of a native JSON data type in ClickHouse, we identified an opportunity to migrate from our string data type column to JSON. Adopting this also meant offloading all associated operations to ClickHouse, allowing us to leverage ClickHouse’s highly optimised, built-in functions for JSON traversal and data extraction.

But there was a limitation. Clickhouse’s native JSON type is built to handle dynamic paths of JSON keys. In order to do so, it needs them to be predictable. But log data from our customers is hardly predictable. It may contain any number of unique paths.

Before understanding how we overcame this limitation, let’s understand more about ClickHouse’s JSON data type.

Inside the Working of ClickHouse JSON Type

ClickHouse’s JSON data type allows you to store semi-structured JSON documents in a column while preserving efficient, columnar storage for individual JSON fields. Internally, JSON columns flatten nested JSON keys into subcolumns for query efficiency, as demonstrated below.

To define a JSON column, you can provide optional settings like max_dynamic_paths in the column definition, which controls how ClickHouse handles dynamic paths [incoming JSON fields whose schema or structure is unknown].

Understanding this is crucial to the solution we finally designed.

Understanding max_dynamic_paths

The setting max_dynamic_path , limits the number of distinct JSON paths it will treat as separate subcolumns for any given chunk of data. This limit is defined at the table’s column level, but it is enforced per data part; each chunk of stored data [or “part”]. By default, this value falls back to 1024.

But for our customer logs, we can not have this limit.

Sometimes the incoming data can have really high cardinality [sigh], which could lead to an explosive avalanche of paths exhausting the upper limit. When the threshold is surpassed, if another distinct key appears, that key’s values [and any other new keys] will be stored in a shared data structure.

The next question that naturally arises is, what is a reasonable value for this setting to get optimal performance? Let’s dive deeper.

What is a reasonable maximum for JSON’s max_dynamic_paths?

The max_dynamic_paths setting controls how many unique JSON paths can be promoted to dedicated subcolumns per data part. The reasonable maximum is far from straightforward; it depends heavily on your data’s shape and the storage backend.

In most high-cardinality systems, like observability or event analytics platforms, customer-generated data contains extremely diverse JSON keys. A single dataset might include fields like order.id, order.user_id, or even arbitrary UUIDs (yes, seriously!) nested deep in the JSON structure. In such cases, even if you raise max_dynamic_paths to thousands, it gets consumed quickly because every unique key or UUID becomes a new path. No number ever feels enough when users continuously send data with new identifiers baked into the keys.

But what if we set max_dynamic_path = 0 and create columns for dynamic paths on demand.

Building a Two-Tier JSON Storage Model

By setting max_dynamic_path = 0, we stopped the creation of sub-columns for any JSON path. This meant that all the JSON data ingested is stored directly in the shared data structure, not as sub-columns.

This becomes our baseline. Now let’s talk about performance. The effect of the change on querying is better than what existed [storing logs as a string]. With the introduction of multiple JSON serialisation formats in ClickHouse, we faced yet another critical architectural decision —which format would deliver the best for our heavy workloads, especially for all the frequent GROUP BY queries?
Let’s examine that in greater detail.

#1. Storing Data in Advanced Serialisation Format

ClickHouse provides several serialisation formats for storing JSON data, including the Map type, bucketed maps, and the advanced JSON format. You can read more about these formats here.

We performed benchmarks on both map and advanced shared data structure and found there were some big wins for advanced shared data structure.

There were some big wins from opting for this format, which included better performance for aggregation and filtering of data; hence, operations like GROUP BY or WHERE clauses on specific JSON fields could be executed with high efficiency.

You can read more about advanced shared data structures here.

This became Tier 1 of our architecture.

#2. Promoting Frequently Queried Paths

While Tier 1 provides an efficient baseline for querying any JSON attribute, it is not optimised for fields that are accessed frequently. The overhead of checking metadata and decoding values becomes significant at scale for certain hot fields.

To address the performance challenges associated with querying large JSON objects in log data, we implemented Tier 2, designed to minimise query latency by separating frequently accessed fields from the larger, less-queried JSON blob.

The core of this optimisation is the use of two distinct JSON columns for storing log attributes:

1. Primary JSON Blob: A standard JSON column that serves as the default repository for all incoming log attributes, which was discussed in #1. This column accommodates the long tail of infrequently accessed fields.

2. Secondary JSON Column: A second, specialised JSON column [promoted] is dedicated to storing key-value pairs that are frequently used in query filters, aggregations, and dashboards. This column is configured to leverage ClickHouse’s dynamic path settings (default of 1024), which we had set to zero for the primary blob. For example, a path/key called body.status_code is frequently queried, then it becomes stored in our secondary or promoted column.

This provides the expected performance with ClickHouse JSON columns, without compromising consistency in structure. But how does the system determine which fields are commonly queried? Let’s dissect that.

#3. Selecting and Ingesting Promoted Fields

Let’s think of it as a two-part process.

1/ If a user expects to increase performance over a certain key or path, it will be added to a separate table named promoted_paths , let’s call them as hot fields for now. Every 10 seconds, the ingestion service refreshes a cached list of these hot fields. If a new field gains prominence in queries, it is added to this cache list.

2/ During data ingestion, the ingestion service inspects each incoming log. If the log’s JSON payload contains keys that match the list of promoted fields in the cache list, those key-value pairs are extracted and moved into the secondary/ promoted column. To prevent data duplication and reduce storage overhead, these keys are simultaneously removed from the primary JSON blob.

Comparing Results ~ 30% Faster, 100% Lighter

We compared the performance of the two-tier JSON model with the older String Column on filtering and group by queries.

On testing the query performance with a 9TB dataset, we found that the JSON data type is 30% faster in execution time and scans around 99% less data, with a slightly higher memory usage.

Here are the stats for the comparison we did with different combinations of filters on both storage models.

Conclusion

This entire technical optimisation would enable customers to enjoy an enriched experience and query seamlessly without worrying about the shape and form of their data. With the two-tier model, the challenges that plagued our old system were systematically eliminated. Inconsistent JSON structures are now gracefully handled, with hot fields promoted and the rest stored efficiently. The slow string searches that once took minutes are now sub-second queries on structured data.

If you want to try our new logging experience, you can reach out to cloud-support@signoz.io.

If you loved this engineering deep-dive, here are some similar ones:

• 100 Github Releases. Yet it’s day one.

• How we made our Queries 99.5% faster

• Engineering a Trace Details Page That Handles a Million Spans

Feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

On another note, feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

Cheers.

So, you’ve embraced OpenTelemetry, and it’s been great.

Pat, Pat.

That single, vendor-neutral pipeline for your traces, metrics, and logs felt like the future. But now, the future is getting bigger. That simple OTel Collector configuration that worked perfectly for a few services is starting to show its limits as you scale. The data volume is climbing, reliability is becoming a concern, and you’re wondering if that single collector instance is now a bottleneck waiting to happen.

You’re not alone. As applications grow, the question quickly shifts from what OTel can do to how we can deploy it effectively at scale. In this post, we’ll explore some deployment patterns for the OpenTelemetry Collector, moving from a simple agent to a robust, multi-layered architecture. Let’s look at the three main deployment patterns for OTel collectors and break down how each trades off complexity, scalability, and isolation; thus, choosing the right one depends on your architecture and goals.

#1. Load-Balanced/ Gateway Pattern

Instead of relying on a single, large OTel Collector, which you can also think of as a single point of failure 😄, this pattern uses a fleet of identical, stateless collectors sitting behind a load balancer. The idea is to distribute the incoming telemetry data across this fleet, so if any single collector instance fails, the others can seamlessly take over its workload.

Architecture with the Load Balancer

The data flows through a few distinct layers, as shown in the figure below.

Layer 1: Agents

You still have OTel Collectors running as agents. These can be on individual hosts, as sidecars to your applications, or on a single Kubernetes node using a DaemonSet. The agent’s job is simply to collect data locally, batch it, and forward it to a single endpoint, the load balancer.

Layer 2: The Load Balancer

This is the central entry point for all telemetry data from your agents. It can be a cloud load balancer [like an AWS ELB/NLB or a GCP Load Balancer], or a self-hosted one like Nginx or HAProxy.

Its only job is to receive the data and distribute it across the fleet of gateway collectors using a strategy such as round-robin or a standard hashing algorithm.

Layer 3: The Gateway Collector Fleet

This is a group of two or more identical OTel Collector instances. They are the workhorses. Each collector in the fleet receives a fraction of the total data from the load balancer. They perform the heavy processing — advanced filtering, batching, retries, and exporting the data to one or more backends [e.g., SigNoz, Jaeger, etc.].

Trade-offs & Considerations

High Availability [HA]: If Collector 2 fails, the load balancer detects this and automatically redirects its traffic to Collector 1 and Collector 3. The pipeline remains up.

Horizontal Scalability: If your data volume doubles, you don’t need to make your collectors twice as powerful [vertical scaling]. You can simply add more collectors to the fleet [horizontal scaling].

Zero-Downtime Maintenance: You can perform rolling updates. Take one collector out of the load balancer’s pool, update it, and add it back. Repeat for the others without ever interrupting data flow.

Complexity: This architecture introduces a new component, the load balancer, which must also be configured, managed, and monitored.

Stateful Processors: This pattern is ideal for stateless processing. If you use OTel processors that rely on seeing all data for a given entity [e.g., the spanmetrics processor, which needs all spans for a trace], simply spraying data randomly can lead to incorrect results.

In such cases, you may need to configure your load balancer for “stickiness” or use a more advanced collector routing mechanism to ensure related data is routed to the same instance.

#2. Multi-cluster/ Central Control-Plane Pattern

Using a simple deployment strategy across many Kubernetes clusters is causing growing problems. It becomes hard to maintain consistent configurations and control your data with global rules.

Managing each cluster separately also creates security risks by storing credentials across multiple systems. At the same time, costs increase as each cluster sends data over expensive networks. The multi-cluster pattern fixes this by creating a central pipeline, making your data management secure, cost-effective, and easier to control.

The Multi-Stage Architecture

This pattern typically involves at least two layers of collectors, creating a collect and forward chain.

Layer 1: In-Cluster Collection [The Agent Layer]

Inside each of your Kubernetes clusters, you run a local OTel deployment. This usually consists of a DaemonSet of collectors acting as agents [one per node] that scrape local data. These agents then forward their data to a small in-cluster gateway [a Deployment within the same cluster].

The primary role of this layer is to collect all data within its own cluster, add cluster-specific metadata [e.g., cluster.name: prod-us-east-1], and forward it to the next stage.

Layer 2: Regional Aggregation [The Gateway Layer]

This layer involves a central, highly available fleet of OTel Collectors to serve an entire region or logical environment [e.g., all US-East production clusters]. This regional gateway receives data from the in-cluster gateways of all the clusters it manages.

This is also where you can centralise your logic. The regional gateway handles:

• Authenticating with the final observability backends.

• Enforcing global sampling rules.

• Enriching data with region-level metadata.

• Routing data to different backends based on type or team.

Trade-offs & Consideration

Enhanced Security: Only the regional gateways need the secrets to connect to your final backends. The collectors inside your many clusters do not significantly reduce your security footprint.

Centralised Management: You can manage your primary configuration [export destinations, sampling, etc.] in one place [the regional gateway] rather than in dozens. This makes updates and policy changes simple and consistent.

Sizing: Each layer of the pipeline must be sized and scaled appropriately to handle the data volume from the layer below it.

Network Paths: Ensure reliable, secure network connectivity between your clusters and the regional gateway.

#3. Per Signal Pattern

This pattern involves creating separate, parallel pipelines for each telemetry signal type, i.e, instead of a single, unified OTel Collector fleet that processes all signals together, you deploy specialised fleets — one for traces, one for metrics, and one for logs.

Architecture with Agents & Routing

The OTel agents are configured to collect all signals as usual. At the first possible stage [either in the agent itself or in a simple first-layer gateway], the data is split. The OTel Collector’s routing processor is often used here.

• All traces are routed to the Trace Gateway fleet.

• All metrics are routed to the Metrics Gateway fleet.

• All logs are routed to the Logs Gateway fleet.

Each gateway fleet is configured and optimised only for its specific signal type, with its own set of processors, and exports to its corresponding observability backend like SigNoz.

Trade-Offs & Consideration

Independent Scalability: You can scale your logging fleet to handle huge volumes without over-provisioning your tracing or metrics pipelines.

Resource Optimisation: You can use CPU-optimised instances for your log collectors and memory-optimised instances for trace collectors, depending on load and necessity.

Higher Operational Overhead: You are now managing three or more separate collector fleets, each with its own configuration, deployment pipeline, and monitoring. Might get tiring!

Signal Correlation: It becomes more difficult to correlate signals at the collector level [e.g., using the spanmetrics processor to generate metrics from traces], as the data is already on separate paths.

How To Choose the Right Deployment?

The short answer is that there is no hard-and-fast rule for what is right. But we have put out a small guide that can help you understand some potential options you can explore.

If you have many clusters or regions that need unified telemetry, use the multi-cluster [control-plane] pattern. Designate one cluster as the central collector host, and configure each cluster’s agent/sidecar to export to it. This way, you get consistent processing [e.g. cross-cluster tail sampling] at the cost of cross-cluster links.

OR If different teams or customers must be isolated for privacy or regulatory compliance reasons (which are now getting stricter!), use a multi-tenant pipeline strategy. For example, tag data by team and have the collector route it to separate backends or processing paths. This limits the blast radius; one team/tenant’s misconfiguration won’t contaminate another’s data.

OR

When you need maximum ingestion throughput and uptime, deploy a load-balanced collector. Put collectors behind a robust L7 load balancer so you can autoscale instances on demand. This handles bursts by spreading the load and avoiding any single Collector becoming a bottleneck.

OR

If your metrics/trace/log volumes differ greatly, consider splitting pipelines by signal. As we mentioned above, run one collector for metrics [allowing many scraper replicas] and another for traces [optimised for tail sampling]. This lets you scale each pipeline to its workload without interference.

OR

For small deployments or strict budget constraints, start with a single Collector or node-level agents/sidecars to minimise infrastructure costs. As load grows or performance demands rise, move to more complex patterns: for example, add a gateway layer or switch to a load-balanced, multi-instance setup. Conversely, if ultra-low latency and resilience are paramount, an agent-and-gateway hybrid [per-node agents forwarding to central gateways] offers local buffering and global control.

Words of Wisdom from the Field

Here are some snippets with Sreekanth Chekuri, who is a Senior Software Engineer at SigNoz and also a contributor to OpenTelemetry. We hope some of these pointers will help guide you when designing the architecture for deploying your OTel collectors!

Maximising Performance and Throughput

• Embrace Batching for Efficiency: He points out that a simple change like batching your data [e.g., up to 25k entries] significantly improves throughput. This works because it reduces unnecessary system calls and processing overhead, letting your pipeline work smarter.

• The Power of Resource Scrutiny: Remember that resource requirements aren’t static. If your collector is doing heavy data transformation like parsing complex logs or extracting attributes, it will naturally need more CPU and memory. Always size your Collector based on the processing load, not just the ingestion rate.

Strategic Collector Deployment

• Split by Signal for Precision: For optimal resource allocation, Sreekanth recommends running separate collectors for different signals. This allows you to allocate memory and CPU precisely where needed, avoiding a single resource hog.

• Handle Traces with Care: Be mindful that tail-based sampling for traces is memory-intensive and requires specialised handling. If you mix this heavy operation with standard log or metric processing, it can impact the reliability of your entire system. Splitting these signals solves that problem.

Cautions

• Know Your Tools: While alternative data pipelines exist, he cautions against simply swapping out the OpenTelemetry Collector for tools like Vector. You risk losing the many powerful, built-in capabilities and standardised features that the OTel ecosystem provides.

• Watch Out for CPU Hogs: Some OTel processors, such as the transform processor, can be highly CPU-intensive. Use them judiciously, as they can significantly impact performance and scalability if overused in a high-throughput environment.

Thanks to Juraci for suggesting some edits to the initial version of this blog! Also, for his GitHub repository, which acted as a guidepost while I was learning about various patterns.

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

On another note, feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

Cheers.

eBPF is kind of like matcha - it has been around for a long time, yet it’s only within the past couple of years that it emerged as one of the latest trends and buzzwords in the industry.

I can’t explain how matcha became the world’s most popular drink (maybe another time 😉), but I will take today’s blog as an opportunity to tell you how eBPF has become a big deal for revolutionising observability at the kernel level, among many other dope stuff. Let’s look at the history of eBPF, how it works, what problems it solves, and why you – yes, you! – should start taking advantage of it today.

What is eBPF?

eBPF - or the extended Berkeley Packet Filter, as it was formally known - is the name of a kernel execution engine that runs a variety of new programs in a performant and safe sandbox in the kernel.

If the above definition flew right past your head, let me simplify it. It’s almost like putting JavaScript into the Linux kernel. JavaScript can run programs safely in a browser sandbox similar to eBPF in a kernel.

With eBPF, developers can execute custom programs [typically in a restricted C syntax] and load them at runtime in kernel space without the need to modify kernel source code or add additional modules.

Originally derived from the classic BPF used for packet filtering, eBPF greatly extends its scope beyond networking to any part of the system. Since eBPF has evolved way beyond packet filtering, it’s almost an understatement to refer to it as “extended”, and the acronym is not in active use anymore.

If you are interested in the evolution of eBPF, ideas and thoughts in the early days, take a look at the documentary below. This is also a great example of all the work that went behind the scenes to get code merged in a large codebase like Linux.

How does eBPF work?

By now, we have established that eBPF is a crazy technology. What happens BTS of how eBPF programs function is even more mind-blowing. Let me take a moment to explain it in-depth.

Step 1: Write an eBPF Program

Everything starts with writing the logic you want the kernel to execute. This is typically done in a restricted, C-like language. It’s not full C. For example, you can’t have unbounded loops or call just any function you want. The goal is to create a small, efficient piece of code that is guaranteed to run quickly and safely. Instead of calling standard libraries, eBPF programs use a special set of helper functions provided by the kernel to interact with the system, such as getting the current process ID or looking at network packet data.

Step 2: Compilation to Bytecode

Once the C code is written, it’s compiled into eBPF bytecode using a toolchain like Clang/LLVM. This bytecode is a universal, platform-independent instruction set that the Linux kernel can understand. This is similar to how Java code is compiled into bytecode to run on the Java Virtual Machine (JVM). In this case, the virtual machine is a secure one that lives inside the Linux kernel itself. The output is typically an ELF file containing the bytecode and definitions for any maps the program will use.

Step 3: Load the Program and Create Maps

This step is handled by a user-space application. This is a normal program you write in a language like Go, Rust, or Python that acts as the controller for your eBPF code. This application performs two key tasks:

• It reads the eBPF bytecode from the file created in Step 2.

• It uses a special system call (bpf()) to load that bytecode into the kernel.

At this stage, the user-space application also creates any eBPF maps the program needs. These maps are the crucial bridge for communication. They are key-value data structures that can be accessed by both the eBPF program in the kernel and the user-space application.

Step 4: Verification and JIT Compilation

This is the most critical step for ensuring safety and performance. As soon as the kernel receives the eBPF bytecode, it passes it to the Verifier. The verifier performs a static analysis of the code to prove that it is safe to run. It checks for infinite loops, out-of-bounds memory access, and illegal instructions. If the program fails verification, it is immediately rejected.

If the program passes verification, the kernel then uses a Just-In-Time (JIT) compiler to translate the eBPF bytecode into native machine code for the host CPU. This means the code doesn’t have to be interpreted, allowing it to run at nearly the same speed as natively compiled kernel code.

Step 5: Attach and Execute

After being loaded and verified, the eBPF program is in the kernel but is not yet active. The user-space application must explicitly attach it to a specific event hook. This could be:

• A network interface, to inspect incoming/outgoing packets [XDP or TC hooks].

• A system call entry/exit point [a tracepoint].

• The entry or exit of a function in the kernel or a user-space application [kprobe or uprobe].

Once attached, the kernel will automatically trigger the eBPF program every time that event occurs [Yes, eBPF is event-driven!]. The program runs, performs its task [like updating a counter in an eBPF map], and exits all within the kernel context, making it incredibly fast. Meanwhile, the user-space application can periodically read from the eBPF map to collect the data and present it to the user.

eBPF for Observability

Let’s understand how eBPF could be used for observability by checking how it provides visibility into each of the three pillars.

Metrics

eBPF can be used to collect highly granular performance metrics that are impossible to see from the user space. For example, you can attach an eBPF program to kernel functions to precisely measure TCP retransmits, disk I/O latency, or time spent scheduling processes.

Logs

While not a replacement for traditional application logs, eBPF can generate highly contextual event logs. For example, you can create a log every time a process opens a sensitive file, writes to a specific socket, or executes a new program, complete with the process ID and user ID. This provides a powerful audit trail for security and debugging.

Traces

eBPF can automatically trace requests between services without any code changes. By observing the send() and recv() system calls made by applications, eBPF-powered tools can stitch together a distributed trace, even across different programming languages. It can even trace encrypted traffic [like HTTPS] by hooking into the application’s memory before the data is encrypted.

Tracing File Opens with eBPF

Let’s put the above theory into practice. Here’s a small example of how we can count the number of file opens with eBPF. We are controlling the eBPF program via Python. Since we are using the BCC [BPF Compiler Collection] framework, which is a popular Python library for writing and loading eBPF programs, we should have it installed.

Here’s the program/ script for the eBPF program that traces the openat() syscall, and logs the process ID, process name, and file path each time a file is opened.

from bcc import BPF

# eBPF program that hooks into the openat syscall
bpf_code = “”“
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

struct data_t {
    u32 pid;
    char comm[TASK_COMM_LEN];
    char fname[256];
};

BPF_PERF_OUTPUT(events);
int trace_openat(struct pt_regs *ctx, int dfd, const char __user *filename, int flags) {
    struct data_t data = {};

    // Capture process ID and name
    data.pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&data.comm, sizeof(data.comm));

    // Capture file name
    bpf_probe_read_user(&data.fname, sizeof(data.fname), filename);

    // Send the data to user-space
    events.perf_submit(ctx, &data, sizeof(data));
    return 0;
}
“”“

# Load the eBPF program
b = BPF(text=bpf_code)

# Attach eBPF program to the openat syscall
b.attach_kprobe(event=”sys_openat”, fn_name=”trace_openat”)

# Function to print the output
def print_event(cpu, data, size):
    event = b[”events”].event(data)
    print(f”PID: {event.pid}, Process: {event.comm.decode(’utf-8’)}, File: {event.fname.decode(’utf-8’, ‘replace’)}”)

# Open a perf buffer to receive events from kernel space
b[”events”].open_perf_buffer(print_event)

# Continuously listen for events and print them
while True:
    b.perf_buffer_poll()

Execute the script with root privileges, as eBPF requires them to load programs into the kernel.

sudo python3 <name _of_file>

Let’s break down the code into its two main parts.

The eBPF Program [The C Code]

This is the logic that runs securely inside the kernel.

• struct data_t: We first define a C struct. This is the shape of the data we want to send from the kernel to our Python program. In our example, it holds the process ID, the command name, and the filename.

• BPF_PERF_OUTPUT(events) : This is a BCC macro that creates a high-performance communication channel called events. It allows us to efficiently send data from the kernel to user space without slowing the system down.

• int trace_open(struct pt_regs *ctx): This is our main eBPF function. It gets the current process ID [pid] and command name [comm] using eBPF helper functions [bpf_get_current_pid_tgid() and bpf_get_current_comm()].

• The most important part is bpf_probe_read_user_str(). The filename exists in the memory of the application making the system call, not in the kernel. This special helper function safely copies the filename string from the user’s application memory into our data. filename variable.

• Finally, events.perf_submit() pushes our completed data structure into the events perf buffer, making it available to our Python script.

The User-Space Controller [The Python Code]

This Python script loads and manages the eBPF program.

• b = BPF(text=bpf_program): This line is where the BCC magic happens. It takes our C code as a string, compiles it into eBPF bytecode, and loads it into the kernel. The kernel’s Verifier checks the bytecode to ensure it’s safe before allowing it to be loaded.

• b.attach_kprobe(...): This is the crucial step where we attach our trace_open C function to a kernel event. We use a kprobe [kernel probe] to hook into the kernel function that handles the openat system call. Now, every time any process on the system calls openat, our eBPF code will run first.

• b[”events”].open_perf_buffer(print_event): This tells our script to start listening to the events channel we created in the C code. For every piece of data that comes through, it will call our Python function print_event.

• while True: b.perf_buffer_poll(): This is the main event loop. The script sits here, efficiently waiting for data to arrive from the kernel. When data is available, it triggers the print_event callback to print the formatted output to your screen.

Once you run the script with root privileges, you will see output like this,

Although this is a very basic example, it gives a good insight into how eBPF programs work from code to monitoring calls. eBPF is no longer a niche technology, but something that is being widely adopted by orgs at various levels, revolutionising the tech industry — one matcha at a time. 🍵

💌 Hey there, it’s Elizabeth from SigNoz!

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

This piece is written by Anushka, PMM at SigNoz, on account of completion of 100 releases at SigNoz.

Also, feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing Slack community for the latest news!

Cheers.

We just shipped our 100th GitHub release.

You would think a milestone like this would feel like an arrival, a moment to look back and say, “Yay, we made it.”

But when I sat down with the team to understand how they felt, everyone said some version of the same thing - “We are just getting started.”

To understand what it actually takes and feels like to ship 100 releases, I talked to a few of my teammates from different junctures of the product story.

This is their story. Welcome to our journey of 100 releases.

A quick side note on methodology: My highly scientific process for selecting interviewees involved grabbing anyone who wasn’t in a meeting at the eleventh hour. While their stories are amazing, they are just a few of the many that make up our 100-release journey. A huge thank you to the entire team!

The first hire

Ashu was the third person at SigNoz, joining the two founders in May 2021 as a Growth Manager. His job was to let more developers know that SigNoz existed and that it was OpenTelemetry native. When he joined, SigNoz had around 600-700 GitHub stars.

The problem was that nobody was paying attention yet. Our Slack community had crickets. But Ashu believed in the founders and their genuine care to improve the experience of fellow developers. This was his push to keep going.

He kept writing content on dev.to, publishing on Hacker News, and explaining how to implement OpenTelemetry in Java applications when OpenTelemetry itself was brand new.

This soon became a habit. It eventually built up and gave us numbers to chase.

We’d trend on GitHub occasionally, articles would go viral, and when people visited the repo, they’d see the value and star it. The momentum kept attracting people. Soon, contributors started trickling in. Later, customers became contributors too.

Today, SigNoz has 24,200+ stars on GitHub (as of 5th Nov, 2025).

The first backend engineer

While Ashu was getting the word out, Vishal joined us in November 2021 as our first backend engineer, starting on the traces module. For us, OpenTelemetry was never a mere technology choice; it was in our DNA from day one. We learned from the OpenTelemetry community, raised issues upstream, and helped debug.

Community was always core to what we were building.

As the product gained traction, users started asking for a SaaS version. They didn’t want the hassle of setting up and maintaining the open-source infrastructure. The team decided to launch SigNoz Cloud during a workation in Goa in December 2023, with the idea of just doing a one-week soft launch to test for issues and then rolling it back. In the first week, we got four signups. We never rolled it back.

Vishal remembers the chaos of that launch vividly. The team was at the beach when Nitya got a message about the sign-up flow being broken and him running back (literally) to the room to fix it. That workation launch, which they thought was just a pilot, became the real thing.

That’s when Vishal’s role shifted. He went from being a backend engineer to a product manager, spending his days on calls with customers, debugging issues, and doing manual setups. The work wasn’t glamorous. There were sleepless nights and pressure from large companies to prioritize their features.

But that messy, tedious work had to get done. Customers like Kiwi helped shape the product through their feedback and pull requests, pushing us to build for actual scale, not just theoretical scale.

Bringing logs to the product

When Nitya joined in April 2022, we had around 40-50 weekly active users. His first task was to bring logs into the product. We had traces and metrics, but logs were the missing piece.

Just a month after he joined, during a team workation in May 2022, a conversation over lunch set the tone for how we build. Pranay, CEO of SigNoz, asked Nitya what he was working on.

“Logs. Building out the schema,” Nitya said.

“How much data are you testing on?”

“One million.”

“No, no. Test it on one billion.”

Nitya spent the next week running schemas against a billion log lines, again and again. That ambition became our standard. When we released the first logs benchmark, it caught fire on Hacker News. The traffic exploded, and with it, more customers and more data.

But as customers grew, so did the complexity. Nitya manually migrated hundreds of customers to new schemas over six months. We couldn’t afford to lose their data or break their workflows.

What worked at 50 users didn’t work at 500. Every new customer taught us something, and every incident made us more careful.

The first frontend lead

We had a functional product, but it was far from pretty. That’s where Yunus came in, joining in August 2023 as our first frontend engineer.

Frontend engineers typically don’t gravitate toward dev tools, which are often built for SREs and backend folks. But Yunus wanted to build a culture where engineers think beyond their specific skillset.

His philosophy was simple: “You are not a frontend engineer or a backend engineer. You are a software engineer.” He wanted everyone to understand the ‘why’ before getting into the ‘how.’

This thinking was crucial.

When Yunus joined, we were moving fast, but we weren’t always thinking in systems. He focused on building processes to make our frontend more stable and predictable because people make mistakes, but good systems can prevent those mistakes from breaking things.

Joined at the 34th release

Vikrant joined as a frontend engineer in January 2024. The frontend wasn’t stable; fixing one thing often broke another. He felt disconnected from the full picture, but an opportunity came up that changed everything. The provisioning flow for new sign-ups was constantly breaking, and Vikrant had already expressed interest in learning backend.

So he made the shift. He had a one-month crash course, taking over ownership from Pandey. It was a ticking bomb. Either he figured it out, or the sign-up flow stayed broken.

And damn, he did figure it out.

Later, the community asked for something ambitious - loading traces with millions of spans. We didn’t want to build a makeshift solution. We wanted to solve it permanently. After two months of intense work, we could load a million spans on a single screen.

Then came the push for provisioning v1.0. Our deadline was the Tuesday platform retrospective at 6:30 PM. As the clock ticked, the call got pushed to 7:00, then 7:15, then 7:30. We finally deployed to production, tested it, and joined the call right after. The entire team stood (well, practically sat in front of their laptops) together until v1.0 was stable.

The next challenge was to improve our SQL database schemas. For three months, we had to break down our entire infrastructure and rebuild it. Every Wednesday noon, Vikrant, Pandey, and Nitya would send each other memes, about the known pattern - a release would go out, and by 12:30 PM, a bug would be reported. Every single time.

But as Vikrant puts it, “If you’re tired, do it tired.” And when you have the back of your team, it does get easy.

Building processes that scale

Pandey joined in February 2024, when SigNoz was just a bunch of people executing. There were no pods, no real structure. His first task was to stabilize data ingestion. Every other day, a customer complained they couldn’t ingest data. It was a race against time.

After stabilizing ingestion, he turned to a bigger question - What does it take to run a high-performing team?

He laid the foundation for the platform pod, starting with just him and Vikrant. They implemented sprints, retrospectives, and reporting - a process that ran for eight months with just the two of them before being adopted company wide.

Introducing structure wasn’t easy. It led to internal friction and disagreements on how things should be done.

But as Pandey notes, the culture being set today is what new people will embody. That’s how the baton gets passed.

Eight months in

Piyush joined in March 2025, and eight months later, he says it already “feels like forever” in the best way. At previous companies, the push was to ship features fast, any way possible. Here, he found the time and space to do it the right way the first time.

Working with Nitya on logs, he had to learn a new way of collaborating remotely. It took a few weeks to align on the thinking behind testing certain things, but over time, the context builds.

Now, Piyush is in the position Nitya was in when he first joined. He is responsible for making logs better and working on complex features like cloud integrations. He’s also exploring JSON logs, which he believes will boom fast and make many data pipelines redundant.

And well, his story connects everything. Someone is still at their Day 1, even on our 100th release.

Day 1, again

Every Wednesday, I look forward to the release. And it’s not just because I write the changelog. It’s become my favorite part of the week. As a marketer, I couldn’t ask for a better way to stay connected to what we’re actually building.

Yet here I am, writing a nostalgic story instead of a technical, feature-tracking blog. Because at the end of the day, there are humans building these sophisticated features, and their stories are worth hearing. at least sometimes, if not often.

The Day 1 feeling isn’t restricted to engineering. A few days ago, we launched our first-ever mascot. We ran our first integrated campaign. Every one of these feels like a beginning.

It’s Ashu pushing through the crickets. Vishal doing the messy, unglamorous work. Nitya testing for a billion when a million seemed like enough. Yunus building systems, Vikrant doing it tired, and Pandey introducing structure when velocity felt more important.

These stories are limited to a few, but they echo the team’s sentiment at large.

Here’s to the next 100. We’re just getting started!

This spirit extends beyond our internal team. Our community has been with us every step of the way, which is why this past July, we were thrilled to launch the SigNoz Community Advocate Program. It’s our way of recognizing the passionate developers who help others succeed with observability.

Shout-out to our inaugural advocates for their incredible contributions: Mathew Gilham, Matti De Grauwe, Kieran Pilkington, Gil Felot and Nicolas Lamirault.

And in a moment of perfect serendipity, just as I was about to create the PR for this post, we welcomed our 500th paid customer.

This newsletter is an honest attempt to talk about all things - observability, OpenTelemetry, open-source and the engineering in between! We at SigNoz are a bunch of observability fanatics obsessed with OpenTelemetry and open-source, and we reckon it’s important to share what we know. If this passes your vibe-check, we’d be pleased if you’d subscribe. We’ll make it worth your while.

Subscribe now

On another note, feel free to check out our blogs and docs here. Our GitHub is over here, and while you are at it, we’d appreciate it if you sent a star ⭐ our way. You’re also welcome to join the conversation in our growing  Slack community for the latest news!

Cheers.

In any type of software that involves the movement of data or information, there is a pressing need to make the passage of data secure. One way of achieving this is by authentication. You must have experience authenticating API calls or other data streams.

Gemini defines authentication as the process of verifying that a user, device, or system is who or what it claims to be, typically by using credentials like a username and password. When I was first learning about authenticating systems, I related it to the term authenticity, which is closely related to trustworthiness, that is, can the source of incoming data or request be trusted enough to accept it? You can stick with a definition or build an idea based on what works best for you. :)

In modern systems, where even a small mishap can wreak havoc and you might wake up to a $$$ bill the next day, we should do whatever is within our capacity to secure our systems.

That’s why this week, I want to talk about something crucial but often overlooked: Authentication for your OpenTelemetry Collectors. These collectors are the busy data hubs of your observability pipeline, handling huge amounts of information every moment. Securing them is non-negotiable, and also a perfect use case for strong authentication.

Authentication in OpenTelemetry

Firstly, OpenTelemetry on its own doesn’t define an authentication protocol or an auth model. OpenTelemetry's primary aim was to define a standard data model (like for metrics and logs) and a transport protocol (OTLP). It leaves us the flexibility to work with any authentication scheme, based on our collector pipeline and the backend we are using.

In a Collector pipeline, data has one point of entry, the receivers and one point of exit, the exporters. Authentication is critical at both of these points.

Authenticating Incoming Traffic

As we saw before, the receiver is the point of entry for data traffic, hence it’s crucial to examine if the data is coming from a reliable source. We achieve this by auth extensions. You can read more about extensions here.

In this scenario, we will configure our Collector to only accept requests that include a valid secret token in their Authorization: Bearer <token> header. This is a three-step process in your Collector’s config.yaml file.

Step 1: Define the Authenticator in extensions

First, we define our authentication method. We’ll use the built-in bearerauth authentication and provide it with a list of valid tokens.

extensions:
   bearerauth:
   # This defines a list of valid secret tokens the collector will accept.
   # Any client request must present one of these tokens to be authenticated.
       tokens:
          “${CLIENT_A_TOKEN}”
          “${CLIENT_B_TOKEN}”

Just registering the authentication here under the extension doesn’t enforce it. It gets enforced when it’s applied to a receiver, as shown in the next section.

⚠️ Important Security Note!

Never hardcode secrets directly in your configuration file. The ${...} syntax tells the Collector to load the token from an environment variable. You should inject these variables securely using a tool like Kubernetes Secrets or Docker Secrets.

Step 2: Apply the Authenticator to a Receiver

Next, we tell our otlpreceiver that it must use the authenticator we just defined. We do this by adding an auth setting within the receiver’s configuration.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: “0.0.0.0:4317”
        auth:
          authenticator: bearerauth   # use the bearerauth extension
      http:
        endpoint: “0.0.0.0:4318”
        auth:
          authenticator: bearerauth   # same auth on HTTP

Step 3: Enable the Extension in the Service Block

Finally, the extension must be activated for the Collector by listing it in the service section. This is the entire flow of code.

service:
  extensions: [bearerauth]  # This activates the bearerauth extension
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]  

With this configuration in place, your Collector’s incoming traffic is now secure. Any request arriving at the OTLP receiver without a valid token will be rejected, ensuring only your trusted applications can send data into your observability pipeline.

There are other ways of authentication as well, like — basicauth, oidc, etc, depending on your particular use case. Now, let’s see how we deal with outgoing traffic.

Securing Outgoing Traffic

Exporters are the exit point for data leaving the collector. The next destination for your data is most likely an observability backend like SigNoz, and the collector often needs to authenticate itself to prove it has permission to send that data. Now, there are two ways to do this.

The easiest way is to add a headers section directly to your exporter’s configuration in your config.yaml. This tells the exporter to attach the specified headers (containing your secret key) to every outgoing request. The code is shown below,

exporters:
  otlp:
    endpoint: “ingest.us.signoz.cloud:443”
    headers:
      # This header authenticates the Collector with the SigNoz backend
      signoz-ingestion-key: “${SIGNOZ_API_KEY}” # as env var

For more complex authentication, you can follow the same sequence of steps as we did for receivers above. That is, Step 1 - Define the Authenticator in Extensions, AND Step 2: Apply the Authenticator to an Exporter. At the end, we register the extension under exporters. Here’s the entire code sample.

extensions:
  sigv4auth: ## a specialized authenticator for users of AWS.
    region: “us-east-1”
    service: “aoss”  

exporters:
  otlp:
    endpoint: “ingest.us.signoz.cloud:443”
    headers:
      signoz-ingestion-key: “${SIGNOZ_API_KEY}”

  otlphttp/aws:
    endpoint: “https://my-opensearch-domain.us-east1.aoss.amazonaws.com”
    auth:
      authenticator: sigv4auth


 service:
   extensions: [sigv4auth]  
   pipelines:
     traces:
       receivers: [otlp]
       processors: [batch]
       exporters: [otlp]  

In summary, for most backends that use a simple API key, the static headers setting is all you need. For more complex scenarios involving cloud provider IAM roles or OAuth2, we use Collector’s auth extensions.

What’s next?

Now that we’ve laid a foundation for securing data flowing into your OpenTelemetry collectors, you can get hands-on and experiment with different authentication methods to get a well-rounded idea. To read more on OpenTelemetry collectors and their various parts, this is a good read.

Next week, I’ll be back with another deep-dive, and until then, adeiu! 👋

The short version: building, debugging, and improving AI agents in production gets messy fast. There’s multiple competing standards for default libraries for LLM observability. And many such libraries like OpenInference which claim to be based on OpenTelemetry don’t strictly adhere to it’s conventions. This introduces problems for users who are trying to get better observability across their stack.

Here’s a write-up of what we covered and what I think it means for anyone shipping LLM features into real products. Feel free to watch the complete video

The Problem Emerges in Prod

Pranav and I go way back to our YC days in 2021, and it’s always interesting to see how our paths have evolved. Chatwoot has built something really compelling - an open-source customer support platform that unifies conversations across every channel you can imagine: live chat, email, WhatsApp, social media, you name it. All in a single dashboard.

But here’s where it gets interesting. They’ve built an AI agent called “Captain” that can work across all these channels. You build the logic once, and it can handle support queries whether they come through email, live chat, or WhatsApp. Pretty neat, right?

The problem started showing up in production in the most unexpected ways. Sometimes their AI would randomly respond in Spanish when it absolutely shouldn’t. Other times, responses just weren’t quite right, and they had no visibility into why.

The Quest for LLM Observability

This is where Pranav’s journey into LLM observability began, it mirrors what I’ve been seeing across many companies building LLM applications. You need to understand:

• What documents were retrieved for a RAG query?

• Which tool calls were made?

• What was the exact input and output at each step?

• Why did the AI make certain decisions?

Without this visibility, you’re essentially flying blind in production.

The Standards Problem

Here’s where things get really interesting, and frankly, frustrating. Pranav explored several solutions:

OpenAI’s native tracing looked promising with rich, detailed traces showing guardrails, agent flows, and tool calls. But it’s tightly coupled to OpenAI’s agent framework. Also, it only provides traces as an atomic unit. If you want to filter spans based on attributes or just examine specific spans directly, you can’t do that.

New Relic was easy to integrate since they already use it, and it supports OpenTelemetry. But the UI required clicking through 5-6 layers just to see relevant information. Not ideal when you’re trying to debug production issues.

Phoenix caught their attention because it follows the OpenInference standard, which provides much richer, AI-specific span types. You can easily filter for just LLM calls, tool calls, or agent spans. The traces are beautiful and informative.

But here’s the kicker: Chatwoot is primarily a Ruby on Rails shop, and guess what? No Ruby SDK for OpenInference. Moreover, Phoenix doesn’t completely adhere to OTel semantic conventions, so if you send it telemetry data directly via OpenTelemetry, it doesn’t recognize the type of spans, etc.

As shown in the example above, Phoenix doesn’t shows data sent with OpenTelemetry span kinds as unknown.

The OpenTelemetry vs OpenInference Divide

This is where the conversation got really technical and revealed a fundamental industry problem. There are essentially two standards emerging:

OpenTelemetry is the industry standard. It has libraries for every language, it’s production-ready, and it’s widely adopted. But it was built for traditional applications, not AI workflows. It only supports basic span types: internal, server, client, producer, consumer. That’s it.

OpenInference was created specifically for AI applications. It has rich span types like LLM, tool, chain, embedding, agent, etc. You can easily query for “show me all the LLM calls” or “what were all the tool executions.” But it’s newer, has limited language support, and isn’t as widely adopted.

The tragic part? OpenInference claims to be “OpenTelemetry compatible,” but as Pranav discovered, that compatibility is shallow. You can send OpenTelemetry format data to Phoenix, but it doesn’t recognize the AI-specific semantics and just shows everything as “unknown” spans.

The Ruby Problem Makes It Worse

For teams using languages like Ruby that don’t have direct OpenInference SDK support, this becomes even more challenging. Pranav had to choose between:

1. Building an SDK from scratch for Ruby

2. Using OpenTelemetry and losing AI-specific insights

3. Switching to a different language stack just for AI observability (way tougher)

None of these are great options.

Why we (still) bias to OpenTelemetry

At SigNoz we’re all-in on OpenTelemetry. One reason: OTel’s consistency enables out-of-the-box experiences across your whole stack. Example: we can auto-surface external API usage and performance based on span kinds and attributes. When parts of the app send telemetry via non-OTel conventions, those views degrade.

Chatwoot lands similarly: their entire product already emits OTel. Pulling in a second telemetry standard just for LLMs fragments the picture and complicates how they go about observability. This also silos their observability into different products which makes it difficult to solves issues when they occur.

Takeaways for builders

• Pick one telemetry backbone - If most of your app is OTel, prefer staying OTel-native for LLMs too, even if it means adding richer attributes until GenAI conventions catch up.

• LLM specific libraries - Even if you have to use LLM specific libraries like OpenInference, try to keep your usage as close to OpenTelemetry as possible so that you are aware what non-OTel attributes you are using which may break things.

• Follow OTel GenAI working group - There is active work happening in OTel Gen AI working group. Follow the work happening there and do share your use cases so that the standards which OpenTelemetry builds are able to cater to most common use cases.

As the LLM space is still evolving rapidly, we as a community need to share our voices so that the standards are robust.

What we’re doing at SigNoz

We’re continuing to invest in OpenTelemetry-native LLM observability so teams don’t have to choose between stability and clarity. Concretely, that means:

• Clear dashboards and traces when LLM calls are modeled using OTel spans/attributes. You can find examples and dashboards in our LLM observability docs. Though we have also use LLM specific libraries like OpenInference in our docs (as they are still the easiest way for ppl to get started), we have kept the dashboards as close to OTel standards as possible. We also plan to actively update this as OTel GenAI semantic conventions become more mature.

• Guidance and examples for popular frameworks (LangChain, LlamaIndex, etc.) on emitting OTel-friendly telemetry.

• Build features leveraging OpenTelemetry semantic conventions so that you get great out-of-box experience in SigNoz and adhere to thoughtful defaults that keep your services, DBs, queues, and LLM agents—in one coherent picture.

If you’re wrestling with these trade-offs, we’d love to hear what’s breaking for you and what “rich semantics” you actually use day-to-day.

What next?

Huge thanks to Pranav for going deep, especially from the Ruby perspective. If you’re shipping AI features and care about operability, add your voice: push for richer GenAI semantics in OpenTelemetry, and share real traces (sanitized) that show what you need to see.

In this practical guide, we will walk you through setting up observability for your Langchain application with OpenTelemetry

, the open-source standard for generating telemetry data. We'll instrument a demo trip planner agent and show you how to visualize traces, token usage, and tool performance in SigNoz.

The trip planner agent helps users plan their travel itinerary by combining LLM reasoning with external services like flight ticket search, weather APIs, hotel booking engines, and nearby activity recommendations. By instrumenting it with OpenTelemetry, you can trace every step of the planning process, measure latency at each stage, and quickly debug issues that impact the user experience.

Why LangChain Observability Matters

LangChain agents are essentially reasoning loops: the LLM takes user input, decides which tools to call, processes their results, and iterates until it arrives at an answer. In a trip planner agent app, this might look like:

• Calling a flights API to check availability.

• Fetching hotel recommendations from a booking API.

• Looking up weather forecasts to suggest the best travel window.

• Stitching everything together into a coherent itinerary.

This chain of reasoning is amazing when it works but if one tool call fails, takes too long, or returns garbage, the whole experience collapses. Without observability, you won’t know whether the problem was:

• A slow external API call.

• An LLM misunderstanding the tool response.

• The reasoning loop going in circles.

Instrumentation with OpenTelemetry makes all of this visible from under the hood.

How OpenTelemetry and SigNoz can help

What is OpenTelemetry?

OpenTelemetry (OTel) is an open-source observability framework that provides a unified way to collect telemetry data—traces, metrics, and logs—from across your application stack. It’s a CNCF project with support for multiple programming languages and a wide range of integrations. The beauty of OTel is that you instrument your code once, and you can send that data to any observability backend you choose without vendor lock-in.

For LangChain-based agents, this means you can capture detailed performance and error data for each stage of the reasoning process: LLM calls, tool invocations (like flights, hotels, weather, and activity search), and the orchestration logic that stitches them together. Instead of treating your agent as a black box, you get fine-grained visibility into exactly how requests flow through your system.

What is SigNoz?

SigNoz is an all-in-one observability platform built on top of OpenTelemetry. It provides a rich UI to visualize traces, monitor performance metrics, and set alerts all in real time. With SigNoz, you can drill into slow external API calls, trace a single trip planning request end-to-end, or quickly identify where your LangChain agent might be looping or failing.

By pairing OpenTelemetry’s standardized data collection with SigNoz’s powerful analysis tools, you get a complete observability stack tailored for modern, distributed, and AI-driven applications.

To demonstrate how OpenTelemetry and SigNoz work together in practice, we’ll walk through a demo trip planner agent built on LangChain. The agent uses flight search, hotel booking, weather APIs, and nearby activity lookup to craft travel itineraries, and with observability enabled, you can see every step of the process in action.

Building the Example App: A LangChain Trip Planner Agent

To make this guide more concrete, we’ve built a trip planner agent powered by LangChain, OpenTelemetry, and SigNoz. The idea is simple: the user specifies a start location, destination, and check-in/check-out dates, and the agent generates a personalized travel itinerary.

The itinerary includes:

• Flight details for departure and return.

• Hotel booking options covering the entire stay.

• Weather forecasts for the chosen dates.

• Nearby activities to explore at the destination.

Under the hood, the app uses LangChain’s agent framework to orchestrate multiple tool calls: one for flight tickets, one for hotels, one for weather, and one for activities. The LLM reasons over the responses from these tools and stitches them together into a coherent itinerary.

With OpenTelemetry instrumentation baked in, every tool invocation and LLM call is traced and sent to SigNoz, providing a complete picture of the app’s performance and behavior: whether a flight API is slow, a hotel lookup fails, or the agent loops unnecessarily.

To make it more interactive, the trip planner also includes a chatbot feature. Users can ask follow-up questions like “Can you find vegetarian-friendly restaurants near my hotel?” or “What’s the best day trip outside the city?” These conversations are also traced, helping developers see how the agent performs during exploratory dialogue.

Try the Trip Planner Agent Yourself

Want to explore the LangChain Trip Planner in action? Clone the repo, install dependencies, and follow the setup steps in the README to start sending traces from your local app to SigNoz.

git clone https://github.com/SigNoz/langchain-monitoring-demo.git

After cloning the repo, you can run the agent locally and start exploring and creatign travel plans. The README provides step‑by‑step guidance for setting up the demo. If you’d rather instrument your own LangChain app, continue to the next section for detailed instructions on integrating OpenTelemetry and SigNoz.

Instrument your LangChain application

Prerequisites

• A Python application using Python 3.8+

• LangChain integrated into your app

• Basic understanding of AI Agents and tool calling workflow

• A SigNoz Cloud account with an active ingestion key

• pip installed for managing Python packages

• Internet access to send telemetry data to SigNoz Cloud

• (Optional but recommended) A Python virtual environment to isolate dependencies

To capture detailed telemetry from LangChain without modifying your core application logic, we will use OpenInference, a community-driven standard designed to make observability in AI applications easier. It provides pre-built instrumentation for popular frameworks like LangChain, and it’s built on top of the trusted OpenTelemetry ecosystem. This allows you to trace your LangChain application with minimal configuration.

Check out detailed instructions on how to set up OpenInference instrumentation in your LangChain application over here.

Step 1: Install OpenInference and OpenTelemetry related packages

pip install openinference-instrumentation-langchain \
opentelemetry-exporter-otlp \
opentelemetry-sdk

Step 2: Import the necessary modules in your Python application

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from openinference.instrumentation.langchain import LangChainInstrumentor

Step 3: Set up the OpenTelemetry Tracer Provider to send traces directly to SigNoz Cloud

resource = Resource.create({"service.name": "<service_name>"})
provider = TracerProvider(resource=resource)
span_exporter = OTLPSpanExporter(
    endpoint="https://ingest.<region>.signoz.cloud:443/v1/traces",
    headers={"signoz-ingestion-key": "<your-ingestion-key>"},
)
provider.add_span_processor(BatchSpanProcessor(span_exporter))

• <service_name> is the name of your service

• Set the <region> to match your SigNoz Cloud region

• Replace <your-ingestion-key> with your SigNoz ingestion key

Step 4: Instrument LangChain using OpenInference

Use the LangChainInstrumentor from OpenInference to automatically trace LangChain operations with your OpenTelemetry setup:

LangChainInstrumentor().instrument()

📌 Important: Place this code at the start of your application logic — before any LangChain functions are called or used — to ensure telemetry is correctly captured.

Your LangChain commands should now automatically emit traces, spans, and attributes.

Finally, you should be able to view this data in Signoz Cloud under the traces tab:

When you click on a trace ID in SigNoz, you'll see a detailed view of the trace, including all associated spans, along with their events and attributes:

Making Sense of Your Telemetry Data

Once telemetry is enabled in our LangChain trip planner agent, we start seeing detailed traces for each major step in the reasoning workflow. With LangGraph integration, these traces are neatly structured, showing how the agent loop orchestrates model calls and tool invocations. Here are three example spans you’ll encounter:

LangGraph (root span)

The overarching span represents the full request lifecycle of the trip planner agent. From the moment a user asks for a travel itinerary, every downstream operation: LLM reasoning, tool calls, and response generation is captured inside this parent span.

This view makes it clear how long the entire request took. On the right panel, you can explore input values like the initial user query, making it easy to trace back how the request was interpreted at the start.

Agent span

Nested inside the LangGraph span is the agent span, which captures the LLM’s reasoning steps. This includes the decision-making process: when to call a tool, how to interpret the results, and whether the loop should continue or terminate.

Here, you can see the call_model → RunnableSequence → ChatOpenAI flow. Each step shows its latency, and the trace reveals exactly which prompts and tool inputs the agent generated. This makes it much easier to debug cases where the model loops too long or misuses a tool.

Tool call spans

Next, you’ll see spans for each tool invocation: flights, hotels, weather, and activities. These are especially valuable for diagnosing external API performance.

For example:

• get_flight_tickets → duration ~13ms

• get_hotel_bookings → duration ~25ms

• get_weather → duration ~16ms

• get_activities → duration ~11ms

Closing Agent span

After the tool calls, the workflow enters a closing agent span, where the LLM takes all tool outputs (flights, hotels, weather, activities) and composes the final travel itinerary.

This is where the agent stitches together structured API responses into a user-friendly itinerary. By inspecting this span, you can:

• Review the exact prompt the LLM used to summarize tool outputs.

• Measure how much time the final response generation takes.

• Verify the final message content before it’s returned to the user.

With all this data, you can answer critical performance questions about your trip planner agent:

• Where is the time going? Is most of the latency in the agent’s reasoning, external API calls, or final response assembly?

• Which tools are slowest? For instance, if get_hotel_bookings consistently takes longer, you might need caching or a faster API provider.

• Is the agent reasoning efficiently? If the initial or closing agent spans dominate total latency, you may need to optimize prompts or reduce unnecessary loops.

Instead of guessing why an itinerary takes 20+ seconds to generate, SigNoz gives you a connected, end-to-end view of each request turning your LangChain workflow from a black box into a fully observable system.

Visualizing Data in SigNoz with Dashboards

Once your LangChain trip planner agent is instrumented with OpenTelemetry, SigNoz gives you teh ability to create rich dashboards to explore the emitted telemetry data. Built-in filters and span attributes make it easy to drill down into agent reasoning latency, tool performance, or model usage. This gives you a real-time pulse on how your application is performing end-to-end.

Here are some of the most insightful panels we built using the traces from our instrumented trip planner workflow:

p95 Duration for Agent call_model

This panel shows the 95th percentile latency for the LLM calls made by the agent. Since LLM generation is often the longest-running step, tracking p95 duration helps you identify worst-case response times and tune prompts or model choices to improve user experience.

Tool Call Distribution

This panel visualizes how often different tools—flights, hotels, weather, and activities—are invoked across all trip planning sessions. It gives you a clear sense of workload distribution: for example, hotel searches may dominate requests while activity lookups are used less frequently. Understanding this helps with capacity planning and prioritizing optimizations.

Input and Output Token Usage

This panel tracks the total number of input and output tokens processed by the LLM over time. Input tokens include the user query and tool outputs passed to the model, while output tokens are the generated itineraries or chatbot replies. Monitoring this helps you manage costs, optimize prompt length, and detect patterns in response verbosity.

p95 Duration of Each Tool Call

This panel breaks down the latency of each tool: get_flight_tickets, get_hotel_bookings, get_weather, and get_activities. By tracking the 95th percentile duration, you can quickly spot which external API is the slowest under peak load and decide whether caching, retries, or provider changes are needed.

LLM Model Distribution

If your app is configured to use multiple LLMs, this panel shows the distribution of model usage. It’s useful for analyzing trade-offs between speed, quality, and cost. For example, you might run most queries on a smaller, cheaper model but switch to a larger model for complex multi-step itineraries.

With these dashboards in place, you can move beyond anecdotal debugging and gain data-driven insights into your LangChain agent. Whether it’s latency hotspots, tool reliability, or token usage trends, SigNoz provides the observability foundation you need to scale AI-driven trip planning with confidence.

Wrapping it Up

Building LangChain agents like a trip planner is exciting. There’s something magical about watching an AI plan your flights, hotels, activities, and even answer follow-up questions in natural language. But that magic only lasts if the app stays fast, reliable, and trustworthy. To make that happen, you need a clear view of what’s going on under the hood.

By pairing OpenTelemetry’s vendor-neutral instrumentation with SigNoz’s powerful observability platform, you can follow every step of your LangChain workflow from agent reasoning to tool calls and final response generation. With this visibility, debugging becomes faster, performance tuning becomes data-driven, and your users get consistently great experiences.

In AI-powered apps, guesswork is the enemy. Observability is how you replace it with clarity, and that’s how you build LangChain systems you can trust.

Coming Next: Using SigNoz to monitor a LangChain agent that queries SigNoz MCP

In the next part of this series, we’ll go deeper into observability by looking at a LangChain agent that integrates with an MCP (Model Context Protocol) server. This opens up richer interactions, but also more moving parts where observability becomes even more critical.

Full-Circle Observability: Using SigNoz to monitor a LangChain agent that queries SigNoz MCP

That foundation gave us insights into latency hotspots, tool failures, and agent reasoning loops which are all critical for ensuring a reliable user experience in production AI apps.

In this second part, we’ll take observability a step further by introducing MCP (Model Context Protocol) servers into the mix. Specifically, we’ll look at a LangChain agent integrated with a SigNoz MCP server, which allows the agent to directly query logs, metrics, and traces from a connected SigNoz instance.

This means that instead of just sending observability data to SigNoz, the agent itself can consume and reason over observability data in real time.

We’ll walk through how to set up a SigNoz MCP agent with LangChain, instrument it with OpenTelemetry, and explore the kinds of insights it can surface when observability data becomes part of the agent’s context.

Building the Example App: A LangChain SigNoz MCP Agent

In this part, we’ll demonstrate a LangChain agent that integrates with a SigNoz MCP (Model Context Protocol) server. The goal of this app is to make observability data: logs, metrics, and traces queryable through natural language.

Users can interact with the agent just like a chatbot, asking operational and performance-related questions such as:

• “What are all the active services in the last 5 hours”

• “Which service has the highest error rate this week?”

• “Show me the logs generated in the last 1 hour.”

Behind the scenes, the LangChain agent communicates with the SigNoz MCP server, which exposes endpoints for querying observability data. The agent decides which endpoint to call (logs, metrics, or traces), retrieves the relevant data, and then uses the LLM to generate a clear, human-readable summary for the user.

All of this activity is itself instrumented with OpenTelemetry. Each agent reasoning step, MCP server call, and final response generation is captured as spans and sent to SigNoz.

Try the SigNoz MCP Agent Yourself

Want to explore the LangChain SigNoz MCP Agent in action? Clone the repo, install dependencies, and follow the setup steps in the README to connect the agent with your own SigNoz instance.

git clone https://github.com/SigNoz/signoz-mcp-demo.git

After cloning the repo, you can run the agent locally and start asking natural language questions about your observability data—logs, metrics, and traces—from SigNoz.

The README provides step-by-step guidance for configuring the MCP server connection and running the demo.

Making Sense of Your Telemetry Data

Once telemetry is enabled for the SigNoz MCP agent, traces clearly show how a user request flows through LangGraph, the agent’s reasoning, the MCP tool invocation, and the final response assembly. In a typical run, you’ll see this shape:

query_endpoint (root span)

This top-level span represents the entire MCP query lifecycle from the user’s natural-language prompt to the final summarized answer. It’s your single place to track end-to-end latency for an observability question, containing the LangGraph from our previous blog.

Use the right-hand attributes to confirm request metadata and inspect the input/output payloads that kicked off the flow.

Initial Agent span (planning & tool selection)

Nested under the root is the first agent span. Here the LLM interprets the user’s question and decides which MCP capability to call (logs, metrics, or traces). In your example, the chain shows:

call_model → RunnableSequence → ChatOpenAI → should_continue

This span’s duration is a good proxy for prompt complexity and reasoning cost before any external call happens.

What to look for:

• Long initial agent spans can indicate heavy prompts or unnecessary loops.

• Inputs/outputs show the exact messages the model which is great for debugging misinterpretations.

MCP tool span (data retrieval from SigNoz)

Next comes the MCP tool call. For example, a fetch_services operation hitting the SigNoz MCP server to retrieve services, metrics, logs, or traces. This is the place to diagnose backend/query latency and payload size issues.

Closing Agent span (reasoning & final answer)

After the tool response, the closing agent span composes the final answer: it parses MCP results, filters/sorts/aggregates as needed, and generates a clean natural-language summary.

What to look for:

• Long closing spans usually mean large MCP payloads being summarized (token pressure) or extra follow-up reasoning.

• Inspect the prompt the agent used for summarization to ensure it’s concise and grounded in the retrieved data.

Handling Errors with Full Context

Errors are inevitable in AI agents: API limits, bad tool responses, or timeouts. Without observability, it’s hard to know what failed and where.

With SigNoz, errors are tied to specific spans in the trace, so you can see:

• Which component failed (agent reasoning, tool call, or response synthesis).

• What the error was (rate limit, timeout, schema mismatch, etc.).

• When in the request it happened.

In this example, a RateLimitError from OpenAI is clearly flagged in the closing agent span. The trace shows the error message, stack trace, and context all in one place.

Instead of guessing, you know exactly what broke, where, and why, making debugging much faster and safer in production.

What you can answer with these traces:

• Where is the latency?

  Is time spent in planning (initial agent), in the MCP query (tool span), or in summarization (closing agent)?

• Are queries efficient?

  Tool spans reveal slow MCP endpoints and overly broad filters. Tighten time windows or add constraints.

• Is the model working too hard?

  Long agent spans (before or after tools) suggest prompt bloat, unnecessary loops, or passing too much raw data back to the LLM.

• Is the workflow stable?

  Use span status codes and events to spot intermittent errors (schema mismatches, token limits, provider hiccups).

With this structure, SigNoz turns the MCP-powered workflow from a black box into a fully traceable conversation: user prompt → agent planning → MCP tool call → agent summary. That visibility makes debugging faster, optimization data-driven, and your observability assistant consistently reliable.

Visualizing Data in SigNoz with Dashboards

Once your LangChain SigNoz MCP agent is instrumented with OpenTelemetry, SigNoz gives you the ability to create rich dashboards to explore emitted telemetry data. Built-in filters and span attributes make it easy to drill down into agent reasoning latency, MCP query performance, error patterns, and model usage. This provides a real-time pulse on how your observability agent itself is performing end-to-end.

Here are some insightful panels we built using the traces from our instrumented MCP workflow:

p95 Duration for Agent call_model

This panel shows the 95th percentile latency for LLM calls made by the agent. Since generation often dominates total response time, monitoring p95 latency highlights worst-case scenarios and helps you optimize prompts, reduce context size, or adjust model selection.

MCP Tool Call Distribution

This panel visualizes how often the agent queries different MCP endpoints: logs, metrics, or traces. It gives you a sense of workload distribution, showing whether users are primarily asking about latency, error logs, or trace investigations.

Input and Output Token Usage

This panel tracks the total number of input and output tokens processed by the LLM over time. Input tokens include user queries and MCP responses passed into the model, while output tokens are the agent’s natural language answers. Monitoring this helps manage cost and detect patterns in verbosity or context expansion.

Model Call Error Rate Over Time

This panel tracks the error rate of model calls, visualized as a line chart. Spikes here may indicate upstream issues such as invalid MCP responses, token limits being exceeded, or transient API errors. By correlating these errors with traffic patterns, you can quickly pinpoint reliability issues in production.

With these dashboards in place, you can move beyond ad-hoc debugging and gain data-driven insights into your MCP agent. Whether it’s latency hotspots, slow SigNoz queries, token usage spikes, or rising error rates, SigNoz provides the observability foundation you need to ensure your AI-driven observability assistant stays reliable and responsive.

Wrapping it Up

LangChain agents integrated with MCP servers open the door to powerful new workflows, but that power comes with more moving parts: LLM calls, tool interactions, server communications, and error handling. Without the right observability, it’s easy for problems to hide in the noise.

By pairing OpenTelemetry with SigNoz, you get full visibility into the agent lifecycle: where time is spent, which tools are bottlenecks, and what errors are occurring. Whether it’s a slow external API, a looping agent, or a rate limit error, you can see exactly what happened and where.

With this clarity, debugging becomes faster, scaling becomes smoother, and users get more reliable experiences even as your agents grow more complex.

Sample Reddit thread on how to get started with OTel

This blog is an attempt for anyone lost to find their way into observability and a wake-up call for devs to they should think about observability more actively today than ever before.

A dev’s observability playbook.

Why should you, a developer, care?

As devs, we often obsess over making our code neater, maintaining systems better, and reducing technical debt. We think of a couple of edge cases and handle them well. We write some tests, debug a bit, drink some hot brew, then call it a day. However, in 2025, I am unsure if this will make the cut.

Here’s a short elevator pitch on why you, as a developer, should care about observability today.

Product Engineers With Extreme Ownership

Gone are the days when a PM would hand you a requirements document and design, and then you would just code and then leave the testing to a QA and then whatever happens next to the SREs. The role of a dev has expanded beyond this.

Here’s what a day in the life of a product/software engineer looks like today,

You are kind of expected to know everything, at least a little bit. Companies increasingly value product engineers who own the full lifecycle of a feature from design to coding to deployment and monitoring. It means you, the developer, need to know when your application misbehaves in the wild and be ready to fix it, which is exactly what observability enables.

Systems are Scaling Faster (and getting more complex)

Modern software architectures have exploded in scale and complexity. We’re building distributed microservices, deploying to clouds and Kubernetes, handling global user traffic and shipping faster than ever before. When hundreds of containers or functions communicate with each other, failures often cascade in unpredictable ways. We thrive on getting a holistic view of these complex systems, which is exactly what observability solves for.

Testing every Edge Case isn’t Feasible

Building something as simple as an input box itself can include a multitude of edge cases.

• What if the input is too short or too long?

• What if there’s a special character in the input?

• How to handle white spaces?

• How to handle SQL injection?

These are a few from the top of my mind. However, testing and brainstorming for potential edge cases can become increasingly cumbersome as systems become more complex.

Observability acts as your safety net, catching issues that slip through testing and helping you understand real-world system behaviour.

Users don’t like bugs, but they HATE slow resolution more

As an end-user for a lot of products, I am very impatient when something stops working. So I can imagine what users feel like when their product doesn’t work as expected. Bugs and outages are never welcome, but what really frustrates users is when issues drag on without a fix.

We are in a highly fast-paced world, where no one waits for anything, and users have zero tolerance for downtime or latency. Performance of systems is mission-critical. Every hour of downtime or a delayed fix can cost a substantial amount.

Observability is what makes rapid resolution possible; it helps you spot issues immediately and pinpoint the root cause without wasting time, and directly translates to customer retention.

Observability: Beyond APM and Infra Monitoring

So what’s the point of observability, anyway?

Is it just a fancy word for monitoring?

Not really.

Traditional Application Performance Monitoring [APM] and infrastructure monitoring are about tracking known metrics [CPU, memory, request latency, etc.] and alerting on predefined thresholds. Observability goes beyond that by enabling you to infer the internal state of the system from its outputs.

It’s often defined by three pillars of telemetry data — logs, metrics, and traces, but there’s more to it. Together, they give you a 360° view of what’s happening inside your applications.

• Logs are the record of events [think of them as your app’s diary of what it’s doing].

• Metrics are numeric measurements [e.g. memory at 75%, 500 requests/minute] that track trends and health.

• Traces follow the path of a single request or transaction through multiple services [useful in microservices to see how a request flows and where it slows down].

Observability tools unify these signals to help you answer new questions about your system’s behaviour, not just the ones you preset.

For example, a classic monitor might tell you the error rate exceeded 5% and something’s wrong, whereas an observability approach lets you dig in and ask why it’s wrong, which users or inputs caused this? What else was happening on the system at that time?

It’s a more exploratory, investigative mindset.

Crucially, observability isn’t limited to just application performance like APM is. Today it has expanded to cover the health of the entire system, including infrastructure and third-party services. APM might catch known issues [say, a slow database query you anticipated], but observability will help surface the weird, unexpected issues that weren’t explicitly looked for.

Hello World, OpenTelemetry.

By now, hopefully, it’s clear why you should care about observing your systems actively. The next obvious question is how you can achieve it as a developer.

Your observability tooling will often be influenced by what your org has already adopted. Many teams inherit an existing monitoring stack, maybe Prometheus for metrics [along with dashboards powered by PromQL], or a log system that uses LogQL. These tools may already be wired into alerting pipelines, dashboards, and operational runbooks. In such cases, it’s wise to continue using what’s already working well.

The good news is that OpenTelemetry plays nicely with many of these tools, so you can gradually adopt it without disrupting what’s in place.

That said, if you are starting on a fresh slate, I’d strongly recommend OpenTelemetry [OTel]. The advantages that OTel brings to the table are plenty. You can read more about the advantages of having a vendor-agnostic and open-source observability framework from OTel’s official docs.

OpenTelemetry in < 200 words

At its core, OTel introduces the idea of signals, primarily traces, metrics, and logs, that describe what your application is doing. Developers use the OTel API to create and emit these signals, while the OTel SDK handles the heavy lifting of batching, processing, and exporting the data to your chosen backend.

Instrumentation [the process of collecting these signals] can be done automatically or manually. Usually, it’s a humble mix of both.

Once instrumented, your application will emit trace spans, metrics, and logs in OTel’s standard formats. You can configure exporters to send that data to various backends, whether that’s printing to console during development or an observability vendor. The key point is that OTel decouples instrumentation from the backend. You instrument your code once, then choose where to send the data. This means you get the flexibility to start small with any vendor and switch up as you scale to vendors that are better suited for your needs.

To understand OTel in more depth, I highly suggest you to give this a read.

Copy, Paste & Run Example

Let me take you through a small exercise that can quickly show you the power of OpenTelemetry.

Say you have any application, it could be a side project or a micro-service you are an owner of [just start a new branch 🤷🏻‍♀️]. Since Python is a highly common language, the next couple of instructions will be for a Python application, but a simple Google search or LLM input would help to tweak this to any language of your choice.

1. Create and activate a virtual environment

2. Run the following commands, in the given sequence,

pip install opentelemetry-distro

pip install flask requests

opentelemetry-bootstrap -a install

opentelemetry-instrument --traces_exporter console --metrics_exporter console --logs_exporter console python server_automatic.py

You just completed a very basic instrumentation of your Python application, and you should be seeing traces, metrics and logs getting output to your console!

⚠️ Reminder

Reminder: This is a very basic example of instrumentation. OpenTelemetry has way more potential and power, but this is a starter example to quickly give you a hands-own experience on how instrumentation with OpenTelemetry feels like, and what kind of telemetry data can you collect.

Beating the OpenTelemetry Learning Curve

Like almost any other skill in life, OpenTelemetry also has quite a learning curve. In fact, there is a whole Reddit thread titled “OpenTelemetry is great, but why is it so bloody complicated?”.

Depending on how deeply you want to observe your application, the complexity can vary. For instance, getting started with an example shown above is very easy, but the moment you dive deeper into traces, logs, metrics, spans, etc., it can become overwhelming. I’d like to introduce you to the Dunning-Kruger curve of confidence vs. competence.

I just wanted to tell you not to be discouraged. This steep ramp-up is common, and with a step-by-step approach, you will get comfortable with the concepts over time.

I highly suggest following our Blog series on OpenTelemetry, which has articles on almost every topic related to OpenTelemetry.

Next steps..

So where do you go from here?

Review your side projects

First, review your current projects or side projects and assess your observability gaps. Are you logging enough information? Do you have metrics for key behaviours? If not, that’s a great place to begin. Start instrumenting gradually using the tools and tips above. Consider implementing OpenTelemetry in one service and showcasing a trace to your team; it might inspire wider adoption once they see the value.

Make Observability a Habit

Next, consider making observability a habit in your development workflow. For instance, when reviewing code or designing new components, include observability questions in the process [e.g., “How will we know if this fails in production?”]. Over time, you and your team will naturally build more observable systems. This proactive approach eventually pays off by reducing nasty surprises and shortening debug sessions when issues do occur. Keep learning and stay updated. The observability landscape is evolving [with improvements in OpenTelemetry, new analysis tools, etc.], and being knowledgeable will set you apart. Follow a couple of observability blogs or community forums [the r/observability subreddit, devops blogs, CNCF talks] to see what challenges others are solving.

Embrace the Ownership Mindset

Finally, embrace the mindset*: as a developer, caring about observability means caring about your software beyond just writing code. It’s about owning the reliability and performance of what you build. In 2025 and beyond, the ability to quickly understand and fix issues in complex systems is gold. By investing in observability and tools like OpenTelemetry, you’re essentially future-proofing your career and your projects. So grab that playbook, get your hands dirty with some telemetry, and start turning those unknown unknowns into well-understood knowns. Your users [and your on-call self] will thank you!