With the hype for AI ML in the observability space, companies are more likely than ever to reap the benefits of storing data in one system for viewing and training ML models in another.
Translated from Why The Latest Advances in OpenTelemetry Are Significant, author Andy Hoffman has more than 16 years of technical experience and has built and led several DevOps projects, cloud migrations, and architectural overhauls for unicorn startups and established organizations. He finds joy in both in-depth technical work and team building. One of the most popular projects that will be focused on this year's cloud-native computing** is OpenTelemetry and OpenTelemetry Collector. This project is a very exciting development in the field of observability, as it is a cross-industry collaboration to standardize data formats for observability and telemetry. This in itself is important because it allows data to be collected from multiple observables, whereas previously teams were forced to transform data multiple times if they wanted a single picture of an event. With the craze for AI ML in observability, it's more likely than ever that companies can benefit from storing and viewing data in one system and training ML models in another.
What's great is that this project is moving forward thanks to the collaboration of industry leaders and individuals with OpenTelemetry Collector, a standardized telemetry collector that provides high-throughput telemetry collection. Below, I share some of the new features of the project and why they are important to the community.
I've found that many of the syntax makes it very difficult to make meaningful conversions without using some weird yaml or toml. OpenTelemetry Collector still relies on the YAML format, but its new transformation language allows for function-based statements that execute fairly quickly and allow for managing complexity. Check out some examples of that syntax. After about a year of development, log collection is now up to GA. There are several ways to collect logs for this implementation:
First, it runs as a standalone ** and collects logs from the file system. It can be sent directly to the end destination or to OpenTelemetry Collector running in collector mode, where log metrics can be computed on the fly.
Second, there are a number of log SDKs that can be implemented directly in the application and sent to a central collector or directly to the final target, which can help minimize the impact of disk IOs.
Autoinstrumentation refers to the ability to automatically connect applications to emit traces and metrics with little or no change. j**a and ..net is fully supported, and other languages are being developed and released at various stages. The proprietary solution has always showcased this capability as a differentiator because it reduced the complexity of the rollout by minimizing developer time, and now this brings the same powerful capabilities to the OpenTelemetry ecosystem.
This is huge, and benefits from Elasticsearch's donation of ECS (Elastic Common Schema) to the OpenTelemetry project. Standardizing the telemetry structure is challenging because it seems like almost everyone is producing telemetry in a slightly different format; However, in order to be able to analyze, create alerts, and present data in a human-friendly way, all telemetry fields need to be mapped in some way.
If each person and every system is only slightly different, this presents the challenge of making reusable dashboards and components. Software vendors can now be responsible for creating dashboards on many platforms with reasonable confidence that data will appear in the correct format across multiple platforms.
At the same time, those of us who manage large amounts of telemetry can improve ingestion and query efficiency, and we can provide more advanced functionality with less compute resources and memory overhead if the majority of our customers send content that relies on well-known field names.
The full model is still some way from being finalized, but the convention is being gradually approved. For example, on Kubecon, they announced the finalization of the HTTP schema.
The ecosystem is maturing. The extensibility framework allows customization of any stage of the ingestion pipeline. More and more receivers are suitable for a variety of systems, processors have more and more advanced features, as well as destinations. I'm particularly excited about the new version of the OpenSearch extension, which pre-packages the sending of log data in a simplified observability mode or ECS format.
From a developer's point of view, I found that the structure of the schema and the internal "p" message schema was very well thought out and built into protobuf. It strikes a good balance between functional freedom and minimal complexity.
This is not new to the CNCF community, but the speed and impact of this project exemplifies the spirit of the CNCF community philosophy. Competing companies are working together to make a piece of computer better and easier for the rest of us. Some may be concerned that removing vendor lock-in will lead to customer churn, or that sharing may reveal proprietary intellectual property.
However, in the world of telemetry, the core architecture of the collector is often a solved problem. So why not make something that follows conventions and works cross-platform so that companies no longer need to maintain*** 80% of which are duplicates? This allows companies to work on shared plugins for interoperability and proprietary processors where innovation can be delivered through this framework.
*Businesses and all operators also benefit. Using the standardized OpenTelemetry Collector SDK, vendors can create an integration to detect telemetry in their applications and greatly simplify the collection process and try to get all the major observability providers to implement support for your application.
Carriers also benefit from a "collect anywhere" and "send anywhere" mentality. The standard profile format simplifies setup while minimizing the complexity of onboarding new systems. I also suspect that many logging system operators will see a significant reduction in the cardinality of field mapping issues for this issue, thanks to the project's semantic conventions for observability data.
A huge "thank you" to all project contributors and community members! ”!I can't list them all here, but you can track them down in the OpenTelemetry project on GitHub. The capabilities and path forward for OpenTelemetry and OpenTelemetry Collector are advancing rapidly, with the past year being the second largest contribution project in the CNCF portfolio after Kubernetes. With so many contributors staying organized and working together, maturity will continue to accelerate. It is hoped that this will unlock innovation in the field of observability by increasing interoperability and improving the ability of detection systems for telemetry collection.
Editor's note: This article has been revised to clarify the role of OpenTelemetry. )