K8s Cluster Observability Data Offload Best Practices

Under the microservice architecture, multiple sets of services are often deployed in a K8S cluster, which also means that different teams, different roles, and different services will be in the same cluster, and the data of different businesses needs to be managed and viewed in different spaces.

In the traditional host environment, this can be easily achieved by configuring different workspace tokens when deploying datakit on different hosts, but in the k8s environment using daemonset deployment, the same daemonset cannot flexibly configure multiple sets of datakits, and the datakit needs to be restarted when the configuration is changed, and the impact of datakit when it reaches a certain scale is very large.

Therefore, the Dataway Sinker function provided by Observation Cloud is the best solution to the above problems.

As you can see from the figure above, the most important part of this solution is data tag management. Whether the data offload is as expected, accurate, and useful depends on the proper use of tags and planning management. The management and use of tag happens to be one of the core capabilities of the observation cloud platform.

For more information on how to tag, please refer to the Best Practices for Tag in Observing Clouds, which will not be repeated here.

In addition to this, the following attributes are supported for triage:

Observation clouds have built-in custom keys, for example, category for all general data classifications, and its value can be the Name column of the corresponding data classification (for example, time series is metric, object is object, etc.).

object label attribute as well as the out-of-the-box properties of the k8s cluster, for example:namespacecontainer_nameWait.

The following will demonstrate how to use the Dataway Sinker feature to offload and manage data.

In this article, we'll divide the data into workspaces based on a common business attribute, namespace.

As a prerequisite, the Observation Cloud DataKit collector has been deployed in the cluster.

In the test cluster, there are multiple namespaces, as shown in the following figure

In addition, we use the observation cloud datakit to monitor the metrics of the K8S cluster, but all the monitoring metrics are in one workspace OBS, as shown in the following figure

The desired effect is to offload monitoring data to different workspaces based on different namespaces, such as:namespace=datakitAll data is offloaded to the Observation Cloud datakit workspace.

Step 1: Install DataWay

For SaaS users, they can deploy a DataWay locally (K8S Cluster) dedicated to offloading, and then send the data to OpenWay.

1) Refer to the DataWay installation document to install DataWay;

2) Modificationdataway.yamlto add the following sinker-related configuration environment variables;

- Name: DW Secret Token When the data distribution function is enabled, it is used to link with DataKit, note that the 32-bit string value: needs to be added to the TKN"tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"- Name: DW Cascaded When the data distribution feature is enabled, SaaS users use cascading to link value:"on"- Name: DW Sinker File Path mountedjson file address value:"/usr/local/cloudcare/dataflux/dataway/sinker.json"- name: dw remote host configure the cascading address value:""

Here you use the file mode to configure the diversion rule, and etcd is also supported for configuration, for specific configuration, please refer to Dataway Configuration.

Step 2: Edit the diversion rule

Create a filesinker.json, fill in the following information, and mount the file to the dataway container.

Matching rule ], which corresponds to the OpenWay address and token of the workspace"url": "?token=tkn_cb1a9a53fcb04436a4adab6435327fca" }," ],"url": "?token=tkn_c6e8ae1bbfa2489aba843cc56baf3c66" }," ],"url": "?token=tkn_1618f90ef13b482d9f682f30f7118d2f" }

- Name: env dataway Address and Secret Token Value: -name: env sinker global customer keys Specify the key value of the offload: namespace - name: env dataway enable sinker Enable sinker Value:"true"

Only the data in the DataKit workspace has a namespace of DataKit.

Only the data in the utils workspace with namespace as utils is available.

There is no utils and datakit data in the OBS workspace.

At this point, the diversion was successful.

In addition to the above examples, you can also take advantage of DataKit's built-in custom keys, which generally do not appear in the collected data, but DataKit can use these keys to group data. If you need to triage these key dimensions, you can add them to the global custom key list (these keys are not configured by default). We can use some built-in custom keys to implement data offloading. For specific traffic distribution rules, see Built-in Custom Key Traffic Distribution.