An overview of the new features in Kubernetes v1 29

kubernetes v1.29 is the third major release update of 2023 and the last major release of the year, containing 49 major updates.

Translated from An Overview of New Features in Kubernetes v129, by Mohamed Ben Hassine. The first version released this year, v127 close to 60 items, second version v128 has 46 items. Even though Kubernetes has been around for almost 10 years, Kubernetes is still very active!

There are 19 enhancements in this release that are in alpha, 19 in beta, and 11 in stable.

As you can see, there are still a lot of new features that are being introduced gradually.

This feature should be very important for all those who develop on kubernetes. Because for the most part, we use CRD to implement functional extensions in Kubernetes.

In order to provide a better user experience and more reliable input validation, it is necessary to support validation when extending functionality through CRD.

CRD currently natively supports two types of validation features:

Checks based on CRD structure definitions.

OpenAPIV3 validation rules.

For example:

apiversion: apiextensions.k8s.io/v1kind: customresourcedefinitionmetadata: annotations: controller-gen.kubebuilder.io/version: v0.13.0 name: kongplugins.configuration.konghq.comspec: group: configuration.konghq.com names: categories: -kong-ingress-controller kind: kongplugin listkind: kongpluginlist plural: kongplugins shortnames: -kp singular: kongplugin scope: namespaced versions: -name: v1 schema: openapiv3schema: description: kongplugin is the schema for the kongplugins api. properties: apiversion: description: 'apiversion defines the versioned schema of this representation of an object. servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. more info: ' type: string protocols: description: protocols configures plugin to run on requests received on specific protocols. items: description: kongprotocol is a valid kong protocol. this alias is necessary to deal with enum: -http - https - grpc - grpcs - tcp - tls - udp type: string type: array type: object ..x-kubernetes-validations: -message: using both config and configfrom fields is not allowed. rule: '!(has(self.config) &has(self.configfrom))' - message: using both configfrom and configpatches fields is not allowed. rule: '!(has(self.configfrom) &has(self.configpatches))' - message: the plugin field is immutable rule: self.plugin == oldself.plugin

However, these built-in rules are relatively limited. If you want to implement richer validation rules functionality, you can use:

admission webhook

Use a custom validator.

However,Both admission webhooks and custom validators are decoupled from the CRD itself, which also makes it more difficult to develop the CRD and subsequent maintenance costs.

To address these issues, the Kubernetes community has introduced CEL (Common Expression Language)-based validation rules for CRD. The rule can be written directly in the CRD declaration file without the use of any admission webhooks or custom validators, greatly simplifying the development and maintenance costs of the CRD.

In Kubernetes v1In version 29, the CEL-based CRD verification capability reached GA. You just need to usex-kubernetes-validationsDefine validation rules.

It's lightweight and secure enough to run directly in kube-apiserver. Let's look at an example:

apiversion: apiextensions.k8s.io/v1kind: customresourcedefinitionmetadata: annotations: controller-gen.kubebuilder.io/version: v0.13.0 name: kongplugins.configuration.konghq.comspec: group: configuration.konghq.com scope: namespaced versions: -name: v1 schema: openapiv3schema: description: kongplugin is the schema for the kongplugins api. properties: plugin: .x-kubernetes-validations: -message: using both config and configfrom fields is not allowed. rule: '!(has(self.config) &has(self.configfrom))' - message: the plugin field is immutable rule: self.plugin == oldself.plugin

In addition, the CEL has a very rich feature that can be experienced through the ** playground. As I mentioned earlier, this feature has been available for more than two years. It's in kubernetes v1Reached beta in 25 and enabled by default, and now finally reaches GA.

In addition, the Kubernetes Gateway API project is removing all of its Admission webhooks and validating them using CEL-based rules. This is probably the single largest use case in the community right now.

In Kubernetes, native service objects contain the nodeport type, which is used to expose services within the cluster to external entities. Currently, creating a new nodeport and specifying a fixed port comes with some risk. The specified port may already be assigned, resulting in conflicts and service creation failures.

This Kubernetes Enhancement Proposal (KEP) proposes to introduce optional scopes of dynamic and static reservations for services of type nodeport, providing two approaches to port configuration:

Automatically randomly generated: Kubernetes will automatically generate ports based on predefined rules.

Set it manually: Users can manually set the port according to their needs.

For example, kube-apiserver can be used--service-node-port-rangeThe flag controls the range of ports that can be used by nodeport, with the default range being 30000-32767. The formula proposed in this KEP is as follows:

static band start=min(max($min,$node-range-size/$step),$max)

Service node port range: 30000-32767

Range size: 32767 30000 = 2767

Bandwidth offset: min(max(16,2767 32),128)=min(86,128)=86

Static bandwidth starts: 30,000

End of static bandwidth: 30086

Based on this calculation, 30000-30086 is considered a static segment, while the remaining ports are considered dynamic.

Implementing this approach reduces the risk of conflicts when creating a nodeport-type service using a fixed port, improving overall cluster stability.

┌──static │ dynamic │ gf]25c4[/gf]──gf]25ba[/gf] [gf]25c4[/gf]──gf]25ba[/gf]30000 30086 32767

First of all, this feature is an internal aspect of Kubernetes, and for most scenarios, a simple principle is sufficient: "When manually selecting nodeports, try to choose ports in the static range mentioned earlier." "If the port specified by the user is in dynamic range and has not yet been assigned, the creation process will also succeed.

This calculation method is derived from KEP-3070 and is used to assign clusterIPs to services, ensuring a simplified and efficient approach to port allocation in the Kubernetes ecosystem.

The sidecar acts as a secondary container and enhances the functionality of the primary container. While widely used in service mesh scenarios, many users also deploy them in non-service mesh contexts, such as for logging, monitoring, and various other purposes.

In scenarios where sidecars are used, there have been issues in the past. One notable issue is the lack of lifecycle management for sidecar containers.

For example, when a pod is deleted, the lack of proper lifecycle management for the sidecar container can lead to a synchronization mismatch between those services and the lifecycle of the primary container, which can adversely affect service reliability.

This Kubernetes Enhancement Proposal (KEP) addresses these challenges by defining a sidecar container as part of the Init container in the pod specification.

In addition, KEP stipulates that sidecar containers should adhere to an "always reboot" policy. By incorporating sidecars into init containers with consistent restart policies, this proposal aims to enhance the lifecycle management of sidecar containers, thereby promoting better synchronization and reliability in both service mesh and non-service mesh scenarios.

apiversion: v1kind: podmetadata: name: mohamedbenhassine-podspec: initcontainers: -name: log image: mohamedbenhassine/fluentbit restartpolicy: always ..

This ensures that the primary container stops first, and also facilitates control of the component lifecycle of all containers.

This KEP is also interesting, mainly to simplify one of the most common requirements.

Many application pods need to be disconnected when they are shut down to avoid impacting user traffic. As a result, prestop is usually set to perform some related processing or wait before shutting down.

However, the current Prestop hook only supports two types: exec and httpget.

This KEP is designed to enable native sleep operations, such as:

apiversion: apps/v1kind: deploymentmetadata: name: nginxspec: template: spec: containers: -name: nginx image: nginx:1.25.3 lifecycle: prestop: sleep: seconds: 5 readinessprobe: httpget: path: / port: 80

However, this feature is currently in alpha and whether it can finally reach GA depends on feedback from the community.

In Kubernetes, service account tokens are an integral part of ensuring security. They are used to authenticate individual workloads in the cluster and prevent unauthorized access.

kubernetes v1.29 These tokens have been further secured: now each service account can only be associated with a specific pod instance and cannot be misused if compromised. This approach effectively improves the service account and pod lifecycle, greatly reducing the likelihood that an attacker will use a stolen token to carry out an attack.

In v129, kube-apiserver has the following related function gating to control the related functions. Of course, in addition to KEP-4193, this version also promotes KEP-2799, which reduces the number of secret-based service account tokens. This helps shorten the validity of the token and minimize the attack surface.

legacyserviceaccounttokencleanup=true|false (beta - default=true)serviceaccounttokenjti=true|false (alpha - default=false)serviceaccounttokennodebinding=true|false (alpha - default=false)serviceaccounttokennodebindin**alidation=true|false (alpha - default=false)serviceaccounttokenpodnodeinfo=true|false (alpha - default=false)

container_cpu_usage_seconds_total

container_memory_working_set_bytes

container_start_time_seconds

node_cpu_usage_seconds_total

node_memory_working_set_bytes

pod_cpu_usage_seconds_total

pod_memory_working_set_bytes

resource_scrape_error

Here's the output of an example:

# help container_cpu_usage_seconds_total [stable] cumulative cpu time consumed by the container in core-seconds# type container_cpu_usage_seconds_total countercontainer_cpu_usage_seconds_total 0.195744 1691361886865# help container_memory_working_set_bytes [stable] current working set of the container in bytes# type container_memory_working_set_bytes gaugecontainer_memory_working_set_bytes 1.675264e+07 1691361886865# help container_start_time_seconds [stable] start time of the container since unix epoch in seconds# type container_start_time_seconds gauge

A long time ago, there was a componentstatus in Kubernetes, which could be used to view the status of components in a cluster. But as shown below, in v119 has been effectively deprecated.

mohamedbenhassine@k8s-test:~$kubectl get cswarning: v1 componentstatus is deprecated in v1.19+name status message errorscheduler healthy ok etcd-0 healthy ok controller-manager healthy ok

Here's the output of an example:

test@mohamedbenhassine:/$ kubectl get --raw "/metrics/slis"# help kubernetes_healthcheck [stable] this metric records the result of a single healthcheck.# type kubernetes_healthcheck gaugekubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1kubernetes_healthcheck 1

With Kubernetes 1With the release of 29, the readwriteoncepod feature was introduced as a new feature in the stable release. This ensures that a pod that is exclusive to the entire cluster has exclusive read or write access to Persistent Volume Claims (PVC).

apiversion: v1kind: persistentvolumeclaimmetadata: name: my-pvcspec: accessmodes: -readwriteoncepod storageclassname: standard resources: requests: storage: 5gi

apiversion: apps/v1kind: deploymentmetadata: name: teckbootcamps-deploymentspec: template: spec: affinity: podaffinity: matchlabelkeys: -app-2024

apiversion: storage.k8s.io/v1kind: storageclassmetadata: name: encryptedprovisioner: kubernetes.io/aws-ebsvolumebindingmode: waitforfirstconsumerallowvolumeexpansion: true

kubeadm init --cloud-provider=external

Recommended in v129 to turn off this feature first

KEP-2495: PV PVC readwriteoncepod reaches GA state.

The nodeexpandsecret feature reaches the ga state, and kube-proxy now has a new nftables backend.