logo

Kubernetes - Networking

Last Updated: 2024-01-27

Upper Networking = kubernetes defined networking.

Dataplane networking:

  • Pod IPAM (IP Address Management) and connectivity
  • Service resolution and routing
  • Network policy enforcement

Kubernetes lacks network isolation by default; you can use a CNI with NetworkPolicy or ServiceMesh?

Pod-to-Pod

  • "IP-per-pod" model: each Pod has a unique IP; a Pod is equivalent to a VM or a host (which have unique IPs). Pods can be load-balanced.
  • Containers within a Pod use networking to communicate via loopback. Container to container: use localhost.
  • In the absence of any network security policies, Pods can communicate with all other pods on any other node, without NAT.
  • Agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node.
  • Isolation (restricting what each pod can communicate with) is defined using NetworkPolicy. Network policies are implemented by the network (CNI) plugins.
  • Pod to internet traffic needs to go through a NAT Gateway.

NetworkPolicy

By default, a pod is non-isolated for ingress or egress, i.e. all inbound and outbound connections are allowed.

NetworkPolicy can restrict the connections.

spec
  policyTypes:
  - Ingress
  - Egress

NetworkPolicy is enforced by CNI plugin (not all CNI plugins support this though).

Kubernetes Networking

There are two components to networking.

  • Kubernetes cluster networking, i.e., pod-to-pod connectivity, or "data plane". Can be provided by bundling Cilium (eBPF based programmable dataplane) (run Cilium in overlay mode if you do not require L2 connectivity between all nodes or an SDN).
  • L4 load balancing: can be provided by bundling MetalLB.

Kubernetes Engine clusters are provisioned with an IP range that is automatically determined during cluster creation, in an existing project subnet.

CNI

CNI: Container Networking Interface. The goal is to standardize the interface between the container runtime and the network implementation. It is a minimal standard way to connect a container to a network.

  • CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted.
  • CNI plugins handle tasks such as assigning IP addresses, creating network interfaces, and setting up network routes for containers.
  • The CNI plugin configures a network interface in pod A’s container with the assigned IP address. It also sets up necessary routing rules and network policies.

The term "CNI" may refer to:

  • The CNI Specification
  • A golang library which provides an implementation of the CNI specification
  • CNI Plugins

Since Kubernetes 1.24, management of the CNI is no longer in scope for kubelet. CNI plugins are managed by a Container Runtime (e.g. containerd).

CNI is used by container runtimes: The container/pod initially has no network interface. CNI is used so the pods can accept traffic directly, which keeps network latency as low as possible.

CNI flow:

  • When the container runtime expects to perform network operations on a container, it calls the CNI plugin with the desired command.
  • The container runtime also provides related network configuration and container-specific data to the plugin.
  • The CNI plugin performs the required operations and reports the result.

CNI config:

/etc/cni/net.d

When a pod is created in Kubernetes, the container runtime calls the CNI plugins to set up the network environment. Container Runtime should be aware of the CNI plugin, for example, containerd will have a section like this in /etc/containerd/config.toml:

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""

https://github.com/containernetworking/cni

Implementations

  • Flannel and Weavenet: easy setup and configuration.
  • Calico: better for performance since it uses an underlay network through BGP. Relies on iptable.
  • Cilium: utilizes a completely different application-layer filtering model through BPF and is more geared towards enterprise security. Can replace kube-proxy (with iptables). eBPF is superior to iptables.

GKE Dataplane V2 migrated from Calico (Calico CNI and Calico Network Policies, rely heavily on IPtables functionality in the Linux kernel, IPtables provide a flexible, but not programmable datapath that enables K8s networking functions) to a programmable datapath. based on eBPF/Cilium.

anetd is the networking controller, which replaces calico-* and kube-proxy-* pods. anetd/cilium holds metadata (network policies, configured k8s services and their endpoints) and accounts metrics (conntrack entries, dropped, forwarded traffic) related to all networking in the nodes.

Load Balancing

  • L4 Load Balancer: should support TCP/UDP loadbalancing and high availability. It should work in a world where nodes are in different L2 subnets. E.g.
    • MetalLB for workloads.
    • keepalived + haproxy for control plane nodes.
  • L7 Load Balancer: e.g. via Istio ingress gateway

Note about Load Balancers in k8s:

  • Kubernetes ships glue code that calls out to various IaaS platforms (GCP, AWS, Azure...)
  • Kubernetes does NOT offer an implementation of network load balancers for bare-metal clusters. Use MetalLB for bare-metal clusters.

MetalLB

MetalLB receives requests from the outside of the cluster, and balances them across the load balancer(s) in the cluster.

Without MetalLB, nginx ingress service in bare metal stays in pending state because it has no IP assigned to it. MetalLB does the job of assigning nginx an external IP.

LoadBalancer Service specifies no address, it requires metallb to specify an IPAddressPool and use L2Advertisement to advertize.

SNMP

Get metrics from the network switches

snmp-exporter to expose the SNMP data to Prometheus.

SNMP is a known resource hog.

Gateway

https://gateway-api.sigs.k8s.io/

(Ingress api version 2)

The Gateway API is a SIG-Network project being built to improve and standardize service networking in Kubernetes.

https://istio.io/latest/blog/2022/gateway-api-beta/

Gateway => VirtualService => Service

apiVersion: networking.istio.io/v1beta1
kind: Gateway

Gateway=proxy=load balancer

Gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections.

Defines exposed ports and protocols

Every Gateway is backed by a Service of type LoadBalancer.

VirtualService

configure the L7 LoadBalancing or reverse proxy, split traffic e.g. if uri prefix is /api/, go to service 1, if / go to service 2.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService

A VirtualService can be bound to a gateway to control the forwarding of traffic arriving at a particular host or gateway port.

Dualstack

k8s dualstack: both ipv4 and ipv6: https://kubernetes.io/docs/concepts/services-networking/dual-stack/

logo
PRIVACY POLICYABOUT中文