In a Kubernetes cluster, typically each pod has only one network interface (except the "loopback" interface). In reality, there are scenarios where multiple interfaces are required; for example, a virtual network function (VNF) typically needs a "left", "right" and optionally a "management" interface to perform network functions.
A pod may require a "data interface" to carry the data traffic, and a "management interface" for reachability detection. Service Providers also tend to keep the management and tenant networks independent for isolation, and management purposes. Multiple interfaces, thus, provide a way for containers to be connected to multiple devices in multiple networks simultaneously.
In this article, we demonstrate how to create a pod that is bound to multiple VNs.
To create a multiple VNs pod, you need to first create two VNs, and then create a pod to reference these VNs in its yaml file.
Here is a yaml file of two VNs: "vn-left-1" and "vn-right-1."
#vn-left-1.yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: "opencontrail.org/cidr": "10.10.10.0/24" "opencontrail.org/ip_fabric_forwarding": "false" "opencontrail.org/ip_fabric_snat": "false" name: vn-left-1 spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }' #vn-right-1.yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: "opencontrail.org/cidr": "20.20.20.0/24" "opencontrail.org/ip_fabric_forwarding": "false" "opencontrail.org/ip_fabric_snat": "false" name: vn-right-1 #namespace: default spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }'
#vn-left-1.yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: "opencontrail.org/cidr": "10.10.10.0/24" "opencontrail.org/ip_fabric_forwarding": "false" "opencontrail.org/ip_fabric_snat": "false" name: vn-left-1 spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }'
#vn-right-1.yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: "opencontrail.org/cidr": "20.20.20.0/24" "opencontrail.org/ip_fabric_forwarding": "false" "opencontrail.org/ip_fabric_snat": "false" name: vn-right-1 #namespace: default spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }'
Create both VNs:
$ kubectl apply -f vn-left-1.yaml networkattachmentdefinition.k8s.cni.cncf.io/vn-left-1 created $ kubectl apply -f vn-right-1.yaml networkattachmentdefinition.k8s.cni.cncf.io/vn-right-1 created
Examine the VNs:
$ kubectl get network-attachment-definitions.k8s.cni.cncf.io NAME AGE vn-left-1 3s vn-right-1 10s $ kubectl get network-attachment-definitions.k8s.cni.cncf.io vn-left-1 -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"opencontrail.org/cidr":"10.10.10.0/24","opencontrail.org/ip_fabric_forwarding":"false"},"name":"vn-left-1","namespace":"ns-user-1"},"spec":{"config":"{ \"cniVersion\": \"0.3.0\", \"type\": \"contrail-k8s-cni\" }"}} opencontrail.org/cidr: 10.10.10.0/24 opencontrail.org/ip_fabric_forwarding: "false" creationTimestamp: 2019-06-13T14:17:42Z generation: 1 name: vn-left-1 namespace: ns-user-1 resourceVersion: "777874" selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/ns-user-1/network-attachment-definitions/vn-left-1 uid: 01f167ad-8de6-11e9-bbbf-0050569e6cfc spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }'
$ kubectl get network-attachment-definitions.k8s.cni.cncf.io NAME AGE vn-left-1 3s vn-right-1 10s
$ kubectl get network-attachment-definitions.k8s.cni.cncf.io vn-left-1 -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"opencontrail.org/cidr":"10.10.10.0/24","opencontrail.org/ip_fabric_forwarding":"false"},"name":"vn-left-1","namespace":"ns-user-1"},"spec":{"config":"{ \"cniVersion\": \"0.3.0\", \"type\": \"contrail-k8s-cni\" }"}} opencontrail.org/cidr: 10.10.10.0/24 opencontrail.org/ip_fabric_forwarding: "false" creationTimestamp: 2019-06-13T14:17:42Z generation: 1 name: vn-left-1 namespace: ns-user-1 resourceVersion: "777874" selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/ns-user-1/network-attachment-definitions/vn-left-1 uid: 01f167ad-8de6-11e9-bbbf-0050569e6cfc spec: config: '{ "cniVersion": "0.3.0", "type": "contrail-k8s-cni" }'
Here is the yaml file of a multiple interfaces pod:
#pod-webserver-multivn-do.yaml apiVersion: v1 kind: Pod metadata: name: webserver-mv labels: app: webserver-mv annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "vn-left-1" }, { "name": "vn-right-1" } ]' spec: containers: - name: webserver-mv image: contrailk8sdayone/contrail-webserver imagePullPolicy: Always restartPolicy: Always
In pod annotations under metadata, we insert two VNs: "vn-left-1" and "vn-right-1."
Now let's create the pod and verify:
$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE webserver-mv 1/1 Running 0 20s 10.47.255.238 cent222 <none> $ kubectl describe pod webserver-mv Name: webserver-mv Namespace: ns-user-1 Priority: 0 PriorityClassName: <none> Node: cent222/10.85.188.20 Start Time: Wed, 26 Jun 2019 12:51:30 -0400 Labels: app=webserver-mv Annotations: k8s.v1.cni.cncf.io/network-status: [ { "ips": "10.10.10.250", "mac": "02:87:cf:6c:9a:98", "name": "vn-left-1" }, { "ips": "10.47.255.238", "mac": "02:87:98:cc:4e:98", "name": "cluster-wide-default" }, { "ips": "20.20.20.1", "mac": "02:87:f9:f9:88:98", "name": "vn-right-1" } ] k8s.v1.cni.cncf.io/networks: [ { "name": "vn-left-1" }, { "name": "vn-right-1" } ] kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata": {"annotations":{"k8s.v1.cni.cncf.io/networks":"[ { \"name\": \"vn-left-1\" }, { \"name\": \"vn-... Status: Running IP: 10.47.255.238 ...<snipped>...
In Annotations under k8s.v1.cni.cncf.io/network-status , we see a list "[...]," which has three items each represented by a curly brace block " {}" of key-value mappings. Each curly brace block includes information about one interface: the allocated IP address, MAC address, and the VN it belongs to. The second item gives the IP address 10.47.255.238, which is the interface attached to the "default pod network" named "cluster-wide-default," which is created by the system.
k8s.v1.cni.cncf.io/network-status
{}"
Now we can log in to the pod, list the interfaces, and verify the IP address and MAC address.
$ kubectl exec -it webserver-mv sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 37: eth0@if38: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:53:47:06:d8:98 brd ff:ff:ff:ff:ff:ff inet 10.47.255.238/12 scope global eth0 valid_lft forever preferred_lft forever 39: eth1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:53:6b:a0:e2:98 brd ff:ff:ff:ff:ff:ff inet 10.10.10.250/24 scope global eth1 valid_lft forever preferred_lft forever 41: eth2@if42: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:53:8e:8a:80:98 brd ff:ff:ff:ff:ff:ff inet 20.20.20.1/24 scope global eth2 valid_lft forever preferred_lft forever
We now see one lo0 interface and three interfaces plugged by Contrail CNI, each with the IP address allocated from the corresponding VN. Also you will notice the MAC addresses match what we've seen in the kubectl describe command output.
kubectl describe