Kube-API 서버는 Cert-manager가 만든 인증서를 어떻게 이해할까?

kube api server와 webhook 서버는 TLS 통신을 한다.

그리고 오퍼레이터를 만들다 보면 webhook 서버에 대한 인증서는 cert-manager를 통해서 만들기 때문에

api server와 webhook server가 들고 있는 인증서의 root CA는 달라진다. 

 

이때 API 서버에서 webhook 서버에게 요청을 보내기 위해서는 TLS로 통신하긴 하지만

서로의 root CA는 모르기 때문에 insecure 하게 통신해야 하지 않을까?

 

이러한 문제는 cert-manager의 CA injector를 통해서 해결할 수 있다.  

 

일단 cainjector는 Mutating Webhooks, Validating Webhooks, Conversion Webhooks, API Services에 대한 CA 인증서를 주입할 때 사용된다. 

 

cainjector를 사용하면 다음과 같이  ValidatingWebhookConfiguration, MutatingWebhookConfigurationi, CustomResourceDefinition이 가지고 있는 caBundle 필드를 채울 수 있다. 

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: capi-webhook-system/capi-kubeadm-control-plane-serving-cert
    labels:
    cluster.x-k8s.io/provider: control-plane-kubeadm
  name: capi-kubeadm-control-plane-mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1beta1
############################## 이 부분 ###################################
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    service:
      name: capi-kubeadm-control-plane-webhook-service
      namespace: capi-webhook-system
      path: /mutate-controlplane-cluster-x-k8s-io-v1alpha3-kubeadmcontrolplane
      port: 443
############################## 이 부분 ###################################
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: default.kubeadmcontrolplane.controlplane.cluster.x-k8s.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - controlplane.cluster.x-k8s.io
    apiVersions:
    - v1alpha3
    operations:
    - CREATE
    - UPDATE
    resources:
    - kubeadmcontrolplanes
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30

 

그리고 이 caBundle 데이터는 Kubernetes API 서버가 읽어 webhook 서버의 인증서를 확인하는 데 사용된다.

 

위 리소스들에 CA를 주입하기 위해서는 CA injector가 인식할 수 있는 annotation이 필요하다.  

CA bundle을 채우려는 리소스에 다음 중 하나의 annotation만 있으면 된다. 

 

1. cert-manager.io/inject-ca-from: certifcate의 CA 주입하기  

2. cert-manager.io/inject-ca-from-secret: secret의 CA 주입하기

3. cert-manager.io/inject-apiserver-ca: api server의 CA 주입하기

annotation key가 모두 다른데 이는 읽어올 CA 데이터를 k8s secret, cert-manager의 certificate, kube api server의 CA 인증서를 선택해서 가져올 수 있다. 

 

다음은 위 annotation key를 사용하는 방법에 대한 예제다.

 

certificate로부터 CA 데이터를 주입하는 방법

apiVersion: v1
kind: Namespace
metadata:
  name: example1

---

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: webhook1
  # cainjector를 위한 annotation
  annotations:
    cert-manager.io/inject-ca-from: example1/webhook1-certificate
webhooks:
- name: webhook1.example.com
  admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook1
      namespace: example1
      path: /validate
      port: 443
  sideEffects: None

---

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook1-certificate
  namespace: example1
spec:
  secretName: webhook1-certificate
  dnsNames:
  - webhook1.example1
  issuerRef:
    name: selfsigned

---

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: example1
spec:
  selfSigned: {}

 

Secret 리소스에서 CA data를 주입하는 방법

apiVersion: v1
kind: Namespace
metadata:
  name: example2

---

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: webhook2
  # cainjector를 위한 annotation
  annotations:
    cert-manager.io/inject-ca-from-secret: example2/example-ca
webhooks:
- name: webhook2.example.com
  admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook2
      namespace: example2
      path: /validate
      port: 443
  sideEffects: None

---

apiVersion: v1
kind: Secret
metadata:
  name: example-ca
  namespace: example2
  # cainjector를 위한 annotation
  annotations:
    cert-manager.io/allow-direct-injection: "true"
type: kubernetes.io/tls
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5akNDQWQ2Z0F3SUJBZ0lRTkdJZ24yM3BQYVpNbk9MUjJnVmZHakFOQmdrcWhraUc5dzBCQVFzRkFEQVYKTVJNd0VRWURWUVFERXdwRmVHRnRjR3hsSUVOQk1CNFhEVEl3TURreU5ERTFOREEwTVZvWERUSXdNVEl5TXpFMQpOREEwTVZvd0ZURVRNQkVHQTFVRUF4TUtSWGhoYlhCc1pTQkRRVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBS2F3RzVoMzlreHdyNEl0WCtHaDNYVWQrdTVJc2ZlSFdoTTc4TTRQTmZFeXhQMXoKRmNLN1d0MHJFMkwwNUppYmQ4ZjNpb3k5OXNnQ3I4OEw2SWxYZTB0RnkzNysxenJ4TFluR2hDQnZzZjltd0hLbgpIVTEvNERwQjROZkhPbFllNE9tbHVoNE9HdmZINU1EbDh5OWZGMjhXRXVBQ2dwdmpCUWxvRDNlVjJ5UmJvQ2kyCmtSTDJWYTFZL0FQZEpWK21VYkFvZmg0bllmUmNLRTJsSUg0RG5ZdXFPU3JaaituZUQ2M2RTSktxcHQ5K2luN2YKNHljZ2pQYU93MmdyKzhLK291QTlSQTV1VDI3SVNJcUJDcEV6elRqbVBUUWNvUTYxZGF0aDZkc1lsTEU4aWZWUwp4RWZuVEdQKy94M0FXQXR4eU5lanVuZGFXbVNFL3h5OHh0K0FxblVDQXdFQUFhTkNNRUF3RGdZRFZSMFBBUUgvCkJBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkowNkc5eEc2V1VBTHB6T3JYaHAKV2dsTm5qMkFNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUI3ZG9CZnBLR3o4VlRQSnc0YXhpdisybzJpMHE1SQpSRzU2UE81WnhKQktZQlRROElHQmFOSm1yeGtmNTJCV0ttUGp4cXlNSGRwWjVBU00zOUJkZVUzRGtEWHp4RkgwCjM5RU12UnhIUERyMGQ4cTFFbndQT0xZY1hzNjJhYjdidE11cTJUMFNNZzRYMkY5VmNKTW5YdjlrNnA0VGZNR3MKVThCQnJhVGhUZm53ejBsWXMyblFjdzNmZjZ1bG1wWlk4K3BTak1aVDNJZHZOMFA4Y2hOdUlmUFRHWDJmSlo2NQpxcUUrelRoU3hIeXFTOTVoczhsd1lRRUhGQlVsalRnMCtQZThXL0hOSXZBOU9TYWw1U3UvdlhydmcxN04xdHVyCk5XcWRyZU5OVm1ubXMvTFJodmthWTBGblRvbFNBRkNXWS9GSDY5ZzRPcThiMHVyK3JVMHZOZFFXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: ""
  tls.crt: ""

 

Kube-API 서버의 CA 데이터를 주입하는 방법

apiVersion: v1
kind: Namespace
metadata:
  name: example3

---

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: webhook3
  # cainjector를 위한 annotation
  annotations:
    cert-manager.io/inject-apiserver-ca: "true"
webhooks:
- name: webhook3.example.com
  admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook3
      namespace: example3
      path: /validate
      port: 443
  sideEffects: None

 

 

참고로 나도 잘 모르기 때문에 틀린 내용이 있을 수 있다. 

 

Reference

https://cert-manager.io/docs/concepts/ca-injector/