Cluster certificate authority cannot be an intermediate authority

[tvs]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:

When trying to configure the Kubernetes cluster with an intermediate certificate authority, kube-apiserver fails to deploy.

Task 2280

Task 2280 | 01:01:29 | Preparing deployment: Preparing deployment (00:00:07)
Task 2280 | 01:01:45 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 2280 | 01:01:46 | Creating missing vms: master/67e1bb8b-3d5d-455e-99df-478a58a64f5b (0)
Task 2280 | 01:01:46 | Creating missing vms: master/943d30a8-9057-41fe-a0a4-aacd3db74088 (1)
Task 2280 | 01:01:46 | Creating missing vms: master/4ee125b8-f7d2-4d6d-9220-e183be97b67e (2)
Task 2280 | 01:01:46 | Creating missing vms: worker/db3cfa81-8c27-4d6f-858a-ab7eff746be4 (0)
Task 2280 | 01:01:46 | Creating missing vms: worker/688b3883-efd7-450b-a670-fee76d8c77e2 (1)
Task 2280 | 01:01:46 | Creating missing vms: worker/a22a98b6-71bb-4f22-82c0-5875cf53a053 (2)
Task 2280 | 01:02:25 | Creating missing vms: master/67e1bb8b-3d5d-455e-99df-478a58a64f5b (0) (00:00:39)
Task 2280 | 01:02:26 | Creating missing vms: worker/db3cfa81-8c27-4d6f-858a-ab7eff746be4 (0) (00:00:40)
Task 2280 | 01:02:35 | Creating missing vms: worker/688b3883-efd7-450b-a670-fee76d8c77e2 (1) (00:00:49)
Task 2280 | 01:02:36 | Creating missing vms: worker/a22a98b6-71bb-4f22-82c0-5875cf53a053 (2) (00:00:50)
Task 2280 | 01:02:37 | Creating missing vms: master/943d30a8-9057-41fe-a0a4-aacd3db74088 (1) (00:00:51)
Task 2280 | 01:02:38 | Creating missing vms: master/4ee125b8-f7d2-4d6d-9220-e183be97b67e (2) (00:00:52)
Task 2280 | 01:02:39 | Updating instance master: master/67e1bb8b-3d5d-455e-99df-478a58a64f5b (0) (canary) (00:03:05)
                     L Error: Action Failed get_task: Task 70f60713-f01e-4030-5cc1-c3aaef997a69 result: 1 of 4 post-start scripts failed. Failed Jobs: kube-apiserver. Successful Jobs: etcd, bosh-dns, kubernetes-roles.
Task 2280 | 01:05:44 | Error: Action Failed get_task: Task 70f60713-f01e-4030-5cc1-c3aaef997a69 result: 1 of 4 post-start scripts failed. Failed Jobs: kube-apiserver. Successful Jobs: etcd, bosh-dns, kubernetes-roles.

Task 2280 Started  Fri Jun 22 01:01:29 UTC 2018
Task 2280 Finished Fri Jun 22 01:05:44 UTC 2018
Task 2280 Duration 00:04:15
Task 2280 error

Updating deployment:
  Expected task '2280' to succeed but state is 'error'

Exit code 1

From post-start.stdout.log:

Waited for 60s, but kubernetes api is still not healthy

This is likely because the curl command fails:

# curl -X GET --fail ${apiserver}/healthz --header "Authorization: Bearer ${token}" --cacert ${cert} -w "\n"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

If you add the -k option, you get the correct response:

# curl -X GET --fail ${apiserver}/healthz --header "Authorization: Bearer ${token}" --cacert ${cert} -w "\n" -k
ok

From kube-apiserver.stderr.log (10.0.16.7 is the master node):

I0622 01:04:43.693659       1 logs.go:49] http: TLS handshake error from 10.0.16.7:58548: remote error: tls: unknown certificate authority

Presumably the Root CA needs to be added to the list of trusted roots.

What you expected to happen:

I should be able to create the cluster using an intermediate authority from any depth of the chain. This is useful for establishing a proper trust chain that can be shared amongst all of the deployments in my environment. In the case of a tool like PKS, this would mean having a trust hierarchy that chains all the way back to a single known root for each of my clusters.

How to reproduce it (as minimally and precisely as possible):

As an example the manifest might look like:

- name: root_ca
  options:
    common_name: root-ca
    is_ca: true
  type: certificate
- name: kubo_ca
  options:
    ca: root_ca
    common_name: kubo-ca
    is_ca: true

Environment:

• Deployment Info (bosh -d <deployment> deployment):

Name  Release(s)          Stemcell(s)                                     Team(s)  Cloud Config
crap  bosh-dns/1.5.0      bosh-google-kvm-ubuntu-trusty-go_agent/3586.24  -        latest
      bpm/0.6.0
      cfcr-etcd/1.3
      docker/32.0.0
      kubo/0.17.0+dev.33

1 deployments

This is a development pre-release of what will go in to 0.18.0.

• Environment Info (bosh -e <environment> environment):

Name      bosh-thall-cfcr
UUID      55f6a6ad-6dc7-4290-b5ef-3f963f8ca2a2
Version   264.7.0 (00000000)
CPI       google_cpi
Features  compiled_package_cache: disabled
          config_server: enabled
          dns: disabled
          snapshots: disabled
User      admin

• Kubernetes version (kubectl version): 1.10.4
• Cloud provider (e.g. aws, gcp, vsphere): All.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/158547527

The labels on this github issue will be updated when the story is started.

[tvs]

The above example was, at least partially, the result of a bug with how Credhub generates intermediate and leaf certificates. That was fixed as a result of cloudfoundry/credhub@35e66ff .

That said, kubo-release still needs the ability to construct the various certs files as chains. For example, kube-apiserver would need to use a server cert including every layer of intermediate in the chain except the root.