ZOOKEEPER-5070: Support Single EKU certificates

[PDavid]

Goal

Support certificates with a single Extended Key Usage (EKU) - one cert with serverAuth for incoming connections, another with clientAuth for outgoing connections

Approach

Use separate SSLContext instances for client and server role instead of one. Each is initialized with standard PKIX managers from its own keystore / truststore.

If these new properties are not set, ZooKeeper falls back to the existing keyStore.* and trustStore.* configs (shared for both roles), so the change is fully backward-compatible.

Configuration changes

Introduces the following new config properties:

Client keystore

Presented when initiating connections (client role). Allows using a certificate with only clientAuth EKU.

zookeeper.ssl.client.keyStore.location=/etc/zookeeper/certs/client-keystore.jks
zookeeper.ssl.client.keyStore.password=clientKeystorePass
zookeeper.ssl.client.keyStore.type=JKS

Server truststore

Used when accepting connections (server role). Validates the connecting client's certificate during mTLS.

zookeeper.ssl.server.trustStore.location=/etc/zookeeper/certs/server-truststore.jks
zookeeper.ssl.server.trustStore.password=serverTruststorePass
zookeeper.ssl.server.trustStore.type=JKS

Quorum client keystore

zookeeper.ssl.quorum.client.keyStore.location=/etc/zookeeper/certs/quorum-client-keystore.jks
zookeeper.ssl.quorum.client.keyStore.password=quorumClientKeystorePass

Quorum server truststore

Validates clients when accepting connections

zookeeper.ssl.quorum.server.trustStore.location=/etc/zookeeper/certs/quorum-server-truststore.jks
zookeeper.ssl.quorum.server.trustStore.password=quorumServerTruststorePass

TODO

• Add documentation

[anmolnar]

Please add documentation to zookeeper-website/app/pages/_docs/docs/_mdx/admin-ops/administrators-guide/communication-using-the-netty-framework.mdx

[anmolnar]

Where do you configure the context to work in "EKU-mode", in order to pick certificate from the store based on the right purpose? e.g pick "serverAuth" on the server side and pick "clientAuth" on the client side.

Does that happen automatically?

[PDavid]

The JVM will validate the certificate EKU when a socket is opened. So in case we misconfigure e.g. give "clientAuth" on the server side then we will get an Exception. So the application does not have to check this.

[anmolnar]

Please help me to understand the concept better.

You introduced only the following new properties:

ssl.(quorum).client.keyStore
ssl.(quorum).server.trustStore

Shouldn't we also have:

ssl.(quorum).client.trustStore
ssl.(quorum).server.keyStore

In my understand and EKU-supported zoo.cfg should look like this:

secureClientPort=2182
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.server.keyStore.location=/Users/andor/work/ssl/server-keystore.jks
ssl.server.keyStore.password=password
ssl.server.trustStore.location=/Users/andor/work/ssl/server-truststore.jks
ssl.server.trustStore.password=password
ssl.client.keyStore.location=/Users/andor/work/ssl/client-keystore.jks
ssl.client.keyStore.password=password
ssl.client.trustStore.location=/Users/andor/work/ssl/client-truststore.jks
ssl.client.trustStore.password=password
...
sslQuorum=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.server.keyStore.location=/Users/andor/work/ssl/server-keystore.jks
ssl.quorum.server.keyStore.password=password
ssl.quorum.server.trustStore.location=/Users/andor/work/ssl/server-truststore.jks
ssl.quorum.server.trustStore.password=password
ssl.quorum.client.keyStore.location=/Users/andor/work/ssl/client-keystore.jks
ssl.quorum.client.keyStore.password=password
ssl.quorum.client.trustStore.location=/Users/andor/work/ssl/client-truststore.jks
ssl.quorum.client.trustStore.password=password

The original ssl.(quorum).keystore/truststore settings should be blank in the EKU case and vica versa.

[PDavid]

Please add documentation to zookeeper-website/app/pages/_docs/docs/_mdx/admin-ops/administrators-guide/communication-using-the-netty-framework.mdx

Thanks, this is a very good point, I'll add it. 👍

[PDavid]

Please help me to understand the concept better.

You introduced only the following new properties:

ssl.(quorum).client.keyStore
ssl.(quorum).server.trustStore

Shouldn't we also have:

ssl.(quorum).client.trustStore
ssl.(quorum).server.keyStore

Sorry, I did not mentioned this in the PR but my idea was to only introduce new TLS properties which were not yet present and use the existing TLS properties with the existing names.

Here is an example of the relevant TLS properties from zoo.cfg:

...

# === Existing TLS properties (unchanged) ===

# Server keystore — presented when accepting connections (server role)
zookeeper.ssl.keyStore.location=/etc/zookeeper/certs/server-keystore.jks
zookeeper.ssl.keyStore.password=serverKeystorePass
zookeeper.ssl.keyStore.type=JKS

# Truststore — used when connecting to other nodes (client role)
# Validates the remote server's certificate
zookeeper.ssl.trustStore.location=/etc/zookeeper/certs/client-truststore.jks
zookeeper.ssl.trustStore.password=clientTruststorePass
zookeeper.ssl.trustStore.type=JKS

# === New properties ===

# Client keystore — presented when initiating connections (client role)
# Allows using a certificate with only clientAuth EKU
zookeeper.ssl.client.keyStore.location=/etc/zookeeper/certs/client-keystore.jks
zookeeper.ssl.client.keyStore.password=clientKeystorePass
zookeeper.ssl.client.keyStore.type=JKS

# Server truststore — used when accepting connections (server role)
# Validates the connecting client's certificate during mTLS
zookeeper.ssl.server.trustStore.location=/etc/zookeeper/certs/server-truststore.jks
zookeeper.ssl.server.trustStore.password=serverTruststorePass
zookeeper.ssl.server.trustStore.type=JKS

And the quorum equivalents for inter-node communication:

# Quorum server keystore
zookeeper.ssl.quorum.keyStore.location=/etc/zookeeper/certs/quorum-server-keystore.jks
zookeeper.ssl.quorum.keyStore.password=quorumServerKeystorePass

# Quorum client keystore (New)
zookeeper.ssl.quorum.client.keyStore.location=/etc/zookeeper/certs/quorum-client-keystore.jks
zookeeper.ssl.quorum.client.keyStore.password=quorumClientKeystorePass

# Quorum truststore (validates servers when connecting as client)
zookeeper.ssl.quorum.trustStore.location=/etc/zookeeper/certs/quorum-client-truststore.jks
zookeeper.ssl.quorum.trustStore.password=quorumClientTruststorePass

# Quorum server truststore (New)
# validates clients when accepting connections
zookeeper.ssl.quorum.server.trustStore.location=/etc/zookeeper/certs/quorum-server-truststore.jks
zookeeper.ssl.quorum.server.trustStore.password=quorumServerTruststorePass

...

This approach makes less changes but might be a bit confusing as we don't have some client / server counterpart properties.

What do you think? Is this a good approach or should we introduce those additional new properties?

[anmolnar]

I've found (thanks to Codex/ChatGPT) the bug with client mTLS authentication. You missed to modify X509AuthenticationProvider to adopt the new configuration:

tm = x509Util.buildTrustManager(config);

should be

tm = x509Util.buildServerTrustManager(config);

Otherwise it builds the trust relation chain with the wrong truststore. You might also want to add a unit test to X509AuthTest to cover this scenario.

p.s. I'm still not sure about the config parameters. Your approach makes sense to me too.

[anmolnar]

I created a pull request with the code that ChatGPT generated:
PDavid#2