JWS key rotation

The JWS token provider issues tokens using asymmetric signing. This
document attempts to describe how to manage key pairs in a deployment of
keystone nodes that need to validate tokens issued by one another.

The inherent benefit of using asymmetric keys is that each keystone
server generates it’s own key pair. The private key is used to sign
tokens. Anyone with access to the public key has the ability to verify
the token signature. This is a critical step in validating tokens across
a cluster of keystone nodes.

It is necessary for operators to sync public keys across all keystone
nodes in the deployment. Each keystone server will need a corresponding
public key for every node. This only applies to public keys. Private
keys should never leave the server they are generated from.

Initial setup

Before a deployment of keystone servers can issue JWT tokens, each
server must set keystone.conf [token] provider = jws.
Additionally, each API server must have its own asymmetric key pair
either generated manually or using
keystone-manage create_jws_keypair. If you’re generating
the key pairs manually, they must be usable with the ES256
JSON Web Algorithm (JWA). It is worth noting
that the keystone-manage create_jws_keypair command line
utility will create an appropriate key pair, but it will not
automatically deploy it to the key repository locations defined in
keystone.conf [jwt_tokens]. It is up to operators to move
these files accordingly and resolve possible file name conflicts.

After generating a key pair, the public key from each API server must
be shared with every other API server in the deployment. Ensure the
private key used to sign JWS tokens is readable by the process running
keystone and available in the
keystone.conf [jwt_tokens] jws_private_key_repository
location. Keystone will automatically use a key named
private.pem to sign tokens and ignore all other keys in the
repository. To validate tokens, keystone will iterate all available
public keys in
keystone.conf [jwt_tokens] jws_public_key_repository. At a
minimum, this repository needs to have the corresponding public key to
the private.pem key found in
keystone.conf [jwt_tokens] jws_private_key_repository.

Continued operations

Depending on the security requirements for your deployment, you might
need to rotate out an existing key pair. To do so without prematurely
invalidating tokens, follow these steps:

1. Generate a new asymmetric key pair for a given keystone API server
   (see keystone-manage create_jws_keypair for more
   details)
2. Copy or sync the newly generated public key to the public key
   repositories of all other keystone API servers, the public key should be
   placed in
   keystone.conf [jwt_tokens] jws_public_key_repository
3. Copy the new private key to the private key repository on the API
   server you’re performing the rotation on and make sure it’s named
   private.pem, at this point the server will start signing
   tokens with the new private key and all other keystone API servers will
   be able to validate those tokens since they already have a copy of the
   public key from step #2
4. At this point, you must wait until the last tokens signed with the
   old private key have expired before you can remove the old corresponding
   public keys from each keystone API server, note this should be a minimum
   of keystone.conf [token] expiration
5. Once you’re confident all tokens signed with the old private key are
   expired, it is safe to remove the old corresponding public key from each
   API server in the deployment, which is important in case the original
   private key was compromised and prevents attackers from using it craft
   their own tokens