Pushing the boundaries of Spring Cloud Config

Spring Cloud Config provides server-side and client-side support for externalized configuration in a distributed system. We’ve used it for two and a half years as the principal configuration mechanism for all our JVM services and Pytests, and have plans to put it to use within our front-end projects, too.

Photo by Joel Holland on Unsplash

While some parts of the Cloud Config family haven’t found favour with us, and we decided not to embrace Spring Cloud Bus with its promise of broadcastable changes and hot reloading of service configuration at runtime, we’re far happier with Cloud Config than our previous solution. It didn’t offer the same flexibility which made configuration changes a slow and painful job for developers. I’ll outline how we’ve gone far beyond the standard Cloud Config offering, and have provided the additional automation to bring this all home to developers.

Automated credential generation and expiry, server-side

Right at the start of our microservice journey, we were determined to avoid the problems of long-lived database and infrastructure credentials, guarded by SysAdmins, though invariably leaked and reused across multiple systems, with the inevitable unexpected breakages should these credentials ever change. Likewise we were determined not to share credentials across services, to minimise unexpected or unwelcome access to our microservices’ private assets.

Photo by Ian Battaglia on Unsplash

With just one annotation:

@MySqlDataSource(schema=”name”,grants=”SELECT,…”)

… our JVM services declare the access they require to a particular database. This information is encoded into the generated bootstrap.yml at build time.

We have our own implementation of a Spring Cloud Config Server, which overrides CompositeEnvironmentRepository that will always generate a brand new MySQL user with a random password and the grants required, store a mapping between the client services’ name/host identifier and the generated username, and assign a TTL. A background task cleans out and deletes any users whose time is up.

Generated credentials are written to standard spring.datasource.username and spring.datasource.password properties, which are ultimately merged into the aggregated PropertySource objects that get returned to the client in JSON form.

A similar mechanism exists to provide and manage RabbitMQ users, and we may well extend to include PostgreSQL.

Every microservice can opt into this process with one line of code, and thus be guaranteed a unique and private set of credentials that matches its requirements, whatever the environment, without any knowledge of the actual values or the precise mechanism used, and with no way (or reason!) for the developer to try to track the values.

Automated credential expiry handling, client-side

Generally, our microservices are changed sufficiently frequently that credentials don’t have time to naturally expire. However, the contract is that credentials may expire at any time, individually, or en masse, and the services must autorecover quickly.

For RabbitMQ credentials, all services automatically subscribe to the amq.rabbitmq.event topic, and listen out for user.deleted and user.authentication.failure events. If the name header matches the service’s last-known RabbitMQ user, the service knows it must refresh itself, and re-bootstrap against the Cloud Config Server. There is a similar mechanism to listen for database authentication errors and initiate refresh. This is baked into the one shared library that all our Spring microservices use, and needs no development time or effort from service writers.

Automated database migration

Photo by Shripal Daphtary on Unsplash

Having been surprised in the past when Flyway database migrations failed the first time they hit the Production database, after passing on all other environments, we were determined to fix this. We wanted to make it both impossible for database-using services to start up without a migration having been run, even on the desktop — but, as ever, make this a friendly and seamless process.

Our seemingly-rudimentary but thoroughly-effective solution was, via our Maven plugin, to tar-gzip the directory holding our database migration files, Base64 the result, and add it to the same JSON object we pass to our Cloud Config Server via bootstrap.yml. Having first generated any database credentials for our client, the Server will then unpack any encoded archive data and perform a database migration for our service, against the target database, using its own elevated credentials. A result code and migration count is then injected into the response JSON.

Thus our microservices can bootstrap and then start up in the certain knowledge that Hibernate’s view of their database schema matches what has been laid down by the DBMS.

Environmental and Negative Profiles

Spring Cloud Config enables the use of arbitrary Spring Profiles to pull customised configurations for a particular service, thus myapp-testenv.yml extends and overrides myapp.yml which in turn extends and overrides application.yml. At the Helm level, we inject an environment variable into the list of profiles to ensure that each microservice can override configuration by deployment environment/Kubernetes cluster. Very useful, but not exceptional.

Our customised Cloud Config Server also supports profile names prefixed with a “-” to allow certain Profiles to be masked out by other Profiles. One practical example is a set of microservices which, while notionally clients of a “security system”, are also operators of said system. A custom annotation defines a Spring Profile for a client; another annotation defines both a Profile for a server while also negating the client Profile. Thus two sets of uses can be accommodated with just one set of explicit configuration, and since clients are more numerous, this is a net win for developers.

Cross-service configuration

Photo by Nick Chalkiadakis on Unsplash

Spring Cloud Config has long supported individual Spring applications opting into multiple configurations using a comma-separated list of names, not merely by setting spring.application.name:

spring.cloud.config.name=myapplication,s-server,reporting-tools

Given that our bootstrap.yml files are generated, we need a way to make this easier for developers to use and harder to abuse. Again, a custom annotation and meta-annotation, backed by our Maven plugin, can fill the gap:

@SecurityServer  // wraps @CloudConfigProfiles({“s-server”,”-s-client})
@CloudConfigProfiles({“reporting-tools”})
public class MyApplication {
    public static void main(String[] args) {
         SpringApplication.run(MyApplication.class, args);
   }
}

This results in the above spring.cloud.config.name being generated, and each of the named configuration names being queried at bootstrap time.

Overall:

This is a very rapid overview of a number of extensions to Spring Cloud Config that were not particularly complicated to implement, but that convey a feeling of robustness, freedom, and flexibility to developers. A feeling that configuration is within their hands, not the responsibility of SysAdmins, that microservices can be built and run with the minimum of unnecessary sweat, and that both static and dynamic configuration can be combined in a powerful and flexible, yet relatively secure way.

Andrew Regan — Technical Architect at Crunch