Open Banking at Crunch: a developer’s perspective
As a provider of accountancy services to thousands of businesses across the country, it’s important to us that we make all parts of our system as straightforward and quick to use as possible.
One area of our system which wasn’t as automated as it could have been was bank reconciliation. Clients were required to manually upload bank statements which proved time-consuming and frustrating for many.
Clients have made this clear to us, with the highest number of requests asking for this process to be automated in four of the last five surveys carried out by Crunch.
Although we provide automatic transaction import from banks that we have partnerships with (Metro Bank, Cater Allen, and Barclays), many of our clients bank elsewhere. This was why we were excited to hear about the recent government initiative “Open Banking”.
The Open Banking website describes their goals as follows:
“Open Banking aims to open data access and standardise technical specifications across the finance industry so that consumers and businesses in the UK have better access and control over their data”.
To help us support as many of our clients as possible, we chose to use Truelayer, who provide a single API. This allows us to integrate the bank data of 98% of the UK population. At first, this may seem alarming, but bank statement data only becomes available to Crunch if each client gives explicit access using the secure OAuth authentication and authorisation standard. Clients are also able to revoke access whenever they wish.
We’ve recently started a closed beta with a small group of Crunch clients for rigorous testing. The system will be released to more clients soon, after our trials are complete.
The remainder of this article discusses the design choices we made for our Open Banking solution and the consequences of these choices.
Designing for Flexibility
Design Overview
We decided early on in the project that we needed to make it as straightforward as possible for us to provide additional bank feed sources to our system. For example, we may wish to integrate a bank not supported by Truelayer, or we may wish to connect some of our existing integrations (e.g. Barclays). To this end, the functionality was split into two services: one to handle communication with Truelayer and the other for storing the retrieved data and providing access to it from our accountancy software.
Client authorisation is obtained using a standard OAuth 2.0 flow. To aid in this, we made use of our OAuth service to handle obtaining and refreshing access tokens on behalf of our clients. For additional information on OAuth 2.0 and out OAuth service, please see our previous post on the topic.
A basic overview of our design is shown in the diagram below:
Communication between all of our services is handled by RabbitMQ; a popular implementation of the ActiveMQP protocol which has proven itself to be extremely reliable and efficient. Our accountancy software interacts with the Open Banking service over REST.
As can be seen in the diagram, our Truelayer service has been deliberately designed to be stateless, providing an API for Open Banking to retrieve data from Truelayer. Both the Open Banking service and accountancy software contains scheduled jobs for periodically retrieving data from Truelayer and our Open Banking service respectively.
Although splitting the Open Banking system into multiple services has many advantages, it also carries risks — all services have to be correctly running and all services which wish to talk to one another must share a common understanding of their interfaces. To ensure that the system always hangs together correctly, a number of black-box tests run on every code commit, which exercises the system in its entirety. For example, we have tests which execute the authorisation flow and check the results.
Our infrastructure also allows the definition of metrics and alerts to give us a good idea of system performance and any issues which occur respectively.
Flexibility through Configuration-Based Routing
To ensure the flexibility of our Open Banking solution, we chose to externalise the provider mappings for each bank. In this way, it becomes extremely easy to specify an alternative provider without needing to rebuild or modify the source code of the Open Banking service.
For example, the following configuration tells our service to use Truelayer for HSBC, Lloyds and Natwest banks:
// Mapping the internal bank representation to the provider we are using for obtaining datainternaltopprovidermappings:
HSBC: TRUELAYER
LLYODS: TRUELAYER
NATWEST: TRUELAYER
...
The flexibility our design gives is great, but comes at the cost of additional complexity, as we’re required to chain requests through each of our services when making requests. For example, the below diagram illustrates our authorisation flow:
This chaining is necessary as we don’t know before making our request which Open Banking provider will be used.
Isolating provider-specific details
An additional benefit of isolating the Truelayer-specific functionality within the Truelayer service is that the Open Banking service can be kept as simple as possible.
For example, at the time of writing, the Truelayer API only allows clients to download six months of bank statement data at one time. If additional data is needed, our Truelayer service makes several requests, merges the data into a single response and returns it to the Open Banking service. The Open Banking service is oblivious to this complexity.
Future Work
Although the full user journey now works within our accountancy software, we plan to continue refining the process to improve it from both a usability and technical perspective.
For example, we currently download transactions for all registered clients once a day using a scheduled job. Although this will generally work fine, clients often have a very large amount of data to download on initial sign up, which can take in excess of two minutes.
If many clients sign up on the same day, the scheduled job could take longer to execute than desired. To alleviate this issue, we intend to begin the initial download immediately after the client authenticates with their bank, hopefully resulting in a more uniformly distributed load on the services.
Conclusion
In this article, we’ve discussed our new Open Banking system which allows clients to have their bank statements imported automatically into our accountancy software.
Our solution was designed with flexibility in mind, aiming to make it as straightforward as possible to provide alternative implementations for different banking providers and to keep the Open Banking service simple by encapsulating complexity within the provider-specific services.
Although this flexibility is desirable, it was also more time consuming and expensive to develop. In this case, the additional expense was deemed worthwhile, but this should be considered on a case-by-case basis. If the probability of future changes being needed is small, it’s likely not worth the investment.
Further Reading
The Crunch development team have been hard at work over the last few years migrating away from our monolithic system to a string of microservices. Further reading on our approach to this can be found in the following articles:
Written by Danny Matthews, Java Developer at Crunch, currently working on improving our accountancy software for our clients.
Find out more about the Technology team at Crunch and our current opportunities here.
