Guidance on metrics

Introduction

The following notes on metrics are gathered from Subject Matter Expert guidance and responses to participant questions. 

Performance requirements for ADRs

There are no performance requirements for Accredited Data Recipients (ADRs). Performance requirements apply only to Data Holders (DHs).

ADR arrangement revocation endpoint

If an ADR's CDR arrangement revocation endpoint performs unreliably, the DH is required to take reasonable measures to notify the ADR of the consent withdrawal.

Transaction Per Second (TPS) requirements for Data Holders

The TPS requirements stated in the Consumer Data Standards (CDS) give an upper threshold for the number of transactions so that Data Holders (DHs) can develop their implementations adequately. It is not an expectation of infrastructure sizing. If a DH does not have a customer base that would necessitate such transaction loads, then it is acceptable for a DH to implement a solution that suits its needs.

Metrics sessionCount

In the Get Metrics API response schema, a session is counted as the successful provisioning of an access token. The sessionCount is the number of times an access token has been provided.

Calculating average TPS and peak TPS

To calculate average Transactions Per Second (TPS) and peak TPS, the Data Standards Body (DSB) recommends aggregating over no more than a minute.

Outage disaster recovery and availability metrics

If a DH has a major outage and needs to declare a disaster recovery situation and takes many hours to recover, then under the CDS, this is a breach of the non-functional requirements. The CDS discuss only the level of availability and make no comment on the cause of any unplanned outages.

Reporting exemptions for incorrect API calls

Incorrect API calls are not considered part of metrics reporting as there is no disclosure of CDR data because the APIs are not defined by the CDS.

Inclusion of rejected API calls in Metrics

As error responses should be timely for a client. It is appropriate to include the response times for rejections in the average response metrics.

Calculating Average TPS and Peak TPS metrics

For Average TPS metrics and Peak TPS, hourly transactions should not figure in the calculation.

Monitoring availability metrics

A DH can decide to use external API monitoring to infer the availability of their CDR solution. The DH would need to be confident that it can infer unavailability from errors or scheduled outages. It is also likely that external polling would be only part of the solution for DHs monitoring availability. Each DH must decide, based on their architecture, how far into the application stack internal monitoring is required.

Get Metrics API cannot be shared across multiple brands

The GET Metrics API is tied to the record held in the Register for a Data Holder Brand. 

Every DH brand entry in the Register is considered a separate DH brand for the purposes of measurement of NFRs and reporting via the GET Metrics API.

Availability measurement

It is acceptable to measure availability of the (Authorisation Endpoint) by using an unregistered ADR client, and if the system responds with an error thereby blocking the request to proceed, this will infer that the API is functional.