CDR metrics and reporting by Data Holders – Consumer Data Standards Australia

Introduction

There are reporting requirements under the Consumer Data Right (CDR). The CDR rules outline the information that Data Holders and Accredited Data Recipients must record and retain, as well as requirements for reports that CDR participants must prepare and provide to the ACCC and OAIC.

CDR reporting is intended to provide insight into consumer activity in the ecosystem, including the systems involved in data sharing, their performance and availability.

To facilitate this reporting under rule 8.11(f), the Consumer Data Standards include the Get Metrics endpoint. The Get Metrics endpoint can be used by the ACCC to obtain a subset of the information, that they periodically collect, to allow for real time monitoring of the health of the CDR ecosystem.

This guidance aims to provide a consolidated source of reference for how the metrics included in the Get Metrics endpoint are to be interpreted and calculated. Where the technical definitions provided are also applicable for more general reporting under the CDR Rules, this is stated.

Definitions

Where this guidance refers to Standards, it is referring to the Consumer Data Standards.

Sections of this guidance refer to a Data Holder, Primary Data Holder and a Secondary Data Holder. The Secondary Data Holder concept is currently only applicable to the Energy sector, where an additional service is involved in the data sharing process. In the Banking and Telco sectors, the terms Data Holder and Primary Data Holder are equivalent.

Collection of operational data

There are many reasons a data holder should collect and analyse data about their CDR services, including:

To monitor and maintain a stable and performant service for operational purposes
To support technical enquiries and to demonstrate operational compliance for audit purposes under Rule 9.6
To fulfil Rule 9.4 requirements, including reporting of complaints, requests, and refusals
To support the Get Metrics endpoint required by the Standards in order to provide real-time insight to the Regulator.

This guidance focuses on the metrics included in the Get Metrics endpoint and extends to the definition of requests (invocations) and refusals for the purposes of Rule 9.4 reporting.

Measurement layer

The scope of the Get Metrics endpoint is limited to requests received at what a data holder may determine to be their measurement layer

As referred to in the Standards, the measurement layer is expected to be a technical layer or capability as close to the client (an application making requests) as practicable, where endpoint activity can be recorded appropriately.

Any requests that do not reach the measurement layer, for instance due to the caching of a successful response, or a 404 response for a path that is not supported at that layer, would not need to be included in the invocation count or related metrics derived from requests reaching this layer.

As noted above, this guidance is applicable only to the calculation of metrics for the Get Metrics endpoint. Please note that the reporting obligations in the Rules require information from sources outside the endpoint measurement layer to be retained for other reporting purposes.

Non-functional Requirements

Some of the metrics described in this guidance relate to non-functional requirements of the Standards.

The CDR Rules require all valid CDR Requests to be responded to, which equates to perfect availability, performance, and scalability. Operationally, this may not be achievable and so non-functional requirements in the Standards provide relief for Data Holders from needing to provide a CDR implementation with unreasonable operational service requirements.

Invocations

Standards reference: Invocation metrics

Invocations are all requests received at the measurement layer for endpoint paths defined in the Standards, including all infosec endpoints such as userinfo, authorise and token introspection. These may be included in an OpenID Configuration for the purposes of CDR.

For clarity, requests received outside the scope of the Standards and its related endpoints, or invalid requests blocked by systems above the measurement layer, such as a WAF or CDN, are not expected to be included in the definition of invocations for the purposes of the Get Metrics endpoint.

All requests received in the measurement layer are considered invocations, even where an error is returned according to the Standards. This also includes Shared Responsibility invocations that are passed on to a Secondary Data Holder.

No requests received during a planned or unplanned outage are expected to be included in any Get Metrics reporting, but the duration of unplanned outages must be considered in Availability reporting.

In summary, all invocations are expected to be:

Included in the invocations metric
Included in the averageResponse and performance metrics
Included in the averageTps and peakTps metrics
Included in Rule 9.4 reporting for the Number of requests received item

Errors

Standards reference: Error metrics

There are two main types of errors defined in the Standards:

The status code 4xx series of errors: error codes beginning with 4. 4xx errors generally relate to the format of an API request made by a client, such as requesting a page of results that does not exist. These are categorised as client-side errors. In the CDR, a data recipient software product is an example of a client making requests to data holders.
The status code 5xx series of errors: error codes beginning with 5. 5xx errors generally relate to processing a response, such as a database connection error. These are categorised as server errors. In the CDR, the data holder API endpoints would be considered server-side.

In Get Metrics v3, the errors property is intended to capture only server-side errors recorded in the measurement layer.

In Get Metrics v4 and v5, only server-side errors are to be recorded in the errors.aggregate property, but they must also be included under the corresponding HTTP error code property alongside all other 4xx and 5xx errors in their respective properties under errors.unauthenticated and errors.authenticated.

The response code 504 Gateway Timeout is a special type of 5xx server error code where an excessive response time may be recorded. Invocations resulting in this status code should be included in the errors count in Get Metrics but should be excluded from the Average Response time and Performance metrics as it may be expected to distort those results.

The Get Metrics endpoint has separate errors fields to report errors generated by the Primary Data Holder and the Secondary Data Holder. If a server error is generated by the Secondary Data Holder and passed on by the Primary Data Holder, it is still defined as an error. Errors should not be double counted, however, so these errors should only be reported in the Secondary Data Holder errors field and should excluded from the Primary Data Holder errors field.

Errors generated by requests made prior to an obligation date for an endpoint, or for voluntary data that are not being made generally available by the data holder should not be counted as server errors in Get Metrics.

In summary, status code 5xx series responses should be:

Included in the invocations count
Included in the averageResponse and performance metrics (except for code 504)
Included in the averageTps and peakTps metrics
Included in the errors property of Get Metrics v3
Included in the errors.aggregate property of Get Metrics v4 and v5
Included in the errors.unauthenticated and errors.authenticated properties of Get Metrics v4 and v5
Excluded from the rejection property of Get Metrics
Included in Rule 9.4 reporting as part of the Number of requests items

Status code 4xx series responses should be:

Included in the invocations count
Included in the averageResponse and performance metrics
Included in the averageTps and peakTps metrics
Excluded from the errors property of Get Metrics v3
Excluded from the errors.aggregate property of Get Metrics v4 and v5
Included in the errors.unauthenticated and errors.authenticated properties of Get Metrics v4 and v5
Included in Rule 9.4 reporting as part of the Number of requests items

Rejections

Standards reference: Rejection metrics

Rejections are defined in the Standards as the Number of calls rejected due to traffic thresholds [over time].

A rejection is communicated to the client by the response code 429 Too Many Requests. A rejection may be an appropriate response to an invocation that was recorded at the measurement layer, but where data was not disclosed as the request was deemed to be above a defined traffic threshold for the purposes of maintaining service performance and stability. Though these may be considered unsuccessful requests, they should not be recorded as server errors in Get Metrics.

The Get Metrics endpoint has separate rejections fields for the Primary Data Holder and the Secondary Data Holder. If the Secondary Data Holder rejects a request and this result is passed on by the Primary Data Holder, it is still a rejection. Rejections should not be double counted, however, so these rejections should only be reported in the Secondary Data Holder rejections field and should excluded from the Primary Data Holder rejections field.

By responding with code 429, the data holder is using their discretion to refuse to disclose data. As such, each of these responses must also be recorded as a refusal to disclose, according to Rule 9.4.

As an example, any requests received by a data holder because of a Distributed Denial of Service (DDoS) attack that are responded to with the 429 code should also be classified as refusals according to the applicable reasons defined in the rules:

for product data requests; rule 2.5(1) may apply
for consumer data requests made by consumers (rules not currently enlivened); rule 3.5(1)(b)
for consumer data requests made by accredited persons;
- where disclosure in response to the request would impact security, integrity or stability; rule 4.7(1)(b)
- where the requests have exceeded service-level thresholds; rule 4.7(1)(d).

In summary, status code 429 responses should be:

Included in the invocations count
Included in the averageResponse and performance metrics
Included in the averageTps and peakTps metrics
Excluded from the errors property of Get Metrics v3
Excluded from the errors.aggregate property of Get Metrics v4 and v5
at the Data Holders discretion as to whether they are Included or Excluded from the errors.unauthenticated and errors.authenticated properties of Get Metrics v4 and v5, but they must always be included in the rejections property
Included in the rejection property of Get Metrics
Included in rule 9.4 reporting as part of the Number of requests items
Included in rule 9.4 reporting as part of the Refusals item with respective reasons

Rejections that are recorded outside the measurement layer (such as in a separate cloud CDN or WAF service) are not expected to be included in Get Metrics, though records of these instances may still be required for Rules 9.3(1)(e) and 9.4(1)(d), or for monitoring and audit purposes.

Refusals

Reference: Rules, ACCC reporting forms

The Rules provide that data holders may refuse to disclose required consumer data in response to a request in some specified instances. These are known as refusals. A refusal is given in response to an invocation where a data holder has used their discretion to deny an otherwise valid request.

The relevant Rules that provide this discretion are:

Rule 2.5 - Refusal to disclose required product data in response to product data request
Rule 3.5 - Refusal to disclose required consumer data in response to consumer data request
Rule 4.7 - Refusal to disclose required consumer data in response to consumer data request

An instance of refusing to ask for authorisation is not considered a refusal for the purposes of Rule 9.4 reporting.

If a data holder refuses to share data because the Rules do not permit them to share data, these will not be considered as refusals that must be reported on, as they are prohibited by the Rules rather than a data holder exercising a discretion not to share. Examples of this include:

A data holder not sharing data as the as the non-disclosure option has been selected in accordance with rule 4A.10(6), or where a joint account holder has withdrawn or refused to give an approval to share data from that account under rule 4A.10(3).
A data holder not sharing data pursuant to a request made by a secondary user where the account holder has given an indication under rule 4.6A.

In addition, refusing to share data to an ineligible data recipient or refusing to share the data of an ineligible consumer would not be refusals as they would not be considered valid invocations.

Rule 2.5 requires refusals to map to common HTTP response codes, for the purpose of inform[ing] the requester of such a refusal. However it is not appropriate to treat all responses of a particular error code as a refusal, or even as the same type of refusal. For example, error code 404 or 422 responses due to requests that are invalid are not considered refusals. Because of the overlap between error response codes and refusals, refusals should be recorded separately from general error code logging, to ensure reasons for refusal can be accurately distinguished.

Availability

Standards reference: Availability Requirements

The Standards state an availability requirement of 99.5% per month.

To support the availability metric, the Standards define a period of unavailability as any period of time when any of the API end points defined in the standard is unable to reliably provide a successful response to an appropriately constructed request.

For clarity, a period of unavailability should be reported even where a data holder did not record any requests in that period. This means that an absence of requests cannot be used to determine that a period of unavailability did not occur.

Planned outages, where at least one week notice is provided through the Get Outages endpoint, or where they are conducted without notice to resolve a critical service or security issue, do not need to be factored into the Availability calculation as a period of outage.

All other outages (whether receiving requests in that period or not) must be included in the availability measurement as a period of outage.

To calculate availability, an appropriate formula would be:

(Total minutes in the period* - Minutes of outage) / Total minutes in the period*

*The period of measurement in the Get Metrics endpoint is monthly only but other periods may be requested for other reporting obligations and this formula would still be appropriate that purpose.

Performance

Standards reference: Performance Requirements

The Standards state a performance requirement of 95% of calls per hour responded to within a nominated threshold.

Performance metrics are calculated differently for Get Metrics versions prior to v5.

To report this value, the Performance property of Get Metrics prior to v5 requires the Percentage of calls within the performance thresholds, which should be calculated across all performance tiers combined. This continues to apply to the aggregate field in Get Metrics v5.

Shared Responsibility Requests

For Shared Responsibility invocations, the Primary Data Holder (PDH) and Secondary Data Holder (SDH) performance are reported separately.

The PDH average response time should exclude the time the SDH takes to process the invocation and vice versa.

An example formula showing how average response time could be calculated is below:

Average response time = response time taken by the Data Holder (if PDH, excludes SDH and vice versa) / number of invocations received by that Data Holder.

As a result, if the SDH fails to respond within their threshold, but the PDH responds within theirs, then that invocation should be treated as above the performance threshold for the SDH, but within the threshold for the PDH.

Get Metrics versions prior to v5

In Get Metrics prior to v5 and the aggregate field in v5:

To calculate values for the Performance metrics, an appropriate calculation would be to divide the total number of invocations within their respective tier thresholds for a given period, by the total number of invocations in the period.

Total invocations within the respective thresholds for the period* / Total invocations for the period*

To illustrate:

API tier	Invocations within threshold*	Total invocations*
Unauthenticated	980	1,000
High Priority	498	500
Low Priority	10	100
Unattended	120	400
Large Payload	0	0
Secondary Request	0	0
Large Secondary Request	0	0
Total	1,608	2,000
Total within threshold / Total invocations		(1,608/2,000)
Percentage of calls within threshold		0.804

*The period of measurement may be hourly (to be used for partial reports of the currentDay metric, for example) or daily (for full previousDays) and the formula would still be appropriate.

Get Metrics V5 and later

In Get Metrics v5 for the respective tier properties:

Total invocations within the respective thresholds for the period* / Total invocations in the respective tier for the period*

All versions of Get Metrics

If no requests are received in a performance tier in a certain period, then no requests would be considered to be below the performance threshold, therefore 100% performance for that tier, for that period, can be reported.

TPS

Standards reference: Average TPS metrics, Peak TPS metrics

The Standards require Transactions Per Second (TPS) details in the Get Metrics API schema properties AverageTPSMetrics and PeakTPSMetrics.

The intent of the Average TPS metric is to provide insight into the typical demand of a system over time. Peak TPS is intended to provide insight into the highest demand levels over time.

For the purposes of reporting TPS, the DSB expects logs for invocations recorded at the measurement layer to be aggregated to a maximum interval of one minute.

To calculate TPS at any point in time based on logs aggregated by minute, the formula would be the number of invocations in the reporting minute, divided by sixty seconds.

For Peak TPS in a day the formula would therefore be:

For the Peak minute of invocations in the day: Number of invocations in the specified minute / 60 seconds

For the reporting of average TPS the aggregation period is less of an issue as it would average out.
For implementation simplicity, however, it is not necessary to report on partial minutes for the average TPS currentDay field. Taking this into account the formulae for calculating average TPS are outlined below.

To provide the Average TPS for a partial day (for currentDay), the formula would be:

Number of invocations in completed minutes in the period/ (Number of minutes measured x 60 seconds)

To provide the Average TPS for a full day, the formula would be:

Number of invocations in the day / (24 hours x 60 minutes x 60 seconds)

If logs for unauthenticated and authenticated APIs are collected in two separate systems, the TPS from each system should be summed to provide the Average TPS for the whole implementation. The Peak TPS should be reported as the peak of the whole implementation.

Average response time

Standards reference: Get Metrics

The Get Metrics ‘averageResponse’ object has a description of ‘Average response time in seconds, at millisecond resolution, within each performance tier’.

To report this value for each tier and period (block), the total response time (in seconds) for all requests in that block should be divided by the total number of invocations in that block.

To calculate average response time, an appropriate formula would be:

Total response time (in seconds) in the block / Total invocations in the block

If there are no invocations recorded in a particular tier and period, there is no response time and the value provided should be zero (0).

Abandonments

Data holders are not expected to report on consents abandoned prior to the preIdentification phase.

The consent flow may be abandoned or otherwise exited at a number of stages prior to a consumer authorising or rejecting a request to share data. These exits are captured in the relevant fields listed in the AuthorisationMetricsV2 schema. See CDS Guide, Get Metrics V5 abandonment by stages for more information.

The purpose of the abandonment metric is to capture abandonments by CDR consumers during the Consumer Experience (CX) flow. The CX flow commences when a CDR consumer views the first user-facing page in the preIdentification phase. The first user-facing page appears at the data holder login screen when the CDR consumer arrives at the ‘Enter user identifier’ stage of the CX flow. Abandonments made during the CX flow must be included in Get Metrics reporting. That is, any abandonments in or after the preIdentification phase must be reported by data holders.

The ACCC is aware that ADRs conducting testing may make calls to the Pushed Authorisation Request (PAR) endpoint which are then abandoned. These abandonments do not need to be reported where they occur before the preIdentification phase. In this scenario, there may be no CDR consumer and/or a CDR consumer may not have viewed a user-facing page prior to the abandonment. These abandonments should be excluded from Get Metrics reporting.

Welcome to the Consumer Data Right Support Portal

CDR metrics and reporting by Data Holders Follow