Guidance on metrics

Introduction

The following notes on metrics are gathered from Subject Matter Expert guidance and responses to participant questions. 

Performance requirements for ADRs

There are no performance requirements for Accredited Data Recipients (ADRs). Performance requirements apply only to Data Holders (DHs).

ADR arrangement revocation endpoint

If an ADR's CDR arrangement revocation endpoint performs unreliably, the DH is required to take reasonable measures to notify the ADR of the consent withdrawal.

Transaction Per Second (TPS) requirements for Data Holders

TPS upper threshold is a limit, not a requirement

The TPS requirements stated in the Consumer Data Standards (CDS) give an upper threshold for the number of transactions so that Data Holders (DHs) can develop their implementations adequately. A DH does not have to create infrastructure capable of meeting the upper threshold. A DH needs to create infrastructure adequate to meet the needs of its customer base.

Average TPS and Peak TPS metrics

For Average TPS metrics and Peak TPS, hourly transactions should not figure in the calculation. The Data Standards Body (DSB) recommends aggregating over no more than a minute.

Metrics sessionCount

In the Get Metrics API response schema, a session is counted as the successful provisioning of an access token. The sessionCount is the number of times an access token has been provided.

Reporting exemptions for incorrect API calls

Incorrect API calls are not considered part of metrics reporting as there is no disclosure of CDR data when the APIs are not defined by the CDS.

Inclusion of rejected API calls in Metrics

As error responses should be timely for a client, it is appropriate to include the response times for rejections in the average response metrics.

Availability metrics

Outage disaster recovery

If a DH has a major outage, declares a disaster recovery situation and takes many hours to recover, then under the CDS, this is a breach of the non-functional requirements. The CDS discuss only the level of availability and make no comment on the cause of any unplanned outages.

External API monitoring

A DH can decide to use external API monitoring to infer the availability of their CDR solution. The DH must be confident that it can infer unavailability from errors or scheduled outages. External monitoring may be only part of the solution for DHs monitoring availability. Each DH must decide, based on their architecture, how far into the application stack internal monitoring is required.

Measuring Authorisation Endpoint availability using an unregistered client

It is acceptable to measure availability of the (Authorisation Endpoint) by using an unregistered ADR client, and if the system responds with an error thereby blocking the request to proceed, this implies that the API is functional.

Get Metrics API cannot be shared across multiple brands

The GET Metrics API is tied to the record held in the Register for a Data Holder Brand. 

Every DH brand entry in the Register is considered a separate DH brand for the purposes of measurement of NFRs and reporting via the GET Metrics API.

Consequently each registered master brand must individually support the NFRs.

Amendment Flow

An amendment flow and an initial authorisation flow are reported in the same way in Get Metrics. Abandonment data is to be reported for either. 

Unauthenticated Endpoints

In NFRs, there is a distinction between reporting of authenticated and unauthenticated endpoints. Unauthenticated endpoints include the PRD, status and outage APIs.  The client is not known or authenticated.

DCR and InfoSec endpoints are publicly available, but they are still client authenticated and considered to be part of the authenticated side of the CDR implementation.

Traffic Thresholds for Customer Present Requests

When defining which 'customer-present' requests to measure, the traffic thresholds apply to both resource and authorization requests. Authorization traffic includes all information security endpoints bar dynamic client registration.  This is supported by the Consumer Experience requirements for authorization, as the customer is present for majority of the information security endpoint requests.

OnceOff Authorisation Metrics

To facilitate the specification of the duration for consent, a DH is required to support an additional claim in the authorisation request object named sharing_duration. If the sharing_duration value is zero, absent or any value less than 24 hours, then onceOff access is assumed.

currentDay, previousDays and previousMonths

The metrics schemas, such as AvailabilityMetricsV2, AuthorisationMetricsV2, PerformanceMetricsV3 and InvocationMetricsV3, include currentDay, previousMonths and previousDays fields.

currentDay and previousDays

In PerformanceMetrics, the currentDay field contains an array of 24 hourly metrics. The previousDays field contains an array of up to seven previousDay elements. Each previousDay element represents a day, in the same format as the currentDay field: an array of 24 hourly metrics. 

{
  "aggregate": {
    "currentDay": "string",     // this is a single 'day' value
    "previousDays": [           // this is an array of previous 'day' values
      "string"
    ]
  },
  "highPriority": {
    "currentDay": [            // this is an array of 24 'hour' values for the current day (compare this array to the above single "string" value). "Array of contiguous hourly metrics for the current day..."
      "string"
    ],
    "previousDays": [          // this is an array of up to 7 previous days (compare this array to the above single day array value). "Percentage of calls within the performance threshold for previous days..."
      [                        // this is an array of 24 hourly values (24 items for each previous day entry). "Array of contiguous hourly metrics for the specified day..."
        "string"               // this is the value of an hour. "Percentage of calls within the performance threshold for the specified hour..."
      ]
    ]
  },

Timezone and day change

Timezone is at the Data Holder discretion, however AEST is preferred, and the timezone should be maintained consisently.

CurrentDay data is reported from midnight.
Previous day data is reported as data from the previous 24 hours.  
Current month is reported as data from the first of the month.
Previous month is reported as data from previous whole month.

Reporting when no data available

If there is no data available for previous days and previous months, these fields can be omitted.

Alternatively, they can be populated with zeroes, however this might be interpreted as indicating a zero value was measured. 

Secondary Data Holder Metrics

To fulfill API data requests made by an ADR, a DH may be required to make further API requests to Secondary Data Holders (SDH). In such cases, the performance metrics for individual API calls made to the SDHs are reportable under the CDS.  

GetMetrics is aligned to the NFRs and reports the Primary Data Holder (PDH) and Secondary Data Holder (SDH) performance requirements individually.  The PDH average response times should exclude SDH times and vice versa.
Example formula:

Primary Data Holder (PDH):

Average Response Time = response time taken by PDH/ number of API calls received by the PDH.

Secondary Data Holder (SDH):

Average Response Time = response time taken by SDH/ number of API calls received by the SDH.

Frequency of Calls to Get Metrics

The Consumer Data Right Register will call DHs once per day, around 5:00AM AEST, every day.

Adhoc calls may be required however this would be infrequent and would not occur en masse.

Best Effort for Unattended traffic during high traffic periods

The phrase "best effort", in the CDS, Traffic Thresholds section, means that there are no applicable metrics for Unattended traffic during high traffic periods.