RESO Transport Workgroup

Push Replication Using Webhooks and the RESO EntityEvent Resource

RCP	28
Version	1.0.1
Authors	Joshua Darnell (RESO) Paul Stusiak (Falcon Technologies)
Status	RATIFIED
Date Submitted	April 2019
Date Approved	December 2023
Dependencies	Data Dictionary 2.0 EntityEvent Resource RESO Common Format
Related Links	RCP-027 Webhooks

Synopsis

The RESO EntityEvent Resource provides an efficient and streamlined way to replicate data using the interface of an append-only log.

This proposal adds the ability to push events from the EntityEvent Resource using webhooks.

Rationale

Rather than polling on the EntityEvent Resource for updates, it’s useful for data producers to push events to consumer APIs.

Proposal

When an EntityEventSequenceNumeric, ResourceName, ResourceRecordKey tuple has been published to the EntityEvent resource, providers MAY “push” events to data consumers for replication or other event sourcing applications.

In doing so, the replication process becomes simpler for the consumer, as they only need to receive EntityEvents from the producer and pick up the corresponding records from provider APIs. In general, this is preferable to having to poll continuously for new records.

Examples

The following examples show using the EntityEvent model along with RESO Common Format , OAuth2 bearer tokens, and HTTP POST requests.

RESO Common Format supports both single and multi-valued payloads.

1. Producer Sends a Single EntityEvent to the Consumer API

REQUEST

POST https://example.api.com/webhooks
  Content-Type: application/json
  Authorization: Bearer <token>

{
  "@reso.context": "urn:reso:metadata:2.0:resource:entityevent",
  "EntityEventSequence": 1101,
  "ResourceName": "Showing",
  "ResourceRecordKey": "235021"
}

RESPONSE

HTTP/2 200
Content-Type: application/json

2. Producer Sends a Batch of EntityEvent Records to the Consumer API

REQUEST

POST https://example.api.com/webhooks
  Content-Type: application/json
  Authorization: Bearer <token>

{
  "@reso.context": "urn:reso:metadata:2.0:resource:entityevent",
  "value": [{
    "EntityEventSequence": 1101,
    "ResourceName": "Property",
    "ResourceRecordKey": "235021"
  }, {
    "EntityEventSequence": 1111,
    "ResourceName": "Media",
    "ResourceRecordKey": "ABC123"
  }]
}

RESPONSE

HTTP/2 200
Content-Type: application/json

3. Producer Sends a Batch of EntityEvent Records to a Consumer API With Retry-After

REQUEST

POST https://example.api.com/webhooks
  Content-Type: application/json
  Authorization: Bearer <token>

{
  "@reso.context": "urn:reso:metadata:2.0:resource:entityevent",
  "value": [{
    "EntityEventSequence": 1101,
    "ResourceName": "Property",
    "ResourceRecordKey": "235021"
  }, {
    "EntityEventSequence": 1111,
    "ResourceName": "Media",
    "ResourceRecordKey": "ABC123"
  }]
}

RESPONSE

HTTP/2 429
Content-Type: application/json
Retry-After: 60

Since Retry-After: 60 is present in the response header, the producer should wait 60 seconds before making another request.

4. Producer Sends a Batch of EntityEvent Records to a Consumer API with Optional EntityEventSource Header

REQUEST

POST https://example.api.com/webhooks
  Content-Type: application/json
  Authorization: Bearer <token>
  EntityEventSource: IDX Feed 123

{
  "@reso.context": "urn:reso:metadata:2.0:resource:entityevent",
  "value": [{
    "EntityEventSequence": 1101,
    "ResourceName": "Property",
    "ResourceRecordKey": "235021"
  }, {
    "EntityEventSequence": 1111,
    "ResourceName": "Media",
    "ResourceRecordKey": "ABC123"
  }]
}

RESPONSE

HTTP/2 200
Content-Type: application/json

Requirements

Consumers MUST implement an API endpoint that can receive EntityEvents.
Consumers MUST establish authorization with the provider, using an OAuth2 bearer token.
Producers MUST use an HTTP POST request to the consumer’s API with an EntityEvent payload and the given credentials.

Polite Behavior

It’s important that consumers and producers practice ‘polite behavior’:

If the producer decides the consumer is taking “too long” to process a request, they MAY choose to abandon it and call back at a later time.
If the producer is sending data to the consumer too quickly, the consumer MAY respond with an HTTP 429 response code, indicating the producer should halt the transfer of data to the consumer.
The consumer MAY provide a Retry-After header, either specifying a date or duration in seconds, after which the producer will attempt to resume transfer of data to the consumer.
If the producer fails after some number of retries, to be determined by the producer, they MAY stop the flow of data to the consumer. To resume push from the producer, the consumer MAY have to POST data to the producer’s endpoint to resume the feed.

Polite Behavior is especially important when the consumer is initializing a new replication feed, as there may be a large number of EntityEvents to process and additional work to be done on the consumer before subsequent EntityEvents may be processed. Consumers SHOULD accept and queue events rather than blocking and trying to process them as they arrive.

Consumer Labels for EntityEvent Sources

Optionally, consumers may choose to provide a custom identifier that will allow them to distinguish between feeds from multiple sources from the same producer.

This identifier will be passed in the headers as “EntityEventSource,” if present, and is assumed to be String (255).

Impact

Producers MAY push events from the EntityEvent Resource, specified in RCP-027, using POST requests to the consumer’s API.

The expected authorization mechanism for doing so will be long-lived bearer tokens provided to the producer by the consumer. Consumers are expected to maintain APIs that can respond in a “reasonable” amount of time.

This proposal doesn’t introduce any new technologies since the RESO Web API already uses OAuth2 bearer tokens and HTTP requests.

Compatibility

Compatible with Data Dictionary 2.0+ and JSON Web API. Providers SHOULD also host an EntityEvent Resource so consumers may replay events as needed.

Certification Impact

As this proposal is optional, there is no certification impact at this time.

Original Proposal

Download PDF

This site is open source. Improve this page.