RESO EntityEvent Resource and Replication Model

RCP	27
Version	2.0.2
Authors	Joshua Darnell (RESO) Paul Stusiak (Falcon Technologies)
Status	RATIFIED
Date Submitted	January 2019
Date Approved	December 2023
Dependencies	Data Dictionary 2.0 EntityEvent Resource
Related Links	Push Events using Webhooks

Synopsis

The RESO EntityEvent Resource provides an efficient way to represent event streams and replicate data by using the interface of an append-only log.

Rationale

One of the most common business cases for real estate data is the replication of listing and other data between producers and consumers.

The current state of the art is to use modification timestamps, long polling, and state inference to synchronize data. This places an additional burden on the data consumer to know the state of the producer and coordinate timestamps across systems and vendors. It also results in the need to frequently resynchronize data, as it can become difficult or impossible to resolve timestamps or reconcile the current state of a given system.

Proposal

A new Data Dictionary resource named EntityEvent has been created.

This resource provides a field called EntityEventSequence, which is the logical timestamp of a given event, as well as the ResourceName and ResourceRecordKey of the item that was created, updated, or deleted. An optional field called ResourceRecordUrl allows systems to provide opaque URLs that can be used to fetch records for each EntityEvent.

The logical timestamp is an event identifier that denotes that a business event occurred. Business events are rules or actions that represent the business logic of the resources of the system.

Examples of business events could be a listing price change, a listing status change, an phone number change for an agent, or the addition or deletion of a photo to any other resource that has a media object. These business events and the associated logic exists in current systems implementing the Web API today. By representing these events in a consistent way, eventual record state consistency can be achieved.

EntityEventSequence is a durable, immutable, monotonic identifier that preserves the order that events occur in a system. It can only increase in value.

Given any EntityEventSequence values e1 and e2, if e1 < e2 we can say e1 occurred before e2. For example, let e1 = 200 and e2 = 1001.

Entity Events

The EntityEvent resource provides a service that returns event records as defined in this proposal.

• A producer MUST provide a Resource named EntityEvent.
• An EntityEvent record has three required fields, EntityEventSequence, ResourceName, and ResourceRecordKey, which uniquely identify the given record.
• The data type of EntityEventSequence is the positive portion of int64.
• The data type of ResourceName is string. It is the well-known name for the resource type.
• The data type of ResourceRecordKey is the unique identifier for a specific record within the given resource and MUST be a string.
• The EntityEventSequence MUST conform to the property that is immutable and monotonic.
• EntityEventSequence numbers MAY be hidden from feeds as long as at least one later record contains a final record for that resource name and key.

The EntityEvent resource MAY contain all events that occur in a system or may contain events limited to those events or resources that a consumer has permission to view.

The existence of an identified resource in EntityEvent does not change the visibility of the resource that generated the underlying record. That is controlled by the producer and the permission model they have implemented.

Workflows

Certain limitations are applied to the normal workflow of producing and consuming entities.

Producers

Producers MUST order EntityEvent records in the logical order that they occurred. EntityEventSequence must have the property that it is unique, always increases in value, and cannot change.

Requesting the same EntityEventSequence MUST result in the same ResourceName and ResourceRecordKey.

When possible, providers should order events to simplify referential integrity for data consumers so they can replay events without underlying business knowledge of internal system relationships.

For example, the addition of a Property Resource event should occur before any Media Resource events associated with the Property appearing in the Events Resource.

Consumers

Consumers use standard workflows for replication.

Example

A consumer wishes to get all the events after the EntityEventSequence of 100.

GET /EntityEvent?$filter=EntityEventSequence gt 100

{
  "@odata.context": "/EntityEvent?$filter=EntityEventSequence gt 100",
  "value": [
    {
      "EntityEventSequence": 101,
      "ResourceName": "Member",
      "ResourceRecordKey": "21"
    },
    {
      "EntityEventSequence": 103,
      "ResourceName": "Property",
      "ResourceRecordKey": "539"
    },
    {
      "EntityEventSequence": 110,
      "ResourceName": "Media",
      "ResourceRecordKey": "1239"
    }
  ]
}

Replication

One of the business cases for the EntityEvent Resource is replication. This is an optional, but important, part of the proposal. This section outlines related considerations.

Business Cases

There are two types of consumers and two cases comprise the workflow of a consumer.

• Most consumers will be synchronizing with the most current state of the system.
• A much smaller number of consumers will be collecting state changes throughout history to create analytics.

Initial Synchronization

The consumer is gathering each EntityEvent record for the first time. Based on rules created by the producer or by the consumer, the consumer starts at the lowest value EntityEventSequence number and processes each record.

Example: Get All Records a Consumer Has Access To

Request

GET /EntityEvent?$filter=EntityEventSequence ge 0

This will return the first EntityEventSequence the client has access to. This could be either 0 or 10,000, etc.

If the provider supports Web API Core 2.1.0+ or Data Dictionary 2.0+, @odata.nextLink will be present in the response until all EntityEvent data is returned.

Ongoing Synchronization

The consumer has previously been in synchronization and needs to ‘catch up’ with the current state of the system.

Handling Record Visibility

Based on business rules, EntityEvent records may change the availability and visibility for consumers based on the role of a consumer.

For example, a consumer who had a role to receive IDX listings may not be permitted to only view listings that have a status of ‘Active’.

A consumer with this role would see an event that changes status and would then attempt to retrieve the record from the producer only to receive no record.

This can be used by the consumer to know that the ResourceRecordKey they are attempting to retrieve is no longer part of the permitted records for their system and could take appropriate steps to remove that ResourceRecordKey from their visible record set.

Consumer Visibility Workflow

A consumer asks for the current set of EntityEvent records.

In this set is an event that changes a record that the consumer has from visible to not visible - we can think of this as a consumer delete event.

The consumer does not know what has happened to the record, only that the record has changed state.

When the consumer makes a normal query against the appropriate Resource, they will get a “No Record Found” (HTTP 404) response and can infer that the record and the child resources should be deleted.

Based on earlier sections of this proposal, the child records should have appeared at a lower EntityEventSequence value than the parent record.

This allows the consumer to remove references rather than deal with cascading deletes.

Impact

Producers should expose a new resource that implements the requirements of the proposal.

Compatibility

As new functionality, this does not affect existing standards.

Certification

Additional testing rules beyond those in the Data Dictionary:

• Check metadata for the existence of an EntityEvent with required fields: EntityEventSequence, ResourceName, and ResourceRecordKey.
• Additional fields in an EntityEvent Resource will be ignored by the compliance test.
• Confirm that data is returned for a GET on the EntityEvent Resource.
• Confirm that for a sample set that any two records have sequentially increasing EntityEventSequence values.
• Confirm that for a sample set of EntityEvent records, that all records only contain ResourceName items that are defined in the metadata.
• Confirm that the EntityEvent Resource response is well formed.
• Confirm that for each record in a sample set of EntityEvent records, that either a record with ResourceName and ResourceRecordKey is returned OR an HTTP 400 response value indicating the records are no longer available.
• Test sequence order.
• Confirm that requesting the same EntityEventSequence results in the same combination of ResourceName and ResourceRecordKey.

Original Proposal

Download PDF