Consumer Match Egress

Overview

After you have sent consent for a specific merchant partner indicating that a user has opted into Consumer Matching, you can start to receive consumer matches (CMXs) from Banyan. This can happen in a myriad of ways. You can read them from a Google Cloud Storage bucket owned by Banyan, or ask Banyan to forward those files on to another cloud storage provider like Amazon S3, Snowflake, or an SFTP server. In addition, you can register a webhook with Banyan to receive these events to your own server location in JSON format. Setting up your webhook can be done via the Banyan portal. See further documentation

Batch Output Details

Format Data will be written in Avro format which means each file will have one OCF definition. In addition, Banyan will be sending the full consumer match schema for each record.

Compatibility Data will have backwards transitive compatibility guarantees. This means the following will be true:

1. Any schema from the latest file will be supported by earlier versions of the data.
2. Adding new fields will mean that the field will be included in older output if requested with a null value.
3. If we ever had the need to release a breaking change, we would create a new "v2" path and would write to both outputs until you were able to switch to the new version.

File SizeData will be sent after a certain file size is hit or it has been a certain amount of time since data was last written to the /output path. Our current threshold is set to 10MB file or every 10 minutes. Scenarios below:

1. Enrichments start streaming at 00:00 and accumulate through 00:45 resulting in a 5MB file. If no new enrichments get written for you until 00:55, you will see a 5MB file in your bucket.
2. Enrichments start streaming at 00:00 and accumulate through 00:45 resulting in a 5MB file. Then, at 00:50 there are more enrichments that bring the total size to 15MB, a 10MB file will be present in the bucket with the other 5MB waiting 10 more minutes or for more enrichments to bring it to 10MB.

Heartbeat Files In order for you to know that Banyan's system is not down or in a degraded state, we have instituted a practice of sending schema-only heartbeat files throughout the day. This will happen at a minimum 4x per day (or up to 15x) depending on how often our services are restarted on that day. These files are easily recognizable as they will all be the same file size. Please ensure your process is aware of these files.

Duplicates Banyan does not have a uniqueness guarantee for our streaming to batch data. It is very common for streaming services to restart and resend data it has previously sent. We highly recommend building your integration with deduplication downstream. Duplicates will most likely not occur over a period of days, but will happen frequently within the same day.

Webhook Egress

By registering your webhook for CMX events from Banyan you can receive them as they occur passively to your desired location. The first step is registering them through our customer portal.

See here for the full webhook walkthrough