Batch Retrieval

Google Cloud Storage (GCS)

Overview

Banyan supports sending and receiving batch data at rest in Google Cloud Storage. We provide roles for reading your enriched transaction data and have some commonly used practices to make things easy to test your integration seamlessly.

Our data is held in region: us-central1 (Iowa) and retention is set to 10 years by default on the bucket.

Folder Structure

Within your GCS bucket there will be an /output folder that contains all of your enriched transaction data from Banyan.

Output
Currently Banyan has two different data products: Enrich and Confirm. The /offer path within the output folder contains the data product for Confirm. For enriched transactions you want to set up your ingestion script to pull from only the /enrich folder.

  • Example Path: gs:test-bucket/output/enrich/2022-07-26/
  • Example File: 2022-07-26T16:23:09Z+22837398682.avro

For a full understanding of fields in the enriched transaction, see documentation here.

AWS S3


Overview

Banyan supports data retrieval from S3. If you have already set up a bucket with us for data ingestion, we will use that same bucket to share your enriched data back.

If you already use S3 to send Banyan data, we will update your role to now have read permissions on the /output directory.

Folder Structure

Within your S3 bucket you will see an /output file path that contains all of your enriched transactions.

Currently Banyan has two different data products: Enrich and Confirm. The /offer path within the output folder contains the data product for Confirm. For enriched transactions you want to set up your ingestion script to pull from only the /enrich folder.

  • Example Path: s3:test-bucket/output/enrich/2022-07-26/
  • Example File: 2022-07-26T16:23:09Z+22837398682.avro

Snowflake


Overview:

Banyan will share processed data with you through a secure view. You can access this data directly for analysis.

Steps:

  1. Access Shared Data:

    • Banyan will share a secure view with your account. Query the shared data as follows:
      SELECT * FROM <banyan_shared_view>;
      
  2. Continuous Updates:

    • Banyan is writing files and flushing them every hour or if they have hit XX MBs. This means that data can be updating throughout the day and not all at once. Oftentimes, merchant data comes in batches, so your matches to that data will in essence also be batched.

Actions:

  • Integrate Banyan’s shared data into your analytics workflows using tools like Looker, Tableau, etc.

Credentials & Security

  • Banyan AWS US East Account Locator: GTB18971
  • For other regions or cloud platforms, contact Banyan support.
  • Data is shared securely via direct shares and secure views, ensuring compliance with data privacy standards.