The AWS S3 sync allows you to export your data into an S3 bucket in your own AWS account. Warehouse Sync supports multiple platforms including Snowflake, BigQuery, Redshift, and AWS S3. The S3 sync uploads a full load of your customer data with every run, so the latest partition uploaded will contain all of your account's data. The latest unload reflects whatever schema is configured on your warehouse sync page, and the data is stored in Parquet format.

Note: The warehouse sync currently includes issue titles and metadata, not full message content.

Common use case: You can use the S3 sync to generate recurring account activity reports by running SQL queries against the exported data.

Pylon uses IAM credentials to temporarily assume a role in your AWS account and write data into your S3 bucket. Below are the steps required to set up the S3 sync.

AWS Setup

We provide an AWS CloudFormation template that automatically creates the AWS resources required for S3 warehouse sync.

Otherwise, the manual resource setup is detailed below.

IAM Policy

Create an IAM policy that allows Pylon to write into your S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::{your-bucket-name}",
                "arn:aws:s3:::{your-bucket-name}/*"
            ]
        }
    ]
}

IAM Role

Create a new IAM role with an AWS account trusted entity type. Select Another AWS account and enter the Pylon service account ID:

825765419603

We also require an external ID, which you can supply under Options. This external ID is a key that you will supply to our client for an extra layer of security.

Attach the policy above to this role.

You also need to ensure that the IAM role has STS assume-role permissions so that the Pylon account can assume this role. The trust policy should look like the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::825765419603:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "{your-external-id}"
                }
            }
        }
    ]
}

Now you are ready to create an S3 sync in Pylon.

Pylon Setup

1. Navigate to the Apps directory page and install the S3 App.

2. Click Create to create a new sync and supply the resources that you set up above.

3. In External S3 Bucket Name, enter the bucket name only. Do not include a path or prefix such as bucket-name/pylon.

4. Save the settings, then toggle on the sync so that it is connected and enabled.

Pylon writes data under the root of the configured bucket using date- and sync-timestamp-based folders. The path prefix is not configurable.