Data Sources > Amazon S3

Amazon S3

This page covers how to use an Amazon S3 bucket as a data source in Rockset. This includes:

  • Creating an Amazon S3 integration to securely connect buckets in your AWS account with Rockset.
  • Creating a collection which syncs your data from a Amazon S3 bucket into Rockset.

For the following steps, you must have access to an AWS account and be able to manage AWS IAM policies and IAM users within it. If you do not have access, please invite your AWS administrator to Rockset.

Create an S3 Integration

The steps below show how to set up an Amazon S3 integration using AWS Cross-Account IAM Roles and AWS Access Keys. An integration can provide access to one or more S3 buckets within your AWS account. You can use an integration to create collections that sync data from your S3 buckets.

Step 1: Configure AWS IAM Policy

  1. Navigate to the IAM Service in the AWS Management Console.
  2. Set up a new policy by navigating to Policies and clicking Create policy. If you already have a policy set up for Rockset, you may update that existing policy. For more details, refer to AWS Documentation on IAM Policies. AWS IAM Policies
  3. Set up read-only access to your S3 bucket. You can switch to the “JSON” tab and paste the policy shown below. You must replace <your-bucket> with the name of your S3 bucket. If you already have a Rockset policy set up, you can add the body of the Statement attribute to it.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:List*",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>",
        "arn:aws:s3:::<your-bucket>/*"
      ]
    }
  ]
}
  1. Save the newly created or updated policy and give it a descriptive name. You will attach this policy to a user or role in the next step.

Why these Permissions?

  • s3:List* ⁠— Required. Rockset uses the s3:ListBucket and s3:ListAllMyBuckets permissions to read bucket and object metadata.
  • s3:GetObject — Required to retrieve objects from your Amazon S3 bucket.

Advanced Permissions

You can set up permissions for multiple buckets, or some specific paths by modifying the Resource ARNs. The format of the ARN for S3 is as follows: arn:aws:s3:::bucket_name/key_name.

You can substitute the following resources in the policy above to grant access to multiple buckets or prefixes as shown below:

  • All paths under in mybucket/salesdata:
    • arn:aws:s3:::mybucket
    • arn:aws:s3:::mybucket/salesdata/*
  • All buckets starting with “sales”
    • arn:aws:s3:::sales*
    • arn:aws:s3:::sales*/*
  • All buckets in your account
    • arn:aws:s3:::*
    • arn:aws:s3:::*/*

For more details on how to specifiy a resource path, refer to AWS documentation on S3 ARNs.

Step 2: Configure Role / Access Key

There are two mechanisms by which you can grant Rockset permissions to access your AWS resource. Although Access Keys are supported, Cross-Account roles are strongly recommended as they are more secure and easier to manage.

AWS Cross-Account IAM Role

The most secure way to grant Rockset access to your AWS account involves giving Rockset’s account cross-account access to your AWS account. To do so, you’ll need to create an IAM Role that assumes your newly created policy on Rockset’s behalf. You’ll need information from the Rockset console to create and save this integration.

  1. Navigate to the IAM service in the AWS Management Console.

  2. Setup a new role by navigating to Roles and clicking Create role. Note: if you already have a role for Rockset set up, you may re-use it and either add or update the above policy directly.

AWS IAM Roles

  1. Select Another AWS account as type of trusted entity, and tick the box for Require External ID. Fill in the Account ID and External ID fields with the values (Rockset Account ID and External ID respectively) found on the create integration page of console (under the Cross-Account Role Option). Click to continue.

AWS IAM Create Role

  1. Choose the policy created for this role in Step 1 (or follow Step 1 now to create the policy if needed). Then, click to continue.

AWS IAM Roles Attach Policy

  1. Optionally add any tags and click Next. Name the role descriptively, e.g. 'rockset-role’, and once finished record the Role ARN for the Rockset integration in the console.

AWS Access Key

Navigate to the IAM service in the AWS Management Console.

  1. Create a new user by navigating to Users and clicking Add User. AWS IAM Users

  2. Enter a name for the user and check the Programmatic access option. Click to continue. AWS IAM Create User

  3. Choose Attach existing policies directly then select the policy you created in Step 1. Click through the remaining steps to finish creating the user. AWS IAM Attach Policy

  4. When the new user is successfully created you should see the Access key ID and Secret access key displayed on the screen.

AWS IAM Access Key

  1. Record both these values in the console.

Create a Collection

When creating a collection in Rockset, you can specify an S3 path (see details below) from which Rockset will ingest data. Rockset will continuously monitor for updates and ingest any new objects. Deleting an object from the source bucket will not remove that data from Rockset.

In the Rockset console, you can create a collection from Workspace > Collections > Create Collection.

Create Collection

Using the CLI, you can create a collection by running the following.

$ rock create collection my-first-s3-collection \
    s3://my-bucket/my-path-1 s3://my-bucket/my-path-2 \
    --integration=aws-rockset-readonly

Collection "my-first-s3-collection" was created successfully.

Note that these operations can also be performed using any of the Rockset client libraries.

Specifying S3 Path

You can ingest all data in a bucket by specifying just the bucket name or restrict to a subset of the objects in the bucket by specifying an additional prefix or pattern.

By default, if the S3 path has no special characters, a prefix match is performed. However, if any of the following special characters are used in the S3 path, it triggers pattern matching semantics.

  • ? matches one character
  • * matches zero or more characters
  • ** matches zero or more directories in a path
  • {myparam} matches a single path parameter and extracts its value into a field in _meta.s3 named “myparam”
  • {myparam:<regex>} matches the specified regular expression and extracts its value into a field in _meta.s3 named “myparam”

A few examples are shown below to explain exactly how the patterns can be used.

  • s3://bucket/xyz - uses prefix match, matches all files in the folder xyz>
  • s3://bucket/xyz/t?st.csv - matches com/test.csv but also com/tast.csv or com/txst.csv in bucket.
  • s3://bucket/xyz/*.csv - matches all .csv files in the xyz directory in bucket.
  • s3://bucket/xyz/**/test.json - matches all test.json files in any subdirectory under the xyz path in bucket.
  • s3://bucket/05/2018/**/*.json - matches all .json files underneath any subdirectory under the /05/2018/ path in bucket.
  • s3://bucket/{month}/{year}/**/*.json - matches the pattern according to the above rules. In addition to this, it extracts the value of the matched path segments {month}, {year} as fields of the form _meta.s3.month and _meta.s3.year associated with each document.
  • s3://bucket/{timestamp:\d+}/**/*.json - matches the pattern according to the the above rules. In addition to this it extracts the value of the timestamp path segment if it matches the regular expression \d+ and places the value extracted into _meta.s3.timestamp associated with each document.