Amazon DynamoDB

This page covers how to use an Amazon DynamoDB table as a data source in Rockset. This includes:

• Creating an Amazon DynamoDB integration to securely connect tables in your AWS account with Rockset.
• Creating a collection which syncs your data from a Amazon DynamoDB table into Rockset in real-time.

For the following steps, you must have access to an AWS account and be able to manage AWS IAM policies and IAM users within it. If you do not have access, please invite your AWS administrator to Rockset.

#Create a DynamoDB Integration

The steps below show how to set up an Amazon DynamoDB integration using AWS Access Keys. An integration can provide access to one or more DynamoDB tables within your AWS account. You can use an integration to create collections that sync data from your DynamoDB tables.

#Step 1: Configure AWS IAM Policy

1. Navigate to the IAM Service in the AWS Management Console.
2. Set up a new policy by navigating to Policies and clicking Create policy. If you already have a policy set up for Rockset, you may update that existing policy. For more details, refer to AWS Documentation on IAM Policies.
3. Set up read-only access to your DynamoDB table. You can switch to the “JSON” tab and paste the policy shown below. You must replace <your-table> with the name of your DynamoDB table. If you already have a Rockset policy set up, you can add the body of the Statement attribute to it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:Scan",
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeTable",
        "dynamodb:UpdateTable"
      ],
      "Resource": [
        "arn:aws:dynamodb:*:*:table/<your-table>",
        "arn:aws:dynamodb:*:*:table/<your-table>/stream/*"
      ]
    }
  ]
}

4. Save the newly created or updated policy and give it a descriptive name. You will attach this policy to a user or role in the next step.

#Why these permissions?

• dynamodb:Scan ⁠— Required for initial table scans when reading data.
• dynamodb:DescribeStream — Required for metadata about DynamoDB streams.
• dynamodb:GetRecords — Required for retrieving records from DynamoDB streams.
• dynamodb:GetShardIterator — Required for retrieving records from DynamoDB streams.
• dynamodb:DescribeTable — Required for metadata about DynamoDB tables.
• dynamodb:UpdateTable — Optional. Used to enable streams on a DynamoDB table. DynamoDB streams are required for live sync to work correctly. You can omit this permission if you would like to enable streams on your tables manually as described . You have to specify the StreamViewType as NEW_AND_OLD_IMAGES while creating the stream.

#Advanced Permissions

You can set up permissions for multiple tables, or even all tables by modifying the Resource ARNs. The format of the ARN for DynamoDB is as follows: arn:aws:dynamodb:region:account-id:table/tablename.

You can substitute the following resources in the policy above to grant access to multiple tables as shown below:

• All tables in your account
  • arn:aws:dynamodb:*:*:table/*
  • arn:aws:dynamodb:*:*:table/*/stream/*
• All tables starting with “prod”
  • arn:aws:dynamodb:*:*:table/prod*
  • arn:aws:dynamodb:*:*:table/prod*/stream/*
• All tables in us-west-2
  • arn:aws:dynamodb:us-west-2:*:table/*
  • arn:aws:dynamodb:us-west-2:*:table/*/stream/*

Please note that you must also include the corresponding /stream/* permissions with the above for live sync to work correctly. For more details on how to specifiy a resource path, refer to AWS documentation on DynamoDB ARNs.

#Step 2: Configure AWS Access Key

In this step, you will grant Rockset permissions to access your AWS resource using AWS Access Keys.

#AWS Access Key

1. Navigate to the IAM service in the AWS Management Console.

2. Create a new user by navigating to Users and clicking Add User. If you have already created a user for Rockset in the past, you can attach the policy created in the previous section to that user.

3. Enter a name for the user and check the Programmatic access option. Click to continue.

4. Choose Attach existing policies directly then select the policy you created in Step 1. Click through the remaining steps to finish creating the user.

5. When the new user is successfully created you should see the Access key ID and Secret access key displayed on the screen.

If you are attaching the policy to an existing IAM user, you can navigate to “Security Credentials” under the IAM user and generate a new access key.

1. Record both these values in the console within a new DynamoDB integration.

#Create a Collection

Once you create a collection backed by Amazon DynamoDB, Rockset scans the DynamoDB tables to continuosly ingest and then subsequently uses the stream to update collections as new objects are added to the DynamoDB table. The sync latency is no more than a few seconds when the source is getting updated continuously and no more than 5 minutes when the source gets updated infrequently.

In the Rockset Console, you can create a collection from Workspace > Collections > Create Collection.

Using the CLI, you can run the following:

$ rock create collection my-first-dynamodb-collection \
    dynamodb://my-table \
    --integration=aws-rockset-readonly

Collection "my-first-dynamodb-collection" was created successfully.

Note that these operations can also be performed using any of the Rockset client libraries.