This page covers how to use an Amazon Kinesis stream as a data source in Rockset. This includes:
For the following steps, you must have access to an AWS account and be able to manage AWS IAM policies and IAM users within it. If you do not have access, please invite your AWS administrator to Rockset.
The steps below show how to set up an Amazon Kinesis integration using AWS Cross-Account IAM Roles and AWS Access Keys. An integration can provide access to one or more Kinesis streams within your AWS account. You can use an integration to create collections that sync data from your Kinesis streams.
<your-stream>
with the name of your Kinesis stream. If you already have a Rockset policy set up, you can add the body of the Statement
attribute to it.{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:ListShards",
"kinesis:DescribeStream",
"kinesis:GetRecords",
"kinesis:GetShardIterator"
],
"Resource": [
"arn:aws:kinesis:*:*:stream/<your-stream>"
]
}
]
}
kinesis:ListShards
— Required for listing and getting metadata on each shard.kinesis:DescribeStream
— Required for fetching metadata about the Kinesis stream.kinesis:GetRecords
— Required for reading records from a shard.kinesis:GetShardIterator
— Required for iterating over records in a shard.You can set up permissions for multiple streams by modifying the Resource
ARNs. The format of the ARN for Kinesis is as follows:
arn:aws:kinesis:region:account-id:stream/stream-name
.
You can substitute the following resources in the policy above to grant access to multiple streams as shown below:
arn:aws:kinesis:us-west-2:*:stream/*
arn:aws:kinesis:*:*:stream/sales*
arn:aws:kinesis:*:*:stream/*
For more details on how to specifiy a resource path, refer to AWS documentation on Kinesis ARNs.
There are two mechanisms by which you can grant Rockset permissions to access your AWS resource. Although Access Keys are supported, Cross-Account roles are strongly recommended as they are more secure and easier to manage.
The most secure way to grant Rockset access to your AWS account involves giving Rockset’s account cross-account access to your AWS account. To do so, you’ll need to create an IAM Role that assumes your newly created policy on Rockset’s behalf.
You’ll need information from the Rockset Console to create and save this integration.
Navigate to the IAM service in the AWS Management Console.
Setup a new role by navigating to Roles and clicking Create role. Note: if you already have a role for Rockset set up, you may re-use it and either add or update the above policy directly.
Navigate to the IAM service in the AWS Management Console.
Create a new user by navigating to Users and clicking Add User.
Enter a name for the user and check the Programmatic access option. Click to continue.
Choose Attach existing policies directly then select the policy you created in Step 1. Click through the remaining steps to finish creating the user.
When the new user is successfully created you should see the Access key ID and Secret access key displayed on the screen.
Once you create a collection backed by an Amazon Kinesis stream, Rockset continuously ingests objects from your stream and updates your collection automatically as new objects are added.
In the Rockset Console, you can create a collection from Workspace > Collections > Create Collection.
Using the CLI, you can run the following:
$ rock create collection my-first-kinesis-collection \
kinesis://web-click-streams \
--integration=aws-rockset-readonly
Collection "my-first-kinesis-collection" was created successfully.
Note that these operations can also be performed using any of the Rockset client libraries.