Amazon MSK Privatelink

This page covers how to use MSK multi-VPC private connectivity to enable Rockset to ingest data from private MSK clusters without exposing them to the public internet.

🚧

For the following steps, you must have access to an AWS account and be able to manage AWS IAM policies and IAM users within it.

If you do not have access, please invite your AWS administrator to Rockset.

Requirements

• AWS MSK clusters configured with multi-VPC private connectivity and satisfying the requirements listed in Requirements and limitations for multi-VPC private connectivity.
• Rockset supports IAM authentication for MSK.
• Rockset does not support mTLS authentication.

Steps to Create MSK Integration

Step 1. Enabled multi-VPC private connectivity

Follow the Steps 1 and 2 from the guide on getting started using multi-VPC private connectivity.

Step i. Turn on the VPC connectivity:

This step takes around 30 minutes depending on the number of brokers in the cluster.

Step ii. Attach the cluster policy to allow Rockset to access the cluster and manage VPC connection:

📘

We can use the sample recommended policy with Rockset Account ID: 318212636800

Edit the cluster policy in the Security Settings.

Use the sample policy with the rockset account ID 318212636800

Step 2. Create a VPC endpoint using Rockset API

Create VPC connection using the Rockset API. Generate a new API Key and find the relevant API endpoint in the API Key View.

You can use the following command to create the endpoint:

curl 'https://api.usw2a1.rockset.com/v1/orgs/self/network/configurations' \
  -H 'Authorization: ApiKey <api_key>' \
  -H 'Content-Type: application/json' \
  --data '
{
 "name": "msk-apurwar-private-vpc",
 "msk_vpc_connection": {
    "target_cluster_arn": "<MSK_cluster_arn>"
  }
}
'

Check the Status using the following command until connection is in the READY state.

curl 'https://api.usw2a1.rockset.com/v1/orgs/self/network/configurations' \
  -H 'Authorization: ApiKey <api_key>' | jq
  
# a sample Ready Response
{
  "data": [
    {
      "rrn": "<rrn>",
      "name": "msk-apurwar-private-vpc",
      "description": "",
      "created_by": "[email protected]",
      "created_by_apikey_name": "apurwar_privatelink_test",
      "created_at": 1706300949581,
      "state": "READY",
      "details": null,
      "aws_private_link": null,
      "msk_vpc_connection": {
        "target_cluster_arn": "<MSK_cluster_arn>"
      }
    }
  ]
}

📘

Once the connection is ready, you can remove kafka:CreateVpcConnection action from the cluster policy.

Step 3. Create a new Rockset MSK integration here.

i. Create a new IAM role following the Step 2 to configure the AWS IAM policy to allow Kafka ingest.

ii. To find the private endpoints of the bootstrap servers, go to "View Client Information" and copy the "Private endpoint (multi-VPC) flags".

Use the multi-VPC bootstrap servers to create the MSK integration:

Step 4. Create Rockset collections using the integration created.

Create Collection following the MSK collection creation process.