Resource Allocations

The following settings are limits placed on your Rockset organization which can change based on your Virtual Instance category.

The Dedicated category includes organizations using any Dedicated Virtual Instance, while the Free, Nano, and Micro categories are Shared Virtual Instances. To change your Virtual Instance type, navigate to the Virtual Instances tab of the Rockset Console.

*Refer to Storage Size Limit section below.

Storage Size Limit

This refers to the total amount of data that you can store in Rockset.

Rockset accounts on Dedicated Virtual Instances do not have any data size limits; however, as a convenience to you, an artificial limit of 10 TiB is placed on your account so that any runaway process on your account cannot inundate your account with data that you were not intending to store in Rockset. If you would like to raise this limit for your account, please contact Rockset Customer Support.

🚧

Trial Ingest Limit

Trial organizations have a default ingest limit of 100 GiB for each collection. If you would like to raise this limit for your account, add a payment method or contact Rockset Customer Support.

Collection and Shard Limits

Collections are split up into smaller pieces called shards. A shard is the unit of parallelism for a collection. Your organization’s shard limit is determined by the size of the Ingest Virtual Instance, however this can be bottlenecked if you have a secondary VI configured as the Default Query VI of a smaller size. See the Calculating Shard and Collection Limits section below for more information on this.

The number of collections your organization can have is dependent on your shard limit and the shard count for each collection. The below limits assume all collection have 16 shards.

Shared VIs:

Dedicated VIs:

Note that General Purpose VI shard limits are half of their Memory-Optimized VI counterpart, as MO VIs offer double the memory of GP.

Calculating Shard and Collection Limits

The above collection limits assume that each collection has 16 shards, which is the current default shard count. The default was recently changed from 64 shards to 16 shards as a part of performance improvement initiatives. If you have collections with varying numbers of shards, the collection limit will vary. You can calculate the remaining number of collections you can create for your organization (assuming new collections will have the default 16 shards) using the following equation: VI shard limit - sum(all current collection shard counts) / 16.

For multi-VI setups, it is important to note that your organization’s shard limit is based on the size of your Ingest VI, but the number of collections you can create may be bottlenecked if you have a Default Query VI of a smaller size. All collections are automatically mounted on the Default Query VI upon collection creation. If you have reached the shard limit on your Default Query VI, no new collections can be created as they will not be able to be mounted to the Default Query VI, even if you have not reached the shard limit for your Ingest VI. If you would like to create more collections, you can unmount existing collections from the Default Query VI or size up your Default Query VI.

Determining Shard Count

A higher shard count is more effective with larger collections as it provides more parallelism. With smaller collections however, a higher shard count can perform worse than a lower shard count due to communication overhead. Lowering your shard count for smaller collections will help you save on memory and allow you to create more collections. If your collection size is ≤ 5TB and currently has 64 shards, we recommend recreating your collection so that it will have 16 shards.

See the Shard Count documentation for more information on how shard counts are determined.