This section highlights the best practices for ingest performance. Ingest performance refers to throughput and latency of data being ingested into Rockset. Throughput measures the rate at which data is processed, impacting the database's ability to efficiently support high-velocity data streams. Data latency, on the other hand, refers to the amount of time it takes to ingest and index the data and make it available for querying, affecting the ability of a database to provide up-to-date results.
### Virtual Instance Sizing
Rockset offers different Virtual Instance sizes. It is important to size the instance correctly to handle the ingest workload for your production applications. Refer [here](🔗) for the peak streaming ingest limits for each Virtual Instance offering.
If the Virtual Instance is under sized for your workload, the streaming ingest latency will increase for managed sources. For Write API, this means that the client will receive "Too Many Requests (429)" error.
### Data schema and configurations
Source and collection configurations such as average document size, deeply nested objects and arrays, ingest transformations, and clustering may lead to different observed throughputs.
If your data contains deeply nested objects and arrays which are not needed for querying, you can configure to drop them using Ingest Transformations or use \_stored to exclude such fields from inverted and range indices.
### Virtual Instance Compute Contention
Virtual Instances's resources are shared between ingest and queries. Multiple Virtual Instances allow you to isolate and independently scale compute for ingest and queries. To eliminate the problem of compute contention, we recommend using one Virtual Instance for handling all the streaming ingest workload exclusively (with no query requests routed to that Virtual Instance, sometimes referred to as an “Ingest VI”), and additional Virtual Instances for each query use case.
### Source Configurations
Each managed source has different characteristics which can affect ingest performance. For best practices on configuring your source, refer to the respective data sources page.
### Dedicated Streaming Ingest
Dedicated Streaming Ingest is currently in Beta.
By enabling Dedicated Streaming Ingest, infrastructure and settings are configured to deliver the best performance for your latency sensitive ingest operations. When this feature is enabled,
Dedicated resources are provisioned for tailer operations directed onto your default VI.
Batch sizes and buffer flush times are configured for low-latency.
#### Why Dedicated Streaming Ingest?
Dedicated Streaming Ingest is designed to support ingest latency sensitive use cases. Dedicating an isolated set of compute resources for your tailer activities creates an environment conducive to a low and stable ingest latency. The various batch and flush settings are tuned to achieve a consistently lower median latency.
Non-dedicated streaming ingest has been designed to provide the lowest average latencies at an optimal price point. It is subject to tailer autoscaling activities which may result in observable increases in latencies exceeding the 90th percentile. When Dedicated Streaming ingest is enabled, a fixed number of resources are provisioned proportional to your VI size which are not subject to auto-scaling activities.
If your use case requires consistently lower ingest latencies, this feature is recommended.
#### Toggling Dedicated Streaming Ingest
##### REST API
On your default VI, you can enable dedicated streaming ingest via the [Update Virtual Instance](🔗) endpoint. Sending a request with the following payload will enable dedicated streaming ingest.
After sending the request to enable dedicated streaming ingest, there will be a brief scaling period which occurs while your resources are provisioned. Under typical operating conditions, the scaling duration should be a few minutes. During this time, VI resizing will be blocked. To know when Dedicated Streaming ingest is enabled and resources are provisioned, query the VI using the [Retrieve Virtual Instance](🔗) endpoint. The `dedicated_streaming_ingest_active` field is true when your dedicated streaming resources are provisioned and ready for processing ingest activity.
##### Console
From the virtual instance panel within the console, dedicated streaming ingest can be toggled.
There is currently a default limit of 10 toggle events per day.