Collection Snapshot and Restore

🔐

Collection Snapshot and Restore is currently in Private Preview. Contact Rockset Support to enable this feature.

Snapshotting and restoring offers the opportunity to efficiently freeze a copy of a Collection that can be restored into a brand new collection later. This can be used for backing up data that might be modified by unexpected ingestion pipeline changes.

Creating Snapshots

Creating a snapshot of a collection makes a copy of the current collection (the contents of the snapshot may be up to 10 minutes behind the current contents). Each snapshot doesn’t make a full copy of the files but only tracks the changes from the previous snapshot. Therefore, you will only be billed for the incremental data between snapshots and not for the whole collection size every time a snapshot is created. Snapshots will expire at most 7 days after they are taken.

Example Create Request

# Example Request
curl --request POST \
    --url https://$ROCKSET_SERVER/v1/orgs/self/snapshots \
    -H 'Authorization: ApiKey $API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
    "collection_rrn": "rrn:col:usw2a1:a9999999-888c-77e6-55cb-f44444f3eb22",
    "description": "Collection backup snapshot",
    "expiration_time_millis": 1694472848129
  }'

Creating a snapshot will put the snapshot into a REQUESTED state. You must wait until snapshot state is changed to CREATED before it is eligible to be restored from. You can list all snapshots and their statuses by querying the snapshot endpoint.

Example List Request

# Example Request
curl --request GET \
    --url https://$ROCKSET_SERVER/v1/orgs/self/snapshots \
    -H 'Authorization: ApiKey $API_KEY'

Restoring Collections

Restoring a snapshot materializes all the data referenced by a snapshot into a brand new collection that has it’s own copy of all the data. Changes made to the to source collection will not effect the restored collection and vice versa. The newly created collection will also not be connected to any streaming ingest sources.

It is possible to connect a streaming source to the restored collection. But do note that when connecting to a streaming source all the documents from the source will be re-ingested in a streaming fashion. For instance, connecting to an S3 source will result in all the S3 objects in the path being re-ingested.

For collections with retention set, the retention settings cannot be modified when restoring. This is because when the collection is created, it is clustered by a time interval relative to the retention period. In order to change the retention period, the collection will need to be recreated and reindexed to ensure that the data is clustered on the new time interval. Restoring does not reindex. When a collection is restored, all the documents who’s _event_time is older than the retention period will be dropped.

As restoring a collection requires making a full copy of the data, this is not instantaneous but it is significantly faster than re-ingesting a collection you need to recover. Furthermore, to restore a collection larger than 10TB please contact Rockset Support first. When fully restored, this new collection will be queryable and modifiable like any other Rockset collection.

# Example Request
curl --request POST \
    --url https://$ROCKSET_SERVER/v1/orgs/self/ws/$WORKSPACE/collections \
    -H 'Authorization: ApiKey ' \
    -H 'Content-Type: application/json' \
    -d '{
    "name": "restored_collection",
    "sources": [
      {
        "snapshot": {
          "source_snapshot_rrn": "rrn:snap:usw2a1:22f3333a-f4ac-5555-6c77-8bbac99d0ad1"
        }
      }
    ]
  }'