This page describes how Rockset provides visibility into your account through the notion of events – which are stored as documents in a special collection called
_events. It also describes how you can create collections that ingest and store logs of queries that you run (currently in beta).
Audit events are captured within the
_events collection. Note, that the
_events collection has a retention of 30 days. For access to older event logs, please contact please contact [email protected].
_events collection is created when your organization is initialized and cannot be deleted. All events exist in the
_events collection as documents containing the following fields:
_id- a unique event identifer
_event_time- the time at which the event occurred, in milliseconds after January 1, 1970 (UTC).
label- one of the labels listed in the table below
kind- can be
USER. Note that events in the
QUERYcategory only apply to queries made from the Rockset Console, and does not include events from queries made via API calls.
type- can be
Events may also have additional fields depending on the specific event. All supported events are listed in the table below, as well as additional fields for each event label:
|API_KEY_CREATED||API_KEY||INFO||api_key_name , user_email , details|
|API_KEY_DELETED||API_KEY||INFO||api_key_name , user_email , details|
|API_KEY_ERROR||API_KEY||ERROR||api_key_name , user_email|
|COLLECTION_CREATED||COLLECTION||INFO||collections , user_email|
|COLLECTION_DROPPED||COLLECTION||INFO||collections , user_email|
|INGEST_WARNING||INGEST||WARNING||collections , details|
|INGEST_ERROR||INGEST||ERROR||collections , details|
|INGEST_INFO||INGEST||INFO||collections , details|
|INTEGRATION_CREATED||INTEGRATION||INFO||integrations , user_email|
|QUERY_COLLECTION_NOT_READY||QUERY||ERROR||user_email , details|
|QUERY_ERROR||QUERY||ERROR||user_email , details|
|QUERY_INVALID||QUERY||ERROR||user_email , details|
|QUERY_SUCCESS||QUERY||INFO||user_email , details|
|QUERY_UNIMPLEMENTED||QUERY||ERROR||user_email , details|
|COLLECTION_READY||COLLECTION||INFO||collections , details|
|COLLECTION_PAUSED||COLLECTION||INFO||collections , details|
Query logs are designed to (1) help identify slow and/or compute-intensive queries as candidates for optimization and (2) trouble-shoot high query latencies and/or high CPU utilization. Query Logs show up as a new collection in your account when enabled. In order to enable this feature, you will create a new collection to view query logs of all your existing collections. During the source selection screen of the Query Log collection, choose 'Query Logs' (if you do not see this source option available, please contact Rockset support to have this beta feature enabled for your org).
Note that creating a collection with a query logs source requires the CREATE_QUERY_LOGS_COLLECTION_GLOBAL privilege. Only admins have this privilege unless you use custom roles. During the collection creation process, you can configure the retention period for your logs just like you would with any other collection.
After you have created the collection, logs of your queries will be ingested. All queries will receive INFO level logs (unless a rate limit has been reached). INFO logs contain some basic information about the query. However, INFO logs cannot be used with the query profiler. This is where DEBUG logs come into play. DEBUG logs contain extra data that allows them to be used with the query profiler.
In order to log DEBUG information with your query, you must add the
debug_log_threshold_ms query hint to the end of your query text. You can also add the
debug_threshold_ms parameter to your query or query lambda execution requests. If both the hint and the API parameter are provided, Rockset will always use the debug threshold specified in the API parameter.
SELECT * FROM _events HINT(debug_log_threshold_ms=1000)
Note that because DEBUG logs contain large amounts of data, it is recommended that you use them sparingly. To prevent potential problems with Virtual Instances, limits are enforced on the rate that logs are recorded. INFO logs have a much higher rate limit than DEBUG logs.
The extra information that is logged with DEBUG level logs allow you to generate a query profile. If the query editor detects that you are attempting to query a collection with a query logs source, a column called 'Profiler' will be added to the query results table. Any documents that have a populated
stats field will have a link in this column. Clicking on this link will open the query profile in a new tab.
These profiler links can also be found in the documents preview of the collection details page.
Note: for this functionality to work correctly, both the
_id columns must be
included without any modifications within the results of your query. Renaming these columns using an ingest transformation or SQL alias will also cause issues for the profiler links.
The basic information included with both types of logs includes:
- Query text
- Query lambda path (if relevant)
- Query parameters
- Query end state (status)
- User email (user_id)
- User agent
- Query runtime (runtime_ms)
- Throttled time (queued_time_ms)
- Completion timestamp (completion_timestamp_ms)
- Virtual Instance ID
- Virtual Instance Size
- Result count
- Client timeout (client_timeout_ms)
- Query timeout (timeout_ms)
- Result count is only included for DEBUG-level logs
- Queries that error before the plan is created will not be logged
Updated 1 day ago