- Loading Your Data
- Collections
- Special Fields
Special Fields
This page describes system-generated fields with unique roles and behaviors in Rockset documents.
#Overview
Special fields are automatically generated by Rockset during ingest and are added as new fields to each document. They can be queried like any other field, and you may also apply transformations to them using field mappings.
For instance, you might execute the following query on any collection to view its special fields:
SELECT
_id,
_meta,
_event_time
FROM
mycollection
LIMIT
5;
+---------------------------------------------------------------------------------+----------------+----------------------------------------+
| _meta | _event_time | _id |
|---------------------------------------------------------------------------------+----------------+----------------------------------------|
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '47304-1709397620935'}} | 1535068823524 | '5d99f42c-44ec-2024-a032-ac14d1cbf44d' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '46242-1709397472272'}} | 1535068823426 | 'e5d4c136-8d56-ac39-38f5-2f795ae38007' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '46242-1503238554088'}} | 1535068823326 | '71df712e-04bc-7c9b-0114-97b19741b215' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '47302-1151051235692'}} | 1535068822852 | '517412d2-1a23-1b49-fc3b-441c630ed4ff' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '45189-1709397393868'}} | 1535068822527 | 'd16bf924-f758-21db-c832-4a25323b2938' |
+---------------------------------------------------------------------------------+----------------+----------------------------------------+
#The _id
Field
Every document in a Rockset collection is uniquely identified by its _id
field.
- If the source of a document does not already have an
_id
field, Rockset populates it with an automatically generated uuid. - If the source of a document has
_id
specified, or a field mapping outputs an_id
field, its value is preserved. A newly ingested document will overwrite any existing document with the same_id
value.
#The _meta
Field
Metadata regarding each document is stored in a _meta
field of object type.
If the source of a document specifies a _meta
field, Rockset will ignore the field. Currently,
_meta
holds information about the source from which the document was inserted into the collection
(e.g. the bucket name and path in case of S3). If Rockset is unable to parse the source of a
document, it will create a document without any of the source's fields and will have _meta
with a
nested field named bad
.
#The _event_time
Field
Rockset associates a timestamp with each document in a field named _event_time
, recorded as
microseconds since the Unix epoch. Queries on this field are significantly faster than similar
queries on regularly-indexed fields.
A user can select a field of the source of a document to be copied into the _event_time
field at
collection creation (refer to the event data guide for more information). This
is done by specifying:
- the name of the selected field, which should be of type
int
ortimestamp
- the format with which to interpret the field, can be one of:
milliseconds_since_epoch
seconds_since_epoch
- default time zone, specified as one of the IANA time zone values
If an _event_time
configuration is not specified, Rockset will first check for a pre-existing
_event_time
field (must be type int
or timestamp
) and, if nonexistent, populates it with the
time of document insertion. This field is immutable after document creation.