Rockset Extended JSON

Rockset supports an extended version of the JSON format which allows for Rockset specific types that are not supported by JSON. This format is used for both query parameters and ingestion.

JSON Format

Rockset supports many different data types but by default values will be parsed as standard JSON, so the value hello will be parsed as a string. A numeric type will be parsed as an integer if it falls within the range of the integer data type and does not contain a floating point or exponent. Otherwise, the value will be read as a float.

Extended Format

There are many types that Rockset supports that are not JSON native such as u256 along with Date and Time types, Geography types, etc. To support the interpretation of these types, we wrap them in a JSON object containing a special key __rockset_type which is the string value (not case sensitive) of the desired type that will be used to parse the value in the value key.

For example, the following JSON object will be interpreted as a value of 10 with type u256:

{
   "__rockset_type": "u256",
    "value": "10"
}

Like value there are a few special keys outside of __rockset_typebut only __rockset_type will be used to check for the presence of a special typed value. Otherwise these object fields will be read in as part of normal JSON objects.

Special Keys

__rockset_type

Required in order to specify a typed value. Supports all Rockset data types which means in some cases, like integer parsing, it can be used to override the inferred JSON type.

value

Required string that specifies the value which will be parsed as the requested __rockset_type.

format

This field is optional and specifies the format to be used when parsing a Date or Time type. If not specified the default ISO-8601 format for that type will be used. You can lean more about date formatting and Rockset here.

timezone

This field is optional and specifies the TZ identifier that will be used to parse a timestamp type. By default UTC will be used.

encoding

A required field for the bytes data type. Only base64 is currently supported.

Use Cases

Large int values

Rockset's native int type can support integers between -2^63 and 2^63-1, inclusive. However, JSON only supports integers up to2^53. To workaround this JSON limitation, use the following format to ingest larger values without losing precision.

{ "__rockset_type": "int",
  "value": "9223372036854775807"
}

Non-Finitefloat values

JSON does not support non-finite floating point values: NaN, Infinity (positive infinity), or -Infinity (negative infinity). To workaround this JSON limitation, use the following format to ingest these values:

{ "__rockset_type": "float",
  "value": "-Infinity"
}

Encoding bytes

Rockset supports arbitrary byte strings using the bytes type. However, JSON strings must be valid UTF-8. To work around this JSON limitation, encode the byte string in a UTF-8 string. base64 is the only supported encoding.

{ "__rockset_type": "bytes",
  "encoding": "base64",
  "value": "aGVsbG8K"
}

Ingesting datetime

ISO 8601 without timezone designator.

{ "__rockset_type": "datetime",
  "value": "YYYY-mm-ddTHH:MM:SS[.ssssss]"
}

Ingesting date

ISO 8601 without timezone designator.

{ "__rockset_type": "date",
  "value": "YYYY-mm-dd"
}

Ingesting time

ISO 8601 without timezone designator.

{ "__rockset_type": "time",
  "value": "HH:MM:SS[.ssssss]"
}

Ingesting timestamp

Either ISO 8601 with timezone designator, or (if numeric, or a string that can be parsed as numeric) number of microseconds since epoch.

{ "__rockset_type": "timestamp",
  "value": "YYYY-mm-ddTHH:MM:SS[.ssssss]Z"
}

or

{ "__rockset_type": "timestamp",
  "value": 1234
}

or

{ "__rockset_type": "timestamp",
  "value": "1234"
}

Ingesting geography

Geographies represent shapes on the Earth’s surface. Their JSON encoding is GeoJSON. Rockset supports points, linestrings, polygons, and multipolygons. Multipolygons are parsed as lists of polygons.

{
  "__rockset_type": "geography",
  "value": {
    "type": "POINT",
    "coordinates": [10, -12]
  }
}

{
  "__rockset_type": "geography",
  "value": {
    "type": "LINESTRING",
    "coordinates": [[10, -12], [9, -6]]
  }
}

{
  "__rockset_type": "geography",
  "value": {
    "type": "POLYGON",
    "coordinates": [[[0, 0], [1, 0], [1, 1], [0, 1]]]
  }
}

{
  "__rockset_type": "geography",
  "value": {
    "type": "POLYGON",
    "coordinates": [[[0, 0], [1, 0], [1, 1], [0, 1]],[[10, -12], [9, -6]]]
  }
}