In the Collections tab of the console, select “Create Collection” and follow the prompts to create a collection from a File Upload, one of our fully-managed integrations, or an empty collection that you can write data to using the WriteAPI. In the “Transform Data” step, you will be prompted for an Ingest Transformation where you will need to add the VECTOR_ENFORCE function for your embedding field (refer to the example above). Continue following the rest of the Create Collection flow prompts to finish creating the collection to store your embeddings.

Use the Create Collection API endpoint, to create a collection in Rockset. The field_mapping_query is the Ingest Transformation query applied to the ingesting data. In this query, be sure to add the VECTOR_ENFORCE function for your embedding field.

"field_mapping_query": {
    "sql": "SELECT title, author, VECTOR_ENFORCE(book_embedding, 1536, 'float') as book_embedding FROM _input"
 }

In the Query Editor tab of the Console, write a query similar to the above. To create a :target_embedding parameter, click the + button following “Parameters” and populate the fields as shown in the screenshot below. Hint “Enter” on your keyboard or click away from the “Add Parameter” modal to save the parameter. Click “Run” to execute your KNN search query.

Use the Execute SQL Query API endpoint, to execute a query similar to the above. In the sql object, query will be a string containing the SQL query and parameters is an array of parameter objects. You will need to pass one parameter for the target_embedding with the following format:

"parameters": [
      {
        "name": "target_embedding",
        "type": "array",
        "value": "<YOUR_SEARCH_EMBEDDING>"
      }
    ]

In the Query Editor tab of the Rockset Console, write a similar DDL Command to the above. Click “Run” to execute the query.

In the Collections tab of the Console, select the collection you would like to create a similarity index for. Under the “Indexes” heading, click “Create a Similarity Index”. Follow the prompts to create your desired index.

If using the example above, you would input the following parameters:

• Name: book_catalogue_embeddings_ann_index

• Embedding Field: book_embedding

• Dimensions: 1536

• Parameters: Standard, IVF, 256 centroids

• Distance Function: INNER_PRODUCT

Use the Execute SQL Query API endpoint, to execute a query similar to the above DDL command. In the sql object, query will be a string containing the DDL command.

In the Query Editor tab of the Rockset Console, write a similar SQL query to the above. Click “Run” to execute the query.

Use the Execute SQL Query API endpoint, to execute a query similar to the above. In the sql object, query will be a string containing the SQL query.

In the Query Editor tab of the Rockset Console, write a similar DDL Command to the above. Click “Run” to execute the query.

In the Collections tab of the Console, select the collection you created the similarity index for. Under the “Indexes” heading, hover over the row for the similarity index you would like to delete. A red trash icon will appear on the far right. Click this icon and complete the prompt to delete the index.

Use the Execute SQL Query API endpoint, to execute a query similar to the above DDL command. In the sql object, query will be a string containing the DDL command.

In the Query Editor tab of the Console, write a query similar to the above. To create a :target_embedding parameter, click the + button following “Parameter” and populate the fields as shown in the screenshot. Click “Run” to execute your KNN search query.

Use the Execute SQL Query API endpoint, to execute a query similar to the above. In the sql object, query will be a string containing the SQL query and parameters is an array of parameter objects. You will need to pass one parameter for the target_embedding with the following format:

"parameters": [
      {
        "name": "target_embedding",
        "type": "array",
        "value": "<YOUR_SEARCH_EMBEDDING>"
      }
    ]

Embeddings are just arrays of numbers that provide a compact, meaningful representation of large, complex, and unstructured pieces of data. These embeddings are typically generated using ML models which have the ability to ingest complex unstructured data and map the data to a compressed vector form representation of fixed length.

Representing complex data as compact arrays has many advantages including making storage and processing incredibly efficient. Another byproduct of working with embedding representations is that they are easily comparable with each other.

Vector Search has become increasing popular due to the accessibility and advancements in large language models. These language models include: GPT models from OpenAI, BERT by Google, LaMDA by Google, PaLM by Google, LLaMA by Meta AI. The embeddings generated by these models are high-dimensional and can be stored and indexed in Rockset for efficient vector search.

Vector Search refers to the practice of performing a similarity search over a set of vectors or "embeddings". (Learn more: What are embeddings?)
You can think of vectors as points in an N-Dimensional "latent space" and as such you can calculate the distance/similarity between two vectors using standard techniques like finding the Euclidean distance. (Learn more: How do you calculate distance/similarity in vector space?)

These distance functions calculate the semantic similarity of the data that was used to create the vectors and we call searching for vectors that are close to a specific vector "similarity search", otherwise known as vector search.

How close two vectors are in vector space correlates with how closely their semantic meanings are. There are three proximity functions that are commonly used.

Euclidean Distance:

• Geometrically, it measures the “straight-line” distance between two points.
• Algebraically, it’s the square root of the sum of the squared differences between the vectors.

Cosine Similarity:

• Geometrically, it quantifies the similarity of the directions, regardless of the magnitudes.
• Algebraically, it measures the cosine of the angle between two vectors.

Dot Product:

• Geometrically, it measures how closely two vectors align, in terms of the directions they point.
• Algebraically, it is the product of the two vectors' Euclidean magnitudes and the cosine of the angle between them.

K-Nearest Neighbors (k-NN) is a simple, yet effective, machine learning algorithm used for classification and regression tasks. It operates on the principle that similar things exist in close proximity. k-NN search is a linear search that involves computing the distance/similarity between a query vector and all other vectors and selecting the k nearest neighbors. The algorithm is non-parametric, meaning it does not make any underlying assumptions about the distribution of data, making it versatile for various kinds of data, but it can become computationally intensive as dataset size grows, owing to its need to calculate distances between data points.

(Learn more: How do you calculate distance/similarity in vector space?)

When the dataset is small, it is easy to scan the whole dataset to perform a KNN search, and many times all of the vectors that need to be looked at can fit in memory allowing for fast scans and comparisons. However, as the dataset increases, so does the latency.

One method to speed up vector search queries is to add selective predicates to our query (in the WHERE clause). This metadata filtering can significantly reduce the size of the dataset needed to scan over.

Another method to speed up vector search queries is to implement an approximate nearest neighbors search (ANN). In some cases it may be inescapable that we want to query over a billion or more vectors, and the cost of looking up and comparing every single one of our stored vectors is too high. At a certain point an exact KNN search becomes too expensive to scale. Thankfully in most cases exact ordering is not necessary and we really only need a best effort, or Approximate Nearest Neighbor (ANN) search.

(Learn more: What is Approximate Nearest Neighbor (ANN) search?)

Approximate Nearest Neighbors (ANN) search is a computational technique used to efficiently find the approximate nearest neighbors of a point in a high-dimensional space. Unlike exact searches that meticulously compute the closest points with precise accuracy, ANN search aims for high speed and reduced computational cost by allowing for a small margin of error in the results. It employs various algorithms and data structures, such as locality-sensitive hashing, trees, or graphs, to quickly approximate the nearest neighbors without exhaustively comparing every point in the dataset. This trade-off between accuracy and efficiency makes ANN search particularly valuable in large-scale and real-time applications, such as recommendation systems, image and video retrieval, and machine learning tasks where slight inaccuracies are acceptable in exchange for significant gains in performance.

Rockset's index filtering utilizes FAISS Inverted File Indexes for its ANN architecture.

Rockset will train a set of centroids on the collection's vector data with the number of these centroids being N specified in the term IVF{N} used in the factory string. Each centroid acts as a representative for a cluster of similar vectors and their associated documents. Together the centroids form an index on the vector data that can be used at query time. For each centroid Rockset will have a posting list that can be used for iteration.

At query time FAISS will gather the set of centroids (clusters/posting lists) in the index that have the smallest coarse distance from the query vector. The number of centroids that are retrieved and searched are based on the nprobe parameter whose default is set at index creation but can also be specified for each individual query. From here Rockset must simply iterate the posting lists and return the closest vectors.

When searching nprobe posting lists, the results returned may be less than the requested limit due to a selective predicate. In this case increasing the centroids to probe may increase the number of returned results.

You must specify a centroid value that gives you a sufficient number of samples in each cluster based on your total number of vectors. We recommend a value around 16x the square root of (total number of vectors/total shard count of the collection), as long as this value is no more than (total number of vectors/total shard count of the collection). If it is, use a smaller factor. This is because we partition documents for the clustering of the collection based on the number of shards, which are the unit of parallelism for a collection. For more information on shards, view our Shard Count documentation. The default shard count for collection is 16, but be sure to check your specific collection shard count.

For example, if you have 16,000 vectors in a collection with 16 shards, you may use a centroid value of 505 and specify IVF505 since 16x(square root(16000/16)) = ~505.

If you are receiving an 'Insufficient samples' error, this is an indicator that you need to adjust your centroid value.

Rockset vector operations support float and int type vectors.

Vectors are syntactically identical to arrays and there is no limit on array size, so there is no limit on vector size.

There is no general limit on the number of vectors that can be stored in Rockset. Rockset's disaggregated Storage Architecture allows your storage tier to scale independently of your compute needs.

Yes, you manipulate vectors in all of the same ways you would manipulate arrays. Rockset sits on top of an LSM tree based RocksdDB storage engine, which means random mutations are fast. Updates, inserts, and deletes are immediately visible in any ANN index associated with the vector.

Rockset is already built for low-latency complex analytics on real-time data which perfectly complements vector search use cases.