DiskANN Vector Index in Azure Database for PostgreSQL

We’re excited to announce the preview of DiskANN, the leading vector indexing algorithm in Azure Database for PostgreSQL – Flexible Server! developer: Microsoft Research DiskANN, widely used by Microsoft across global services such as Bing and Microsoft 365, enables developers to build highly accurate, high-performing, and scalable Generative AI applications. HNSW and IVFFlat in pgVector Both in latency and accuracy. DiskANN also overcomes the long-standing limitations of pgVector in filtered vector search. Returns incorrect results.

The DiskANN algorithm we are releasing today is built on a new algorithm. High performance implementation Originally started by Microsoft Research (MSR) authors. Our team worked closely with MSR to fine-tune it specifically for PostgreSQL.

What is DiskANN?

DiskANN IsN Approximate nearest neighbor search calculation designed for Efficient vector search at any scale. it provides no way difficult to achieve combination of high recall, throughputand i Required by modern production grade application.

Figure 1: DiskANN-optimized layout on Azure PostgreSQL – Flexible Server

DiskANN implements the following powerful technologies:

• Optimized Storage – Algorithms can scale beyond RAM limits without slowing down searches.
• Vector Quantization – Keep quantized vectors in memory. Implementing DiskANN in PostgreSQL balances the interaction between quantized and unquantized vectors to provide both low latency and high accuracy.
• Iterative post-filtering – Improves the accuracy of filtered vector search results without sacrificing speed or precision.
• New Vamana graph structure robust to index updates – Vamana is more robust to changes than traditional graph indexes by maintaining accuracy despite large numbers of insertions, modifications, and deletions without the need to rebuild expensive indexes.
• Learn more about Microsoft’s DiskANN.

Filtered vector search problem

Vector similarity search involves creating a vector representation of data (such as an image or text), storing it in an index, and searching that index with a query vector to find relevant results. While executing vector search queries, users often apply filters to non-vectorized data to increase relevance. That is, filtering by date or price in a vector query. Despite this common use case, pgVector support for filtered vector searches is limited. Let’s say you want to use the following query to retrieve a list of the top 3 rentals you can stay in for under $30 on a specific date.

WITH all_filtered_listings AS
(
SELECT l.listing_id, l.name, c.price, c.date, l.embedding
FROM listings_search l
INNER JOIN calendar c ON l.listing_id = c.listing_id
WHERE c.date="2025-01-02" AND c.price < 30   
)
SELECT listing_id, name, price, date, summary
FROM all_filtered_listings
ORDER BY embedding <=> @query_vector
LIMIT 3;

When using HNSW by pgVectorVia filtered vector similarity search top 3 The most relevant rows from a collection of important data in the database are converted into a two-stage query plan by the query planner.

1) The first step is to place an order. Most Relevant Rows Dataset (40 rows by default). The number of similar rows to return is defined by: ef_search ​A parameter that controls the size of the dynamic list used during the search process.

2) apply post filter (Price less than $30, as of January 2nd)^nd2025) Remove rows that do not meet a predicate filter. then come back 3 lines Matches the filter (LIMIT 3 defined in the query).

However, the pgVector implementation does not guarantee that: 3 lines After filtering it is returned to the user. Filters with low selectivity may not find matches in the post-filter step. The 40 most relevant rows. When this happens, pgVector returns no results at all, even if there are rows in the table that satisfy the condition.

Figure 2: Using HNSW. Imagine a scenario where a user searches a database of real estate listings to find listings, say with a specific query, and only wants listings under $30 on a specific date.

increase Number of rows involved Returning HNSW can alleviate this problem, but in pgVector’s HNSW maximum value The number of relevant rows that can be returned is 1000. Therefore, if your query selectivity requires retrieving more than 1000 results returned by the index, there is no way to achieve the expected results using HNSW. This situation occurs frequently while searching large datasets and results in lower recall when using HNSW in filtered queries.

Introduction to high-precision filtered vector search with iterative post-filtering

With DiskANN in Postgres, we introduced iterative post-filtering that returns highly relevant results at exceptional speed, especially in filtered search queries. DiskANN improves the accuracy of search results through iterative post-filtering by repeatedly fetching the next closest set of items without sacrificing speed or precision until the desired number of results is met. This innovative approach is useful for filtered search queries where maintaining high relevance and low latency is important.

When using DiskANNVia filtered vector similarity search top 3 The most relevant rows from a collection of important data are converted into a three-stage query plan by the query planner.

1) The first step is to place an order. Most Relevant Rows From the data set. The number of similar rows to return is defined by the l_value_is parameter of DiskANN.

2) apply post filter (Price less than $30, as of January 2nd)^nd2025) Remove rows that do not meet a predicate filter.

3) Continue looping through all available rows in similarity order. 3 lines A matching filter was found (LIMIT 3 defined in the query).

Figure 3: Using DiskANN. Imagine a scenario where a user searches a database of real estate listings to find listings similar to a specific query and only wants listings priced under $30 on a specific date.

Scalability for the future

DiskANN not only improves recall, but leverages SSDs to provide unparalleled scalability and efficiency using less memory, making it easy to store and retrieve large amounts of vector data. DiskANN leverages quantization to store compressed vectors in a graph in memory and then finds the entire vector on the SSD for a final comparison that preserves precision. These advancements ensure that the system remains responsive and cost-effective even with extensive data sets.

Figure 4: How storage is optimized in DiskANN

product quantification apply (To be released)

DiskANN use product Quantization (PQ) dramatically R memory footprinttea of vector. Unlike other quantization techniques, that P.Q. calculation ~can do compress vector more effectively, citybigIt’s getting a lot better performance. DiskANN using PQ Reduce the need to access slow storagelike usI’ll use less calculate When comparing compression vector. additionally, Product Quantification (PQ) maintain high recall rate Rank the top candidates using full-precision vectors loaded from storage.

Figure 5: Vector quantization in DiskANN

Strong against insertion, deletion and modification

DiskANN graph indexes do not degrade over time due to large numbers of inserts, updates, or deletes. This is different from typical vector databases on the market today, which are built using HNSW and other less robust methods that require computationally expensive full index rebuilds to maintain accuracy. DiskANN’s robust design aims to minimize maintenance and ensure consistent accuracy, making it ideal for production environments where compromise in search precision is not acceptable.

Figure 6: Stability of DiskANN compared to HNSW

Embrace the future of AI with Azure Database for PostgreSQL

Integrating cutting-edge vector search capabilities into Azure Database for PostgreSQL is a game-changer for AI applications. Beyond standard vector search, DiskANN vector indexing options power Azure Database for PostgreSQL to deliver unmatched precision and efficiency. DiskANN’s advanced features allow Azure Database for PostgreSQL to:

• Reduce operational complexity and costs: Through DiskANN’s efficient algorithms and well-designed implementation of index access methods, you can benefit from cost savings by efficiently utilizing Postgres storage.
• Unlock scalability: Seamlessly scale your AI applications with the confidence that Azure Database for PostgreSQL leverages SSDs and uses less memory for unmatched scalability.

Using DiskANN with Azure Database for PostgreSQL

Using DiskANN with Azure Database for PostgreSQL is easy.

1. Enables the pgVector and diskann extensions. Whitelist the pgVector and diskann extensions within your server configuration.

Figure 8: DiskANN enabled in Azure Database for PostgreSQL

2. Creating vector columns: Define a table to store vector data, including a column of type vector for vector embeddings.
3. Index vector columns. To optimize search performance, create indexes on vector columns. The diskann PostgreSQL extension is compatible with pgVector and uses the same types, distance functions, and syntax styles.

CREATE EXTENSION IF NOT EXISTS pg_diskann CASCADE;
CREATE INDEX my_table_embedding_diskann_idx ON my_table USING diskann (embedding vector_cosine_ops);

4. Perform a vector search: Use a SQL query to retrieve similar vectors based on different distance metrics (cosine similarity in the example below).

SELECT * FROM my_vectors ORDER BY vector_column <=> @query_vector LIMIT 10;

Are you ready to take the plunge?

use Preview DiskANN today Explore the future of AI-driven applications with the power of Azure Database for PostgreSQL!

1. Sign up for DiskANN early preview
2. Once approved. Run the end-to-end sample to test DiskANN and HNSW on a sample database. here.

Learn more

The integration of DiskANN with Azure Database for PostgreSQL opens new horizons for building powerful, scalable, and efficient AI-based applications. Advanced vector search capabilities can help you improve the performance of your AI applications and deliver more accurate results faster than ever before.

Special thanks to the team behind it. DiskANN open source Project.