Vald Vector Database

What Vald can do?

Core Functionality

• Finds nearest neighbors: It excels at locating the closest matches (nearest neighbors) for a given vector within a massive dataset of vectors.
• Handles high-dimensional data: It's specifically designed to work with high-dimensional data, where traditional search methods become inefficient.
• Performs approximate nearest neighbor (ANN) search: It employs the NGT algorithm, a highly efficient ANN algorithm, to achieve fast search results without sacrificing accuracy.

Key Applications

• Recommendation systems: Recommend similar products, movies, music, or articles based on user preferences.
• Image and video retrieval: Quickly find similar images or videos in large collections.
• Fraud detection: Identify suspicious transactions or activities based on their similarity to known fraudulent patterns.
• Natural language processing (NLP): Facilitate tasks like machine translation, sentence embedding, and text similarity.
• Anomaly detection: Spot unusual data points or outliers that deviate from expected patterns.
• Content-based search: Search for text, images, or videos based on their similarity to a query, rather than just keywords.
• Personalization: Tailor content and recommendations to individual users based on their past interactions and preferences.

Vald's capabilities make it a valuable tool for various industries and domains, including

• E-commerce
• Media and entertainment
• Finance
• Healthcare
• Manufacturing
• Security
• Research
• And any field that deals with high-dimensional data and requires efficient similarity search.

Why Vald?

1. Scalability

• Handles billions of vectors: Designed to manage massive datasets with billions of vectors, making it suitable for large-scale applications.
• Horizontally scalable: Easily expands its capacity and performance by adding more nodes to Kubernetes cluster, ensuring it can grow alongside your data needs.

2. Performance:

• Fast approximate nearest neighbor (ANN) search: Employs the highly efficient NGT algorithm for lightning-fast search results, even with high-dimensional data.
• Low latency: Optimized for quick response times, crucial for real-time applications and interactive user experiences.

3. Ease of Use:

• Relatively beginner-friendly: Simpler to set up and use compared to some other vector databases, making it accessible to a wider range of users.
• Flexible deployment: Can be deployed on various cloud platforms or on-premises infrastructure.

4. Customization:

• Configurable for specific needs: Offers various configuration options to tailor its behavior to match the unique requirements of different use cases.
• Supports multiple data types: Can handle various data types, including images, audio, video, text, and binary data, as long as they can be represented as vectors.

5. Cloud-Native Architecture:

• Designed for Kubernetes: Seamless integration with modern cloud infrastructure, allowing for efficient containerization and orchestration.
• Benefits from cloud-native features: Takes advantage of cloud-native capabilities like automatic scaling, load balancing, and high availability.

6. Automatic Indexing and Backup:

• Simplifies vector indexing: Automatically creates and manages vector indexes, reducing manual effort and potential errors.
• Protects data: Provides backup and disaster recovery features to ensure data resilience and availability.

7. Open-Source:

• Free and open-source: Available for anyone to use and contribute to, fostering a community of users and developers.
• Enables customization and innovation: Allows users to modify, extend, and adapt Vald to their specific needs.

8. Active Development:

• Regularly maintained and updated: Benefits from continuous development and support, ensuring new features, bug fixes, and performance improvements.
• Stays up-to-date with industry trends: Keeps pace with advancements in vector search and related technologies.

How does Vald work?

Architecture

• Microservices-based: Built upon a distributed architecture consisting of independent microservices that communicate with each other. This improves scalability and fault tolerance.
• Apache Cassandra & Google NGT: Leverages a powerful combination of Apache Cassandra for efficient data storage and retrieval, and Google's NGT for fast approximate nearest neighbor (ANN) search.

Data Indexing

• Vector representations: Data points are converted into high-dimensional vectors (numerical representations) for efficient similarity calculations.
• HNSW graph construction: Vald creates a special type of graph called Hierarchical Navigable Small World (HNSW) based on the data vectors. This HNSW graph guides the NGT algorithm to quickly find similar vectors.
• Multi-stage search: The search process employs a two-stage approach. First, the HNSW graph narrows down the search space to a smaller set of potentially similar vectors. Then, the NGT algorithm performs a more refined search within that set to identify the closest matches.

Scalability

• Horizontally scalable: Vald can be easily scaled out by adding more nodes to its Kubernetes cluster. Each node stores and processes a portion of the data and indexes, enabling it to handle larger datasets and higher workloads.
• Load balancing: Workload is distributed evenly across the nodes in the cluster, ensuring efficient utilization of resources and optimal performance.

Additional Features

• Automatic indexing and backup: Vald automatically updates indexes as data changes and provides built-in backup functionalities for reliable data protection.
• Cloud-native integration: Designed for deployment on Kubernetes, making it compatible with modern cloud infrastructure and leveraging its benefits.
• Open-source: Freely available to use and contribute to, fostering a collaborative community and ongoing development.

To summarize, Vald's combination of efficient data storage, intelligent search algorithms, and distributed architecture creates a powerful and scalable platform for finding similar vectors within massive datasets. This makes it a valuable tool for various applications requiring fast and accurate similarity search, especially in domains like recommendation systems, image retrieval, and fraud detection.

Basic architecture

1. Data Layer

• Apache Cassandra: Represented as a cluster of interconnected nodes, handling distributed storage and retrieval of vector data and metadata.
• Vector Indexing: Inside Cassandra, vector data is organized into an efficient HNSW graph, shown as a network of connected points, enabling fast similarity search.

2. Processing Layer

• Microservices: Illustrated as smaller modules within the cluster, each responsible for specific tasks like indexing, search execution (powered by NGT), and data management.
• Load Balancing: Arrows indicate workload distribution across microservices and nodes, ensuring smooth operation and scalability.

3. Interaction Layer

• Client Libraries: Users interact with Vald through these libraries (represented as interfaces), provided for various programming languages like Python and Java.
• APIs: Exposed endpoints (shown as connectors) allow programmatic access to Vald's functionality for integration with other applications.
• Web Interface (Optional): A web interface (if used) is depicted as a separate module for monitoring, managing Vald, and visualizing search results and data insights.

Additional Elements

• Data Flow: Arrows show the overall flow of data, from ingesting new vectors through indexing, search processing, and returning results to the user.
• Communication: Microservices communicate with each other and with Cassandra through messaging queues for efficient processing and coordination.

Use cases

This document will introduce you to the example of what Vald can do. Vald vector database is a highly scalable distributed fast approximate nearest neighbor dense vector search engine, which uses NGT as the core engine of Vald, and Vald manages to integrate with Kubernetes.

You cannot generally search your unstructured data using the inverted index, like images and videos. Applying a model like BERT or VGG can convert your unstructured data into vectors. After converting them into vectors, you can insert them into the Vald cluster and process them in the Vald cluster.

Here are some general use cases of Vald or vector search engines.

Image and video processing

You can use Vald as the image/video processing engine to search the similar images/videos or analyze the image/video for your use case.

Vald is capable of processing a huge number of images at the same time, so its case fits with your use case.

Here are some examples of what you can do with images and videos using Vald.

• Search by image
• Face recognition
• Product recommendation based on images
• Image/Video analysis
• Image/Video deduplication

Audio processing

Audio processing is important for personal assistant implementation.

Vald can act as a brain of the personal assistant function, conversation interpreter, and natural language generation.

Here are some examples of what you can process using Vald.

• Personal assistant
• Speech recognition
• Natural language understanding and generation

Text processing

Using a text vectorizing model like BERT, you can process your text data in Vald.

Here are some examples of the use case of text processing using Vald.

• Search by text
• Product recommendation based on text
• Grammar checker
• Real-time translator

Data analysis

Vald can process the vector data, you can analyze every data you can vectorize.

Here are some examples of the use case of data analysis.

• AI malware detection
• To detect the malware using Vald, you need to vectorize the binary file and insert it into Vald first. You can analyze your binary by searching for a similar binary in Vald. If your binary closely resembles the malware binary, you can trigger the alert for users.
• Price optimization
• By applying the price optimization technique using Vald, you can find the most optimized price for your business. You can apply models like GLMs to achieve it and use Vald as a machine learning engine for your business.
• Social analysis
• To analyze the social relationship of users, you can suggest to them their related friends, page recommendations, or other use cases. You can apply different models to analyze social data and use Vald as a recommendation engine for your business.

Advanced use cases

Besides the general use case of Vald or vector search engine, Vald supports a user-defined filter that the user can customize the filter to filter the specific result.

For example when the user chose a man’s t-shirt and the recommended product is going to be searched in Vald.

Without the filtering functionality, the women’s t-shirt may be searched in Vald and displayed because women’s t-shirt is similar to the men’s t-shirt and it is very hard to differentiate the image of men’s and women’s t-shirt.

By implementing the custom filter, you can filter only the man’s t-shirt based on your criteria and needs.