Introduction

Before we discuss Vector Search Engine and when to consider using them, let’s start with a simple question.

What is a vector?

In terms of data representation, a vector is a sequence of numbers (or scalars) that can represent data points in a multidimensional space.

It’s not a new concept – its use in mathematics was first introduced by polymath Hermann Grassmann in the mid-19th century. In this article, we will discuss how the vector representation of data works with machine learning, natural language processing (NLP), and image processing. Using vectors to represent data has been around since the 1980s (through Spatial databases). You’ve probably used a spatial database more than once – Google Maps? Two billion users a month rely on that database, which uses vectors to deliver accurate results efficiently to get where they need to go.

So, let’s discuss the difference between a traditional database and a Vector Search Engine. A traditional database is organized on structured data (think rows and columns) and can be paired with a relational database. Traditional databases can be queried with exact match (or range) queries and filtered based on specific criteria, relying on structured query languages (like SQL).

Vector Search Engine represent and store data as vectors, numerical representations of those data points in high-dimensional spaces. High-dimensional spaces are datasets that contain many variables (columns) compared to the number of data points (rows). Vectors capture the underlying meaning of data, enabling similarity searches.

Understanding Vectors

To understand vectors, it starts with embedding.

Embedding is the numerical representation (vector) that captures the meaning and relationships of text, image, and audio files. Each embedding or vector has a high-dimensional aspect. That means that each vector can have hundreds or thousands of dimensions—each dimension captures a different feature or aspect.

When creating those vectors, the goal is to keep data with similar meanings close to each other in the vector space to allow for an optimized similarity search.

Word embedding

Here’s an example of word embedding (provided by Google Gemini):
We have two sentences:

Sentence 1: “The cat is on the mat.”
Sentence 2: “A feline rested on the rug.”

An embedding model (like those used in NLP) would convert these sentences into vectors. These vectors might look something like:

Sentence 1: [0.23, -0.87, 0.56, …, 0.12]
Sentence 2: [0.25, -0.85, 0.58, …, 0.10]

Because the sentences have similar meanings, their vectors would be relatively close to each other in the high-dimensional space.

Image embedding

It is common to use a convolutional neural network (CNN) to extract features of an image and generate image embeddings.¹

Here’s an example of image embedding (provided by Google Gemini):

An image of a dog might be represented by a vector like:
[0.78, 0.12, -0.45, …, 0.91]

A similar vector would be used for another image of a dog. Conversely, a significantly different vector would be used for a picture of a cat.

Digging into convolutional neural networks a little more, they excel at learning hierarchical features from images. Their layers can detect patterns like edges, textures, and more complex object parts.

Image embeddings from convolutional neural networks can be used for:

• Similarity search: Finding images that may be visually similar (think two images with two different dog breeds, both are sitting)
• Image retrieval: Searching for images based on their content
• Image classification: Using the embeddings as features for classification tasks
• Facial recognition: Representing faces as embeddings for comparison

An important aspect of Vector Search Engine is their capacity to store, index, and manage high-dimensional data. This concept is not easy to understand, so I asked Gemini to give me an easy way to describe it.

Imagine you want to describe a simple object, like a color. You might use just one number: its position on a rainbow scale (e.g., 0 for red, 100 for violet). That’s a one-dimensional space.

Now, let’s describe a point on a map.

You need two numbers: its latitude and its longitude. That’s a two-dimensional space.

If you want to describe a cube, use its length, width, and height. That’s a three-dimensional space, and we can easily visualize all of these.

Core concepts of Vector Search Engine

A Vector Search Engine is designed for efficiently storing, indexing, and querying high-dimensional vector embeddings. These embeddings are numerical representations of data, capturing the semantic meaning (or features) of various forms of unstructured data. We covered one of the core concepts above – embeddings, and we’ll dig deeper into a few more core concepts.

High-Dimensional Vectors

• Vector embeddings typically have a large number of dimensions.
• Each dimension captures a nuanced aspect of the data, allowing for a rich and detailed representation.
• An embedding with many dimensions requires specialized indexing (and querying) techniques.

 

Similarity Search:

• Unlike a traditional database (which focuses on exact matches), Vector Search Engine excel at finding data points similar to a query vector.
• Similarity is determined by measuring the distance between vectors in the high-dimensional space.²

Standard distance metrics include:

• Cosine Similarity: measures the angle between two vectors, which are useful for text and high-dimensional data, as it’s insensitive to magnitude. Magnitude in this reference is a calculation that allows for the normalization of vectors, enabling the metric to measure the similarity based on the orientation or direction of the vectors rather than their size.
• Euclidean Distance: The straight-line distance between two points in the vector space. This is the fundamental distance metric in various fields, including mathematics, physics, computer science, and particularly in machine learning algorithms.
• Dot Product: Another measure of vector similarity. It’s used in geometry, physics, and computer science.

Indexing:

• Vector Search Engine employ specialized indexing techniques to perform similarity searches across large datasets of high-dimensional vectors efficiently.
• Traditional indexing methods used in relational databases are not efficient for high-dimensional data.

Standard vector indexing algorithms include:

• Hierarchical Navigable Small Worlds (HNSW): builds a multi-layer graph structure for an efficient approximate nearest neighbor (ANN) search.
• Inverted File Index (IVF): Divides the vector space into partitions and searches within the most relevant partitions.
• Product Quantization (PQ): Compresses vectors to reduce memory usage and speed up distance calculations.

Approximate Nearest Neighbor (ANN) Search:

• Due to the computational cost of finding the exact nearest neighbors in high-dimensional space, Vector Search Engine often use ANN algorithms.
• ANN algorithms sacrifice some accuracy to achieve significantly faster search speeds, which is acceptable for many similarity search applications.

 

These core concepts revolve around representing complex data as high-dimensional vectors, indexing these vectors for efficient similarity search using specialized algorithms and distance metrics, and performing approximate nearest-neighbor searches to retrieve semantically related information quickly.

Use cases for a Vector Search Engine

By now, you’ve got a good understanding of the concepts around Vector Search Engine and their differences.

1. Semantic Search and Information Retrieval: Imagine searching for “pictures of cute fluffy animals” and getting back images that look like cute fluffy animals, even if the captions don’t explicitly use those words. Vector Search Engine excel at understanding the meaning behind text and images. By embedding queries and documents (text, images, audio, video) into a shared vector space, the database can find the most semantically similar items, going beyond simple keyword matching. It powers more intelligent search engines and recommendation systems.

2. Personalized Recommendations: Imagine your favorite streaming service suggesting movies or shows you might like. Vector Search Engine can store user preferences (represented as vectors based on their viewing history, ratings, etc.) and content characteristics (also as vectors). By finding the closest content vectors to a user’s preference vector, the system can deliver highly personalized and relevant recommendations for movies, music, products, articles, and more.

3. Fraud Detection: In the financial world, Vector Search Engine can help teams identify unusual patterns that indicate fraudulent activity. By representing user transactions, account details, and other relevant information as vectors, the system can detect anomalies – data points far away from typical behavior in the vector space. This process can identify potentially fraudulent activities that a rule-based system might miss.

4. Chatbots and Conversational AI: When you interact with a chatbot, it needs to understand the nuances of your questions. Vector Search Engine store embeddings of previous conversations, user queries, and knowledge base articles. The chatbot can retrieve relevant information and generate more contextually appropriate and helpful responses by embedding a new user query and finding the most similar existing vectors.

5. Image and Video Similarity Search: Beyond just text, Vector Search Engine are powerful for finding similar images or videos. By extracting features from visual content and embedding them as vectors, you can perform searches like “find me other dresses that look like this one” or “show me all video clips with a similar scene.” It has applications in e-commerce, media management, and content moderation.

6. Drug Discovery: In pharmaceutical research, Vector Search Engine can help accelerate discovery. On average, it takes 10 years to bring a drug from initial discovery to market approval. Utilizing a Vector Search Engine, in drug discovery, molecules can be represented as vectors based on their properties. Researchers can then search the database for molecules similar to known drugs or target compounds, potentially identifying new drug candidates faster – potentially improving the time to clinical trials (and eventually to market).

 

While Vector Search Engine offer exciting possibilities, there are some common challenges and important considerations to keep in mind when working with them:

Challenges:

1. The Curse of Dimensionality: Vector embeddings can have hundreds (or thousands) of dimensions to capture the richness of the data completely. This high dimensionality can lead to the “curse of dimensionality,” where distance metrics become less meaningful, and overall search performance degrades. Finding the right balance between information richness and dimensionality is critical.³

2. Scalability and Performance: Maintaining efficient search performance can become challenging as the data volume and vectors’ dimensionality increase. Indexing high-dimensional data for fast similarity search is a complex problem. Indexing techniques (like HNSW, Annoy, etc.) have trade-offs regarding build time, query speed, and memory usage. Scaling the database horizontally to handle large datasets while preserving query latency requires careful architectural design.

3. Choosing the Right Embedding Model: The quality of the vector embeddings is paramount. The choice of embedding model (e.g., different transformer models for text, CNN architectures for images) significantly impacts the effectiveness of the Vector Search Engine. Understanding the specific data type and the desired semantic understanding is critical; selecting the right model that is appropriate can have an impact on your success.

4. Indexing and Query Optimization: Efficiently indexing and querying high-dimensional vector data requires specialized techniques. Understanding the characteristics of different indexing algorithms and how they perform under various workloads is essential for optimizing query latency and throughput. Choosing the right distance metric (e.g., cosine similarity, Euclidean distance) for the specific use case also impacts performance and relevance.

5. Data Updates and Maintenance: Real-world data is dynamic. Updating vectors in an extensive database while maintaining index integrity can be challenging. You need to develop strategies for handling inserts, deletes, and updates to avoid performance degradation.

Monitoring and Observability: Monitoring the performance and health of a Vector Search Engine is crucial. Effective monitoring and alerting systems are important for identifying and addressing potential issues.

 

Considerations:

1. Defining the Use Case and Data: The first step is to clearly understand the specific problem you are trying to solve with a Vector Search Engine and the nature of your data. This will guide the choice of embedding model, indexing strategy, and distance metric.

2. Embedding Strategy: How will you generate the vector embeddings? Will you use pre-trained models or train your own? What are the trade-offs regarding accuracy, computational cost, and customization?

3. Indexing Technique Selection: Which indexing algorithm best suits your data volume, dimensionality, query patterns, and performance requirements? Understanding the trade-offs between approximate nearest neighbor (ANN) search algorithms (speed vs. accuracy) is essential.

4. Distance Metric: Which distance metric aligns best with the notion of similarity for your data? Cosine similarity is often used for text embeddings, while Euclidean distance might be more appropriate for other data types.

5. Accuracy vs. Speed Trade-off: Many ANN search algorithms offer a trade-off between search speed and accuracy (recall). You need to determine the acceptable level of approximation for your application.

6. Integration with Existing Systems: How will a Vector Search Engine integrate with your existing data pipeline and infrastructure? Considerations should include ingestion, query execution, and synchronization.

7. Cost and Infrastructure: Vector Search Engine can have specific infrastructure requirements – especially for large-scale deployments. You need to fully understand the costs associated with storage, compute resources, and managed service offerings.

8. Security and Privacy: If your Vector Search Engine contains sensitive information, your team needs to ensure that the appropriate security measures are in place to protect the data. Standard security tactics like encryption and access control are necessary to ensure that you’re meeting compliance requirements.

Whether your focus is on developing a Retrieval-Augmented Generation (RAG) for an LLM or a semantic search feature, keeping these challenges and considerations in mind – you’ll be able to tap into the full power of Vector Search Engine.

Successfully integrating a Vector Search Engine into your product or feature hinges on a clear understanding of its capabilities and a planned approach to implementation. You will need to clearly define your use case and have a solid understanding of your data. Those two steps will inform your decisions regarding the following:

1. Embedding Strategy: Which embedding model (pre-trained or custom-trained) will best represent your data’s semantic meaning for your specific problem?

2. Indexing Technique: Given your data volume, dimensionality, and query speed requirements, select the most appropriate approximate nearest neighbor (ANN) indexing algorithm (e.g., HNSW for balanced performance, PQ for memory efficiency).

3. Distance Metric: Choose the distance metric (e.g., Cosine Similarity for semantic text search, Euclidean for more general feature comparisons) that accurately reflects ‘similarity’ for your application.

Outside of the initial setup, you will have ongoing operational considerations that need to be addressed.

From planning for data updates to establishing monitoring and observability to track performance (and resource utilization) and ensure your chosen solution aligns with your scalability, cost, security, and privacy requirements, you can harness the full power of Vector Search Engine for intelligent data retrieval and advanced applications by addressing these factors proactively.

Some analysts predict that the Global LLM and Agentic AI markets could reach $36 billion and $47 billion in revenue by 2030, respectively. Understanding the infrastructure and underlying technology that will support that growth is important. Making the right decisions now will ensure scalability and future growth.

This article was a collaborative effort between Qdrant and Position².

Qdrant is a leading open-source, high-performance Vector Search Engine and vector search engine. Built to power semantic search, recommendation engines, Agentic AI memory and AI-driven retrieval. Qdrant offers both a managed service (Qdrant Cloud) or the option to run it on-premise.

Position² is an AI-first Growth Marketing firm that drives growth for innovative brands. We understand your needs, and that understanding drives us to develop the right marketing mix for your company. Our unique process of mapping content to each customer journey stage and deep knowledge of AdTech & Martech platforms deliver exceptional results for your ABM programs. We power our integrated marketing campaigns with the most advanced tactics in Paid Acquisition, Marketing Automation, CRM, Analytics, and cutting-edge Content Creation, Design, and Web Development.

The buzz over the last year and a half around AI has raised awareness of Generative AI systems (Gen AI) and large language models (LLMs) and shone a spotlight on the issues and gaps in these systems, specifically regarding bias and discriminatory practices.

Here are some definitions (courtesy of Google’s Gemini) so we’re all on the same page.

Gen AI: It is a system that uses machine learning, algorithms, and training data to generate content based on user inputs or prompts. Content can include text, images, music, video, code, and more. Gen AI models are bidirectional, taking cues from human inputs and prompts to shape their content production.

LLMs: Are programs that use deep learning to analyze large amounts of data to understand human language and perform natural language processing (NLP) tasks. LLMs can recognize, generate, summarize, translate, and predict text and other content. They can also understand how words, sentences, and characters work together and recognize distinctions between different pieces of content.

Types of bias

By now, we’ve all seen the headlines and examples of clear biases in the output from prominent Gen AI solutions and LLMs. Earlier this year, UNESCO shared a study highlighting sex and gender biases in prominent LLMs.

The list of potential biases is long, to say the least. They can include:

According to the CompTIA 2024 AI Outlook, 22% of companies are pursuing the integration of AI across a wide variety of products and workflows, and another 33% are engaging in limited implementation of AI.

With so many companies and government entities considering adoption or having already implemented AI and LLMs into their products, processes, and workflows concerns about potential bias and discriminatory practices grow.

The table above clearly illustrates that biases and discrimination have a direct impact on all people, whether they are part of a historically marginalized community or not.

Sources of bias

Where do biases in our AI systems come from? It’d be easy to declare this a technology issue, but it stems from human, systemic, and institutionalized biases resident in our digitized world. When systems use web data and information for training, they reflect the biases present in that training data and it becomes part of any output from that system.

As for systemic or institutional bias – that bias can come directly from that institution’s data and inadvertently exclude people. An example of this is a hiring tool that learned the features of a company’s employees (predominantly men) rejected women applicants for spurious and discriminatory reasons; resumes with the word “women’s,” such as “women’s chess club captain,” were penalized in the candidate ranking.¹

The impact of bias

Real-world examples show that these ingrained biases can impact a person’s financial well-being, the ability to get hired, medical treatment, mental health, and an overall potential loss of opportunity.

What considerations can prevent unconscious AI bias?

To ensure fairer AI systems – it’s universally understood that data privacy, transparency, and accountability have to be at the forefront of all internal use policies.

These policies must be coupled with the ability to detect existing biases in those systems and accompanied by mitigation plans.

So what does implementation look like?

Data Privacy

Eliminating bias in AI begins with data anonymization. Data anonymization is nothing new; industries like Healthcare, Financial Services, and Telecommunications have been doing it effectively for years. Government organizations also use it to protect Personally Identifiable Information (PII).

One component of data anonymization that can preserve privacy while preserving data utility for AI applications is pseudonymization. Pseudonymization anonymizes data by replacing any identifying information with a pseudonymous identifier. Across industries, Pseudonymization protects personal information like first and last names, addresses, and Social Security Numbers.

Additional techniques that can help preserve privacy are federated learning and differential privacy, which enable data analysis without compromising privacy.

Data anonymization has pros and cons. Teams must decide how they utilize PII data to ensure they pursue the right approach.

Transparency

Transparency is key to reducing bias and discrimination in AI systems. It’s a pretty broad term – what it means in terms of AI systems is there is an openness and clarity of AI systems, this applies to their algorithms, decision-making processes, and data sources. Areas to focus on for data transparency include:

• Data collection and processing – What data sources are you using? How is the data collected and processed?
• Assessments – Conduct assessments of these systems for bias and discrimination, including any remediation efforts. Publicly publish any assessments of the system (and maintain records of these).
• Privacy policy – Develop and publish your privacy policy for system users.
• Evidence compliance with all applicable regulations in the regions you operate.

Transparency is a critical component of responsible AI, as it helps ensure that decisions are made fairly, ethically, and without bias—most importantly, that these efforts are shared with system users.

Accountability

Accountability involves establishing processes and frameworks to create an oversight and monitoring function over these systems to ensure that they operate as intended and do not cause unintended consequences.

• Creating a governance framework supports both accountability and transparency. It establishes an oversight body that reviews the development, deployment, and day-to-day performance of AI systems.
• Define ethical guidelines for staff who build and update these systems.

The common elements between transparency and accountability are that both of these functions require active monitoring and reporting on the adherence to stated policies, guidance, and compliance with all applicable regulations. As with most technology solutions, these need to be audited and tested to ensure the controls are working as intended, not causing new issues, and any subsequent bias is identified.

In closing

Bias and discrimination exist in these systems, and the solutions that can help reduce them in AI systems are nothing ‘new.’ They are tried-and-true approaches that multiple industries have used for years to protect sensitive data and create and maintain the confidence of consumers, regulators, and investors. As these systems are being developed, companies need to consider whether speed to market is more important than developing systems that have enshrined fairness and are not actively reinforcing societal bias.

1 – Jeffrey Dastin. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. Oct. 10, 2018.
https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G

Defining Large Language Models (LLM)

Although LLMs have been around since the 1950s, let’s start by defining what they are. A Large Language Model is a type of AI program that uses deep learning and machine learning to perform natural language processing (NLP) tasks. LLMs are trained on large amounts of data to understand and generate human language.

LLMs are built on transformer models (because of the massive datasets they are trained on), which are neural networks that use self-attention to learn context and detect relationships between elements in a sequence.

What is a Marketing LLM?

Marketing-focused LLMs are specifically trained on your data. The data you use to train can be your marketing content, campaigns, analytics, and customer data. These LLMs can ingest and train on large amounts of data.

The clear advantage of leveraging AI in your Marketing Strategies

LLMs and Generative AI aren’t going anywhere, these tools have been around for a while, and they will continue to make an impact on multiple industries. Across those industries and companies, Gen AI is being utilized to create content (like social posts, ad copy, and blogs), creative assets, templates, product descriptions, and metadata to create efficiencies in their marketing and creative teams as well as improve speed to market for test iterations.

In a recent Web FX poll, 50% of the 200 senior marketing and Sales leaders surveyed are using AI in their marketing strategies.

Let’s dive into specific use cases.

Content creation

Over half of marketers (55%) in an eMarketer study use AI for content ideation, indicating the technology’s value in generating new creative concepts and strategies. Let’s look at some examples of how a Marketing LLM can impact your campaign creative:

• Personalization at scale: Marketing LLMs can quickly tailor content to specific audiences, ensuring that the right message gets to the right target.
• Scalable content: You can quickly develop and version existing content for use in omnichannel campaigns.
• Optimizing existing content: With speed and efficiency you can quickly optimize content to improve your engagement and conversion rates on existing campaigns.

Data-Driven Insights and Analysis

In the same eMarketer study from earlier this year, almost half (45%) of the respondents use AI to gather consumer insights, showcasing AI’s role in understanding market trends and customer behaviors. Marketing LLMs can deliver value with:

• Processing customer data with NLP: You can utilize an LLM to clean and correct customer data or personalize responses tied to customer interactions.
• Analyzing customer sentiment: Review and analyze sentiment based on classifying inbound communication.
• Identifying trends and opportunities: With so many touchpoints to process, it’s difficult for teams to identify trends. An LLM can process unstructured data like customer feedback, social media posts, and online reviews – identifying trends and opportunities your team might miss.

Chatbots and Conversational AI

There are clear differences between chatbots and conversational AI. Chatbots use predefined conversation flows that are limited. Think about the last chatbot you used—there are clear limits to what it can answer based on your input. Simply put, Conversational AI uses machine learning and NLP to provide a more robust, personalized experience. There are use cases for both – broadly they can be used for:

• Enhancing customer experience
• Being an always-on, 24/7 support function
• Being a key component in Sales or Business Development flows

Optimization and Testing

Testing your marketing is critical to increasing its effectiveness, reducing waste on strategies that don’t work, and increasing the ROI of your campaigns – Marketing LLMs are a great tool to help you with all of that.

• Creative copy and content optimization
• A/B testing and experimentation strategies

Integration with Marketing Automation

Marketing LLMs can have an impact on Marketing automation – from identifying prospects that can convert to finding the best time to market to them to get that conversion. They can also assist in:

• Streamlining workflows and processes
• Personalized nurturing campaigns
• Enhancing marketing efficiency

Challenges and Considerations

In 2017, The Economist declared that data was the most valuable resource. Although the article focused on US tech companies and their use and storage of massive amounts of data – it’s a very relevant topic for AI and Marketing LLMs.

Data quality and privacy concerns are top considerations when using a Marketing LLM – the data the system is trained has a direct impact on the results. Poor data quality can quickly cause issues with an LLM. These issues can be mitigated before they even start by:

• Data cleaning: Identify, correct, and remove inconsistencies, errors, and missing values from your data set.
• Data collection: Establishing strong controls (process and procedures) on the collection of data.
• In addition to the above, creating rules and procedures for data validation, integration, and monitoring.

Privacy concerns: You need to ensure that training data does not contain personally identifiable information (PII)—if it does, it increases the risk of being exposed by hackers and is subject to multiple laws worldwide. The best policy regarding privacy concerns with PII is to ​​ensure that data is anonymized and that consent is obtained where necessary.

Ethical and responsible AI implementation: We’ve all seen the news stories about bias and discrimination in LLMs, and it’s clear that improved data quality and human oversight are critical to preventing bias, discrimination, and misleading information that can hurt historically marginalized communities.

Future Outlook

As LLMs continue to evolve, we can count on a future with them making an impact on both consumers and businesses. Advancements in LLMs will include a deeper level of personalization, advanced conversational capability, and an industry-level focus on specific industries.

Today, LLMs can provide businesses with a clear advantage by enhancing productivity by optimizing operations, reducing costs, and improving delivery. LLMs can process and analyze vast amounts of data faster and more efficiently than people, driving quicker, more accurate decision-making.

Conclusion

When it comes to Marketing LLMs, there are so many practical applications that can uplevel your marketing and customer experience for your customers—and these capabilities will only grow as new LLMs enter the marketplace.

Although there are some clear concerns about data quality, potential biases, and privacy, these can all be mitigated by strong controls (process and procedure) combined with actual human oversight.

We believe there’s a clear competitive advantage to using Marketing LLMs and that the benefits clearly outweigh the concerns.