Search and Information Analysis with LLMs

Search and Information Analysis with LLMs

Introduction

Language models can be used to build intelligent search systems and sophisticated information analysis tools. This tutorial explores the different ways to implement these capabilities with LLMs.

Types of Search and Analysis

1. Semantic Search

• Search by concept
• Search by context
• Search by similarity
• Search by relationship
• Search by intent

2. Information Analysis

• Document analysis
• Data analysis
• Trend analysis
• Sentiment analysis
• Context analysis

3. Knowledge Extraction

• Entity extraction
• Relationship extraction
• Fact extraction
• Concept extraction
• Summary extraction

Implementation

1. Semantic Search System

from openai import OpenAI
import json
from typing import List, Dict, Any
import numpy as np

class SemanticSearchSystem:
    def __init__(self, api_key: str):
        self.client = OpenAI(api_key=api_key)
    
    def search(self, query: str, documents: List[Dict[str, Any]], 
               search_type: str = "semantic") -> List[Dict[str, Any]]:
        prompt = self._create_search_prompt(query, documents, search_type)
        
        response = self.client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[
                {"role": "system", "content": "Tu es un expert en recherche sémantique."},
                {"role": "user", "content": prompt}
            ],
            temperature=0.3
        )
        
        return json.loads(response.choices[0].message.content)
    
    def _create_search_prompt(self, query: str, documents: List[Dict[str, Any]], 
                            search_type: str) -> str:
        return f"""
        Effectue une recherche {search_type} pour la requête suivante :
        
        Requête : {query}
        Documents : {json.dumps(documents, indent=2)}
        
        Pour chaque résultat, fournis :
        1. Le document pertinent
        2. Le score de pertinence
        3. Les raisons de la pertinence
        4. Les extraits pertinents
        
        Réponds au format JSON.
        """