Vector search vs. Keyword search — War of the worlds vs. we come in peace

  1. War of the worlds vs. we come in peace
  2. Data structures and algorithms
  3. LSMT-IVF for Billion-Scale Approximate Nearest Neighbor Search
  4. Benchmarking

Vector vs. keyword search 1: War of the worlds vs. we come in peace

Keyword search

  • high indexing speed (for large document numbers)
  • medium index size
  • high query speed (for large document numbers)
  • good scaling (for large document numbers)
  • perfect precision (for exact keyword match)
  • recall: perfect for exact keyword match, low for semantic meaning
  • unable to capture meaning and similarity
  • efficient and lossless for exact keyword and phrase search

Vector search

  • slower indexing speed (for large document numbers)
  • large index size
  • slower query speed (for large document numbers)
  • limited scaling (for large document numbers)
  • lower precision (for exact keyword match)
  • recall: high for semantic meaning (80/90%), medium for exact keyword match
  • able to capture meaning and similarity
  • inefficient and lossy for exact keyword and phrase search

Tip of the iceberg

Combine and conquer

--

--

Founder SeekStorm (Search-as-a-Service), FAROO (P2P Search) https://seekstorm.com https://github.com/wolfgarbe https://www.quora.com/profile/Wolf-Garbe

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store