Quick answer:
An embedding is a piece of data (a sentence, a document, an image, a song) translated by a neural network into a vector: a long list of numbers that captures its meaning. The magic property: things with similar meaning get similar numbers, so “cheap flights to Rome” and “affordable airfare to Italy” end up neighbors, even though they share almost no words.
Embeddings are how software gets a handle on meaning without understanding anything the way we do. They’re the workhorse behind semantic search, recommendations, and RAG.
What is an embedding?
Computers only crunch numbers, and words aren’t numbers. The old workaround was counting: represent a document by which words it contains. That breaks the moment vocabulary shifts, because “car” and “automobile” count as totally different things.
Embeddings fix this by learning the numbers. A model trained on enormous amounts of text learns to assign each input a position in a high-dimensional space such that similarity in meaning becomes closeness in position. The output for one input, that list of maybe 768 or 1,536 numbers, is the embedding. It’s a learned summary of what the thing means, in coordinates.
Where did embeddings come from?
The idea that meaning could live in vectors goes back decades in linguistics (“you shall know a word by the company it keeps”). The modern era starts with word2vec, published by Tomas Mikolov’s team at Google in 2013. It embedded single words, and it produced the demo that made everyone sit up: take the vector for king, subtract man, add woman, and the nearest vector is queen. Arithmetic on meaning.
From there: GloVe, then transformer-based models (BERT, 2018) that embed whole sentences with context, so “bank” in “river bank” and “bank account” finally get different vectors. Today you mostly consume embeddings as an API call: OpenAI’s text-embedding-3 models, Cohere’s, or open-source models you can run yourself (the MTEB leaderboard on Hugging Face is where those get compared).
What do people build with them?
- Semantic search: embed the query, embed the documents, return the nearest ones. Ranks by meaning instead of keyword luck.
- RAG: the retrieval half of retrieval-augmented generation is exactly that search, feeding an LLM the passages it should answer from.
- Recommendations: embed users and items in the same space; recommend what’s nearby.
- Clustering and dedupe: support tickets that pile up in one region of the space are the same issue; 2 near-identical vectors are probably the same record twice.
- Classification: embeddings as ready-made features for downstream models, which is roughly what a feature pipeline feeds on.
What should a practitioner actually worry about?
Three things, in order. Model choice: embeddings from different models live in incompatible spaces, so you can’t compare a vector from model A with one from model B, ever. Pick one, note its version. Re-embedding cost: when you switch or upgrade models, every stored document must be re-embedded. Budget for that day, because it comes. Chunking: for long documents you embed pieces, and how you cut those pieces quietly dominates retrieval quality. Bad chunking with a great model loses to decent chunking with an average one.
Dimensions, by contrast, are mostly a storage-and-speed dial (more dimensions, more nuance, more cost). OpenAI’s current models even let you truncate dimensions on request.
What are the benefits and drawbacks of embeddings?
Benefits of embeddings
Semantic understanding as an API call
A capability that took research labs decades now costs fractions of a cent per document. The barrier to “search that understands meaning” is an afternoon of integration.
One representation, many uses
The same stored vectors power search, recommendations, clustering, dedupe, and classification features. Compute once, reuse everywhere: rare economics in ML.
Language and phrasing stop mattering
Synonyms, typos, and even cross-language queries land near their targets. Multilingual models put “delivery failed” and “livraison échouée” in the same neighborhood.
They compress honestly
A 30-page document becomes 1,536 numbers that preserve what it’s about. For retrieval purposes, that’s a wildly efficient summary.
Drawbacks of embeddings
Locked to their model
Vectors from different models, or different versions of the same model, can’t be compared. Every model upgrade drags a full re-embedding of the corpus behind it.
Nuance gets averaged away
Negations (“do NOT ship to this address”) and fine distinctions can embed suspiciously close to their opposites. Similar vibes, opposite meanings: a known failure family.
Debugging is statistical
You can’t inspect a vector and see why retrieval went wrong. Quality work means building evaluation sets and measuring, which most teams skip until it hurts.
Freshness is your job
Edit the document and the old vector keeps getting retrieved until your pipeline re-embeds it. Stale embeddings are the vector world’s version of a broken sync, minus the error message.
Where do embeddings sit in the data stack?
They’ve become a column type. You compute them in your pipeline (often at ingestion, the same place other transformations happen), store them in a vector database or in Postgres with pgvector, and index them for fast nearest-neighbor search. Warehouses joined in too: Snowflake, BigQuery, and Databricks all ship embedding generation and vector search natively now.
That’s the quiet takeaway: embeddings stopped being an ML-team specialty and became regular data infrastructure, one more thing your pipeline produces and your database indexes.