Costs, Latency, and Updates in Production

A RAG service goes into production with 300 users. Two weeks later, the monthly bill reaches four figures and median latency exceeds five seconds at peak hours. Nobody had calculated the cost per query, because during development each test cost a hundredth of a euro. That same month, a document is removed from the internal site because of a legal error; it remains in the vector index, and the system keeps citing it for three weeks.

These two problems share a common cause: the prototype has no lifecycle. It indexes once, answers a few dozen times, and nothing changes. A production service has to absorb documents that evolve, documents that disappear, and load spikes, and it has to account for its cost. These constraints are not something to add at the end: they drive architecture decisions that are very expensive to revisit after the fact.

Calculating the Cost per Query

Break it down before optimizing. A query consumes input tokens, output tokens, an embedding of the question, and possibly a reranking step.

TARIFS = {                     # euros pour 1 000 jetons, à ajuster
    "entree": 0.00015,
    "sortie": 0.00060,
    "plongement": 0.00002,
}

def cout_requete(jetons_contexte, jetons_question, jetons_reponse, reclassement_local=True):
    entree = (jetons_contexte + jetons_question + 250)   # 250 pour les consignes
    total = (entree / 1000) * TARIFS["entree"]
    total += (jetons_reponse / 1000) * TARIFS["sortie"]
    total += (jetons_question / 1000) * TARIFS["plongement"]
    return total

unitaire = cout_requete(3000, 30, 300)
print(f"{unitaire*1000:.2f} € pour 1 000 requêtes")
print(f"{unitaire*300*20*22:.0f} € par mois pour 300 personnes, 20 requêtes par jour ouvré")