Turning chat conversations into knowledge-base documents
The same pipeline that reads call transcripts, pointed at a Facebook page inbox. It works — but this channel raises a question calls never do: who actually wrote the answer?
What it does
Each conversation is read once and reduced to atomic claims — single, self-contained statements of fact about a product, policy, procedure, limit or price. Claims about only one customer are discarded. The remainder are grouped by subject (the thing they are about) and by kind (the form of knowledge they represent), and each group is written up as one section of a document.
A page inbox mostly routes people, it does not answer them. The single most common agent message in this corpus appears 128 times: “thank you, an agent will reply shortly”. Of 111 conversations processed, 39 contain nothing but canned replies and 10 more are pure lead capture. The knowledge is real but sparse, and most of the work is telling it apart from the boilerplate around it.
Document shape
Every document follows the same three-level structure as the call POC, chosen to match how the backend’s structure-aware splitter chunks and embeds content.
# R-CRM ← subject
## Pricing ← kind
### Package pricing ← task
- ...The splitter builds a breadcrumb of the full heading path and prepends it to every chunk, so a retrieved passage arrives carrying its product name and task — inside the embedded text, not merely as metadata.
What was measured, not assumed
| Decision | Why |
|---|---|
| The cutoff date was derived, not guessed | Splitting the three bot markers apart dated the shutdown to 8 July, from the last disclaimer. Collapsing them into one boolean gives the wrong answer, because the only marker still present at the end of the corpus is the one carrying no knowledge either way. A string with 183 near-identical instances is a template; one attached to 77 different bodies of text is a behaviour, and only a behaviour can be switched off. |
| Provenance is tagged, not filtered away | Merging the two cohorts silently would have looked far better — 15 documents instead of 4 — while making the largest document the least attributable one. Volume and provenance moved in opposite directions, which a single blended number hides. |
| A pasted product link is not evidence | Agents send the same company link regardless of what they are discussing — a CRM price arrives beside a website-product link in two separate conversations. Trusting the link would have corrected one mis-attributed price and broken two others. |
| Report the uncertainty, then act on it in code | Asking the model to ignore a misleading signal failed. Asking it to report whether the subject was stated or inferred worked — and that report is what lets deterministic code resolve the collisions afterwards. Prompts are weak at suppression and strong at self-report. |
| Routing is explicitly not knowledge | In a phone call, an agent talking is usually an agent explaining. Here the most common agent utterance is a request for a callback number. Without an explicit prohibition the extractor turns those into contact facts in nearly every thread, burying the real content under confident-looking noise. |
| Grouping decided by the facts, not the label |
Extraction saw each conversation in isolation, so one product arrived as
Readyplanet website builder, Ready-made website services,
Turnkey website service and Website design. Showing sample
facts beneath each label let them be merged from evidence rather than wording
— 64 labels down to 26 subjects.
|