signature - the schema learned from the rows

The strongest idea in the library, and the reason a reading survives an issuer nobody has seen.

A table describes itself: the date column holds a date on every row, the money column an amount, the columns that are always filled are filled. A row that breaks that is not a row of the table — it is a total, a balance, a footer.

import { findRowAnomalies, thresholdsFor } from 'truecopy/signature';

const anomalies = findRowAnomalies(rows, {
  kindOf: (cell) => (isDate(cell) ? 'date' : isAmount(cell) ? 'amount' : 'text'),
  thresholds: {
    ...thresholdsFor(['amount', 'text'], 0.6),
    date: { share: 0.6, emptyIsAnomalyAbove: 0.7 }
  }
});

anomalies[5]; // { cause: 'empty', column: 0, kind: 'date' }

Why this beats a list of forbidden words

A word list only recognises what was put in it, which means it recognises yesterday’s documents. It reads TOTAL and misses SOMME DES DÉBITS, Nouveau solde, Saldo, Übertrag.

The signature reads no label at all. It notices that one row has an empty date where every other row has one, and that is true in every language and at every issuer.

The thresholds

share	share of cells of that kind past which the column is of that kind
emptyIsAnomalyAbove	share of filled cells past which an empty cell is an anomaly, for a column of this kind

emptyIsAnomalyAbove is the one that catches the total line. On a statement, a date column nearly always filled with one cell empty is a balance line — it has no value date. On a money column, empty is ordinary, so you leave it out there.

thresholdsFor(kinds, share) gives every kind the same share and is spread over for the one kind that needs a rule of its own — a signature written out repeats { share: 0.6 } until the shape drowns the one number that matters.

Two refusals built in

Below minimumRows (five by default) it returns null. Too few rows to learn a shape from, and it is the caller’s call what to do with a table that short. Saying nothing beats learning from a sample that teaches nothing.

judgedColumns narrows the judgement to the columns whose role is known. A column whose content is a mystery may not condemn anyone.

findRowAnomalies(rows, signature, (column) => roles[column] !== undefined);

What an anomaly says

type RowAnomaly =
  | { cause: 'wrong-kind'; column: number; expected: string }
  | { cause: 'empty'; column: number; kind?: string };

kind tells which kind of column the hole was in, when it had one: a hole in a date column does not read like a hole in any old column. describeAnomaly turns either into a sentence.

Seeing the shares the thresholds are compared against

sharesByKind returns the share each kind holds in a column, before any threshold is applied — the same numbers dominantKind reads to decide.

import { sharesByKind } from 'truecopy/signature';

sharesByKind(signature); // { date: 0.94, amount: 0.03, text: 0.03 }

It is the answer to the only question a failing signature raises: a column was not recognised, and the threshold says nothing about how close it came. A kind at 0.58 against a share of 0.6 is a threshold to lower; one at 0.11 is a kindOf that does not recognise the cells at all. The two look identical from the outside and want opposite fixes.

A share out of what?

shareOfKind is always a share of every row, which is the cautious reading and the one this module needs: a column of dates empty on a third of its rows is not a column of dates, it is a mess, and judging rows against it would condemn the honest ones.

Role inference asks the other question and takes its share out of the filled cells. The two are not interchangeable — measured on a real statement, the same column scores 0.27 one way and 0.62 the other, and at a threshold of a half the first answer loses it outright.