# Stylometry validation results

Generated by `research/stylometry-validation.ts` from SBLGNT lemmas (MorphGNT).  Do not edit by hand.

Word counts:  Ro 7055, 1Co 6812, 2Co 4473, Ga 2226, Php 1626, 1Th 1473, Phm 334, Eph 2416, Col 1580, 2Th 820, 1Ti 1591, 2Ti 1235, Tit 659, Heb 4935, Jas 1739, 1Pe 1678, 2Pe 1098, 1Jn 2137, Jud 459, Ac 18412, Re 9833.

## 50 most frequent lemmas

### Test A:  distance from Paul (one-group outlier test)

Every Paul letter is held out in turn;  the profile is the other six.  In each fold the held-out letter's blocks (positives) and every non-Pauline block (negatives) are scored against the same profile, so both groups face identical conditions.  AUC is the chance that a non-Pauline block lies farther from Paul than a genuine Paul block scored in the same fold (0.5 = no separation, 1.0 = perfect).

| Block length (words) | Paul blocks | Non-Pauline blocks | AUC | Non-Pauline scores flagged when 5% of Paul blocks are | Same, 10% | Method check:  AUC against Acts and Revelation blocks |
|---|---|---|---|---|---|---|
| 250 | 92 | 44 | 0.44 | 14% | 16% | 0.73 |
| 400 | 57 | 28 | 0.46 | 18% | 18% | 0.81 |
| 600 | 36 | 16 | 0.53 | 20% | 21% | 0.90 |
| 800 | 26 | 13 | 0.52 | 18% | 18% | 0.91 |
| 1200 | 16 | 7 | 0.50 | 18% | 18% | 0.98 |
| 1600 | 12 | 6 | 0.56 | 19% | 33% | 0.96 |

Disputed letters, for information only:  each block's fold-averaged distance as a percentile of the genuine-Paul blocks of the same length (higher = less like Paul;  above 95 would be flagged at the 5% cut-off).  These are not attributions:  see the AUC column for how little a high or low percentile means.

| Block length | Letter | Blocks | Percentile of each block |
|---|---|---|---|
| 250 | Eph | 9 | 99, 62, 35, 48, 43, 14, 4, 86, 84 |
| 250 | Col | 6 | 85, 90, 63, 43, 30, 29 |
| 250 | 2Th | 3 | 96, 5, 29 |
| 250 | 1Ti | 6 | 22, 55, 30, 28, 45, 13 |
| 250 | 2Ti | 4 | 71, 4, 29, 28 |
| 250 | Tit | 2 | 18, 46 |
| 400 | Eph | 6 | 100, 26, 61, 26, 53, 70 |
| 400 | Col | 3 | 95, 89, 39 |
| 400 | 2Th | 2 | 53, 25 |
| 400 | 1Ti | 3 | 44, 19, 44 |
| 400 | 2Ti | 3 | 46, 9, 26 |
| 400 | Tit | 1 | 39 |
| 600 | Eph | 4 | 100, 61, 8, 89 |
| 600 | Col | 2 | 100, 72 |
| 600 | 2Th | 1 | 67 |
| 600 | 1Ti | 2 | 33, 33 |
| 600 | 2Ti | 2 | 17, 33 |
| 600 | Tit | 1 | 61 |
| 800 | Eph | 3 | 100, 65, 54 |
| 800 | Col | 1 | 100 |
| 800 | 2Th | 1 | 54 |
| 800 | 1Ti | 1 | 27 |
| 800 | 2Ti | 1 | 0 |
| 1200 | Eph | 2 | 100, 38 |
| 1200 | Col | 1 | 81 |
| 1200 | 1Ti | 1 | 38 |
| 1200 | 2Ti | 1 | 6 |
| 1600 | Eph | 1 | 92 |

### Test B:  Paul versus other letter writers (two-group nearest centroid)

Every book (Paul's seven and the six non-Pauline letters) is held out in turn;  its blocks are assigned to whichever reference group, Paul or non-Paul, has the nearer centroid.  Correct = Paul blocks assigned to Paul, non-Pauline blocks assigned to non-Paul.

| Block length (words) | Paul blocks correct | Non-Pauline blocks correct | Balanced accuracy |
|---|---|---|---|
| 250 | 45 of 92 (49%) | 30 of 44 (68%) | 59% |
| 400 | 30 of 57 (53%) | 17 of 28 (61%) | 57% |
| 600 | 20 of 36 (56%) | 11 of 16 (69%) | 62% |
| 800 | 16 of 26 (62%) | 10 of 13 (77%) | 69% |
| 1200 | 9 of 16 (56%) | 4 of 7 (57%) | 57% |
| 1600 | 6 of 12 (50%) | 4 of 6 (67%) | 58% |

## 100 most frequent lemmas

### Test A:  distance from Paul (one-group outlier test)

Every Paul letter is held out in turn;  the profile is the other six.  In each fold the held-out letter's blocks (positives) and every non-Pauline block (negatives) are scored against the same profile, so both groups face identical conditions.  AUC is the chance that a non-Pauline block lies farther from Paul than a genuine Paul block scored in the same fold (0.5 = no separation, 1.0 = perfect).

| Block length (words) | Paul blocks | Non-Pauline blocks | AUC | Non-Pauline scores flagged when 5% of Paul blocks are | Same, 10% | Method check:  AUC against Acts and Revelation blocks |
|---|---|---|---|---|---|---|
| 250 | 92 | 44 | 0.40 | 10% | 16% | 0.58 |
| 400 | 57 | 28 | 0.43 | 15% | 17% | 0.68 |
| 600 | 36 | 16 | 0.52 | 22% | 22% | 0.81 |
| 800 | 26 | 13 | 0.48 | 15% | 16% | 0.82 |
| 1200 | 16 | 7 | 0.57 | 22% | 22% | 0.91 |
| 1600 | 12 | 6 | 0.54 | 19% | 19% | 0.94 |

Disputed letters, for information only:  each block's fold-averaged distance as a percentile of the genuine-Paul blocks of the same length (higher = less like Paul;  above 95 would be flagged at the 5% cut-off).  These are not attributions:  see the AUC column for how little a high or low percentile means.

| Block length | Letter | Blocks | Percentile of each block |
|---|---|---|---|
| 250 | Eph | 9 | 92, 55, 28, 55, 55, 3, 3, 80, 68 |
| 250 | Col | 6 | 80, 78, 24, 18, 13, 13 |
| 250 | 2Th | 3 | 86, 0, 50 |
| 250 | 1Ti | 6 | 10, 40, 12, 18, 13, 0 |
| 250 | 2Ti | 4 | 65, 5, 9, 7 |
| 250 | Tit | 2 | 5, 48 |
| 400 | Eph | 6 | 91, 25, 81, 11, 40, 58 |
| 400 | Col | 3 | 84, 42, 12 |
| 400 | 2Th | 2 | 40, 35 |
| 400 | 1Ti | 3 | 19, 32, 19 |
| 400 | 2Ti | 3 | 42, 2, 19 |
| 400 | Tit | 1 | 33 |
| 600 | Eph | 4 | 86, 78, 3, 81 |
| 600 | Col | 2 | 78, 50 |
| 600 | 2Th | 1 | 58 |
| 600 | 1Ti | 2 | 17, 3 |
| 600 | 2Ti | 2 | 25, 100 |
| 600 | Tit | 1 | 56 |
| 800 | Eph | 3 | 81, 65, 38 |
| 800 | Col | 1 | 69 |
| 800 | 2Th | 1 | 38 |
| 800 | 1Ti | 1 | 12 |
| 800 | 2Ti | 1 | 4 |
| 1200 | Eph | 2 | 88, 25 |
| 1200 | Col | 1 | 56 |
| 1200 | 1Ti | 1 | 19 |
| 1200 | 2Ti | 1 | 100 |
| 1600 | Eph | 1 | 75 |

### Test B:  Paul versus other letter writers (two-group nearest centroid)

Every book (Paul's seven and the six non-Pauline letters) is held out in turn;  its blocks are assigned to whichever reference group, Paul or non-Paul, has the nearer centroid.  Correct = Paul blocks assigned to Paul, non-Pauline blocks assigned to non-Paul.

| Block length (words) | Paul blocks correct | Non-Pauline blocks correct | Balanced accuracy |
|---|---|---|---|
| 250 | 48 of 92 (52%) | 34 of 44 (77%) | 65% |
| 400 | 31 of 57 (54%) | 20 of 28 (71%) | 63% |
| 600 | 25 of 36 (69%) | 9 of 16 (56%) | 63% |
| 800 | 15 of 26 (58%) | 10 of 13 (77%) | 67% |
| 1200 | 10 of 16 (63%) | 2 of 7 (29%) | 46% |
| 1600 | 9 of 12 (75%) | 3 of 6 (50%) | 63% |

## 150 most frequent lemmas

### Test A:  distance from Paul (one-group outlier test)

Every Paul letter is held out in turn;  the profile is the other six.  In each fold the held-out letter's blocks (positives) and every non-Pauline block (negatives) are scored against the same profile, so both groups face identical conditions.  AUC is the chance that a non-Pauline block lies farther from Paul than a genuine Paul block scored in the same fold (0.5 = no separation, 1.0 = perfect).

| Block length (words) | Paul blocks | Non-Pauline blocks | AUC | Non-Pauline scores flagged when 5% of Paul blocks are | Same, 10% | Method check:  AUC against Acts and Revelation blocks |
|---|---|---|---|---|---|---|
| 250 | 92 | 44 | 0.35 | 4% | 4% | 0.51 |
| 400 | 57 | 28 | 0.36 | 13% | 14% | 0.61 |
| 600 | 36 | 16 | 0.44 | 21% | 24% | 0.76 |
| 800 | 26 | 13 | 0.37 | 13% | 16% | 0.72 |
| 1200 | 16 | 7 | 0.56 | 27% | 27% | 0.86 |
| 1600 | 12 | 6 | 0.38 | 2% | 19% | 0.82 |

Disputed letters, for information only:  each block's fold-averaged distance as a percentile of the genuine-Paul blocks of the same length (higher = less like Paul;  above 95 would be flagged at the 5% cut-off).  These are not attributions:  see the AUC column for how little a high or low percentile means.

| Block length | Letter | Blocks | Percentile of each block |
|---|---|---|---|
| 250 | Eph | 9 | 78, 36, 38, 41, 47, 2, 2, 64, 75 |
| 250 | Col | 6 | 76, 43, 30, 30, 15, 17 |
| 250 | 2Th | 3 | 76, 1, 39 |
| 250 | 1Ti | 6 | 7, 30, 23, 4, 8, 2 |
| 250 | 2Ti | 4 | 76, 2, 8, 2 |
| 250 | Tit | 2 | 9, 42 |
| 400 | Eph | 6 | 70, 26, 68, 4, 19, 54 |
| 400 | Col | 3 | 60, 35, 14 |
| 400 | 2Th | 2 | 23, 39 |
| 400 | 1Ti | 3 | 23, 23, 7 |
| 400 | 2Ti | 3 | 47, 2, 7 |
| 400 | Tit | 1 | 40 |
| 600 | Eph | 4 | 69, 78, 3, 78 |
| 600 | Col | 2 | 61, 36 |
| 600 | 2Th | 1 | 36 |
| 600 | 1Ti | 2 | 11, 3 |
| 600 | 2Ti | 2 | 11, 100 |
| 600 | Tit | 1 | 50 |
| 800 | Eph | 3 | 65, 54, 12 |
| 800 | Col | 1 | 50 |
| 800 | 2Th | 1 | 23 |
| 800 | 1Ti | 1 | 12 |
| 800 | 2Ti | 1 | 0 |
| 1200 | Eph | 2 | 81, 31 |
| 1200 | Col | 1 | 44 |
| 1200 | 1Ti | 1 | 19 |
| 1200 | 2Ti | 1 | 100 |
| 1600 | Eph | 1 | 58 |

### Test B:  Paul versus other letter writers (two-group nearest centroid)

Every book (Paul's seven and the six non-Pauline letters) is held out in turn;  its blocks are assigned to whichever reference group, Paul or non-Paul, has the nearer centroid.  Correct = Paul blocks assigned to Paul, non-Pauline blocks assigned to non-Paul.

| Block length (words) | Paul blocks correct | Non-Pauline blocks correct | Balanced accuracy |
|---|---|---|---|
| 250 | 58 of 92 (63%) | 34 of 44 (77%) | 70% |
| 400 | 38 of 57 (67%) | 22 of 28 (79%) | 73% |
| 600 | 27 of 36 (75%) | 10 of 16 (63%) | 69% |
| 800 | 22 of 26 (85%) | 8 of 13 (62%) | 73% |
| 1200 | 12 of 16 (75%) | 2 of 7 (29%) | 52% |
| 1600 | 10 of 12 (83%) | 3 of 6 (50%) | 67% |