Fighting Overfit with PreciseDist

Brian Muchmore

2026-07-14

“Everybody has their taste in noises as well as in other matters; and sounds are quite innoxious, or most distressing, by their sort rather than their quantity.”

-Jane Austen, Persuasion, p 160

A clustering algorithm always finds clusters

That is the problem. Run k-means and you will get k groups whether or not the data has any. In unsupervised analysis there is no held-out label to check against, so the danger is believing structure that is really just noise. We need a way to ask: is there anything here worth finding, or are we overfitting?

PreciseDist’s answer is triangulation. Structure you can trust is structure that survives more than one arbitrary choice. One of the easiest ways to fool yourself is to let a single distance metric dominate the story. This vignette asks a narrower but useful question: if you choose a small set of plausible metrics and fuse them, does the fused structure depend heavily on any one of them?

set.seed(1)
idx <- c(1:5, 61:65, 121:125)
cell_cycle <- as.matrix(data_cell_cycle[idx, -1])[, 1:50]
storage.mode(cell_cycle) <- "double"
rownames(cell_cycle) <- paste0(substr(data_cell_cycle$Cell_cycle[idx], 1, 2), "_", idx)

Build the candidate matrix set

Start with several candidate metrics and coerce them into a common distance space. This is the object PreciseDist owns: a typed collection of relatedness matrices.

d_candidates <- precise_dist(
  cell_cycle,
  dists = c("euclidean", "manhattan", "canberra", "cosine"),
  verbose = FALSE
)
d_candidates <- precise_transform(d_candidates, to = "distance")
d_candidates[, c("distance", "type")]
#> # A tibble: 4 × 2
#>   distance  type    
#>   <chr>     <chr>   
#> 1 euclidean distance
#> 2 manhattan distance
#> 3 canberra  distance
#> 4 cosine    distance

Drop one metric at a time

precise_stability() fuses the full set once, then repeats the fusion while leaving each metric out once. It does not touch the original data, and it does not resample matrix cells. The unit of resampling is the whole matrix row from the typed candidate set.

stability <- precise_stability(
  d_candidates,
  scheme = "loo",
  fusion = "mean",
  verbose = FALSE
)
stability[, c("distance", "role", "fusion")]
#> # A tibble: 5 × 3
#>   distance           role      fusion
#>   <chr>              <chr>     <chr> 
#> 1 reference          reference mean  
#> 2 loo_drop_euclidean loo_drop  mean  
#> 3 loo_drop_manhattan loo_drop  mean  
#> 4 loo_drop_canberra  loo_drop  mean  
#> 5 loo_drop_cosine    loo_drop  mean

Compare the replicate consensuses

Now compare the full-set reference consensus with each leave-one-out consensus. When a drop-one row remains close to the reference, that metric was not carrying the fused structure by itself. When a drop-one row moves far away, the omitted metric had real leverage.

agreement <- precise_correlations(stability, permutations = 99, verbose = FALSE)
drop_rows <- grep("^loo_drop_", rownames(agreement$statistic), value = TRUE)
round(agreement$statistic["reference", drop_rows], 3)
#> loo_drop_euclidean loo_drop_manhattan  loo_drop_canberra    loo_drop_cosine 
#>              0.999              0.844              0.951              1.000

See the stability structure

The agreement view shows the same object visually. The reference row and the drop-one rows should form a tight block when the candidate metric set supports a coherent consensus.

precise_viz(
  stability,
  views = "agreement",
  agreement = agreement,
  verbose = FALSE
)$panel[[1]]

This does not prove that the original measurements contain a true structure. It does answer a practical metric-choice question: whether the structure you plan to carry downstream is stable across the candidate relatedness matrices you selected.