A Parallel Future

“A happy man is too satisfied with the present to dwell too much on the future.”

Albert Einstein

TL;DR

Choose a backend with future::plan(), then tell PreciseDist to use it with parallel = TRUE.

library(future)

# precise_dist() and precise_correlations() use future.apply directly, so they
# need only the plan. precise_graph() in PreciseDist and precise_viz() in
# PreciseViz use foreach and require doFuture::registerDoFuture() when their
# parallel flags are TRUE.

future::plan(
  future::multisession,
  workers = min(4, future::availableCores())
)

options(future.globals.maxSize = 8 * 1024^3)

static_distances <- precise_dist(
  your_data,
  dists = "static",
  parallel = TRUE,
  max_time_per_metric = 60,
  verbose = TRUE
)

future::plan(future::sequential)

That is the PreciseDist parallel contract. The package does not decide how many cores to use, how much memory to export to workers, or which future backend is appropriate for the machine. Those are user decisions. PreciseDist only needs to know whether it should use the plan the user registered.

The Contract

precise_dist() can run many metrics against the same data. That is the right kind of work to parallelize: each metric can be evaluated independently, and the result is then collected into the usual typed matrix tibble.

The controls are deliberately split:

Future controls execution topology; PreciseDist controls workload planning, batching, accounting, persistence, and typed outputs.
parallel = TRUE tells PreciseDist to dispatch metric batches with future.apply::future_lapply() on the user’s plan. parallel = FALSE (the default) is a plain lapply() and touches no future machinery.
future::plan() chooses the backend and worker count.
options(future.globals.maxSize = ...) controls how much data future may export to workers.
max_time_per_metric is the per-metric time limit inside precise_dist(); max_time_total bounds the whole run.
seed makes stochastic metrics reproducible — identically — across sequential and parallel runs (see Reproducibility below).
Every requested metric is accounted for in a run ledger; failures never disappear silently (see precise_run_report()).

This matters because users run different jobs on different machines. A small example may need no parallelism at all. A broad metric sweep over a larger matrix may be worth running overnight. A cluster job may need a completely different backend from a laptop. The package should not pretend those are the same situation.

A Local Workstation Recipe

For a local workstation, start with multisession. It works across operating systems and keeps each worker in a separate R session.

library(future)

future::plan(
  future::multisession,
  workers = 4
)

options(future.globals.maxSize = 8 * 1024^3)

distance_collection <- precise_dist(
  your_data,
  dists = "static",
  parallel = TRUE,
  max_time_per_metric = 120,
  verbose = TRUE
)

future::plan(future::sequential)

The number 4 is only an example. It is often better to leave cores available for the operating system, the R session, and anything else running on the machine. If the job is memory-heavy, fewer workers may be faster and safer than more workers.

Resource Limits

PreciseDist exposes the limits that belong inside the package and leaves the backend limits to the backend.

CPU is controlled by the future plan:

future::plan(future::multisession, workers = 2)

Memory exported to workers is controlled by future:

options(future.globals.maxSize = 4 * 1024^3)

Per-metric and whole-run time are controlled by precise_dist():

precise_dist(
  your_data,
  dists = "static",
  parallel = TRUE,
  max_time_per_metric = 30,
  max_time_total = 1800,
  verbose = TRUE
)

A metric over max_time_per_metric is recorded in the run ledger as errored with reason timeout; once max_time_total is exceeded, remaining metrics are recorded as skipped with reason time_budget and the completed rows are returned. One honesty note: the per-metric limit uses setTimeLimit(), which cannot interrupt long compiled calls (randomForest, kernlab, philentropy) or blocking waits — treat it as best-effort, not a hard kill.

Memory for the results themselves is bounded by you, in GB:

# Refuses (before computing anything) if the estimated peak exceeds your cap.
precise_dist(your_data, dists = "all_dists", max_memory = 16)

# Inspect the plan and the estimate without running:
plan_ledger <- precise_dist(your_data, dists = "all_dists", plan_only = TRUE)
attr(plan_ledger, "est_result_gb")

Each dense result is exactly 8 * nrow(data)^2 bytes, so the estimate is cheap and honest about what it covers: the input/result footprint, not worker-side transients or the environments carried by custom dist_funcs. Without max_memory, a run estimated above getOption("PreciseDist.large_run_gb", 8) GB warns and proceeds — the package never refuses on its own.

These limits are intentionally explicit. A broad metric sweep is powerful because it is broad, but breadth needs boundaries. A metric that cannot finish within the user’s budget should not hold the entire run hostage.

Checkpointing and Recovery

The file= argument writes a versioned RDS envelope. Only the master process touches it, so it combines freely with parallel = TRUE. While the run is in flight the envelope holds the completed native matrices plus the run ledger (state "partial"), rewritten atomically after every batch; when precise_dist() returns normally it is rewritten once as state "complete", holding exactly the object the call returned.

cache_file <- "precise_dist_static.rds"

distance_collection <- precise_dist(
  your_data,
  dists = "static",
  file = cache_file,
  parallel = TRUE,
  max_time_per_metric = 120,
  verbose = TRUE
)

# Later — or after an interrupt/crash — read it back:
recovered <- precise_read(cache_file)
precise_run_report(recovered)

If you interrupt a run (Ctrl-C), it checkpoints what it has and then re-signals the interrupt: precise_read() on the file recovers the completed metrics, and the ledger says exactly which metrics finished, errored, were skipped, or were never attempted. Without file=, interrupted work is lost. If a parallel worker dies, the run checkpoints, marks the batch worker_crash, and stops with an error carrying the ledger. An existing path is refused unless you pass overwrite = TRUE.

Use the returned distance_collection for the typed downstream workflow. The live return value and the completed cache carry the same full typed contract used by precise_transform(), precise_fusion(), precise_graph(), and precise_viz().

Reproducibility

Stochastic metrics — the random_forest_* family and the kernel metrics (rbf_*, laplace_*, whose sigma estimation consumes randomness) — accept an explicit seed:

a <- precise_dist(your_data, dists = "rbf_1", seed = 1)
b <- precise_dist(your_data, dists = "rbf_1", seed = 1, parallel = TRUE)
identical(a$matrix[[1]], b$matrix[[1]]) # TRUE, on any plan

With an integer seed, every metric draws from its own RNG stream keyed by (seed, metric name): results are identical across sequential and parallel runs, and stable when other metrics are added to or removed from the request. The session RNG state is saved and restored around the call. Seeded results are reproducible for a given R version and platform endianness. With seed = NULL (the default) nothing is promised: sequential runs use the ordinary session stream, and parallel runs use future.seed = TRUE.

Progress

If the progressr package is installed, precise_dist() signals one progress step per metric. Display is the user’s choice, in the usual progressr way:

progressr::with_progress(
  distance_collection <- precise_dist(your_data, dists = "static",
                                      parallel = TRUE, verbose = FALSE)
)

progressr is inactive in non-interactive sessions unless you enable it explicitly. Independent of progressr, verbose = TRUE narrates run stages via message() (so suppressMessages() silences it), and the run ledger is always attached regardless of either setting.

Backend Patterns

Sequential execution is still useful. It is the simplest debugging mode and the right choice for small jobs.

library(future)

future::plan(future::sequential)

On Unix-like systems, multicore can be useful, but it is not available on all platforms. Use it only when it fits the machine and session.

library(future)

future::plan(future::multicore, workers = 4)

For an explicit cluster object, create the cluster yourself, pass it to future::plan(), and stop it when the work is done.

library(future)

cl <- parallel::makeCluster(4)
on.exit(parallel::stopCluster(cl), add = TRUE)

future::plan(future::cluster, workers = cl)

distance_collection <- precise_dist(
  your_data,
  dists = "static",
  parallel = TRUE,
  max_time_per_metric = 120,
  verbose = TRUE
)

future::plan(future::sequential)

Scheduler-backed clusters can also be handled through future-compatible backends, but those choices are site-specific. The important package-level point is the same: choose the plan, then call precise_dist(..., parallel = TRUE) or precise_correlations(..., parallel = TRUE). For graph and visualization parallelism, also register the plan with doFuture::registerDoFuture().

Practical Guidance

Start small. Run a few metrics sequentially, verify that the data and metric family are appropriate, then widen the sweep. plan_only = TRUE shows what a broad request resolves to — and what it costs — before anything runs. When a broad run is ready, choose workers, memory, and timeout values that reflect the machine and the analysis budget, and set file= so the run is recoverable.

Parallelism is not the point of PreciseDist. The point is breadth over relatedness representations. Parallelism is the practical tool that lets that breadth remain usable when the search becomes large.

Brian Muchmore

2026-07-14