You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Provide an exact internal_method for da.percentile #12686
Dask has no exact percentile for a chunked 1-D array, and da.median and da.quantile raise for axis=None on chunked arrays. da.percentile is approximate by design (#3099). There are existing algorithms that compute these statistics exactly out of core. In particular, multi-pass radix selection, which is what CUDA torch.kthvalue and the radix variants of RAFT's select_k use (Alabi et al., ACM JEA 2012), does it in at most four passes over the input for 64-bit dtypes, two for 32-bit and one for 16-bit, with one 65,536-bin int64 histogram per tracked rank per task. Each pass is a map over the chunks plus a sum, and the next pass depends only on the summed histogram, so chunks are re-read rather than retained. It should be possible to match np.percentile(a.compute(), q, method=method), with NaN propagating as in NumPy.
I'd propose this as an opt-in internal_method="exact" for da.percentile, leaving the default alone (#6566 is about changing it). It would be 1-D only like the existing methods; quantile(axis=None) on N-D input is the same computation on raveled chunks and could follow. If there's interest, I would be happy to put together a PR.
I'd be interested in putting together the exact internal_method="exact" for da.percentile. My plan: implement multi-pass radix selection (Alabi et al. 2012) as a graph of per-chunk histogram maps plus a sum per pass, with one 65,536-bin histogram per tracked rank per task, iterating until the pass bounds converge — at most four passes for 64-bit dtypes, two for 32-bit, one for 16-bit. It would be opt-in (internal_method="exact"), 1-D only, matching np.percentile(a.compute(), q, method=method) including NaN propagation, leaving the default method alone. Before opening the PR I'd want to confirm the approach with you — particularly where the per-rank state should live in the task graph and whether you'd prefer the 16/32-bit special-cased pass counts or a uniform loop.
Activity
AtharvRG commentedon Oct 31, 2026
I'd be interested in putting together the exact
internal_method="exact"forda.percentile. My plan: implement multi-pass radix selection (Alabi et al. 2012) as a graph of per-chunk histogram maps plus a sum per pass, with one 65,536-bin histogram per tracked rank per task, iterating until the pass bounds converge — at most four passes for 64-bit dtypes, two for 32-bit, one for 16-bit. It would be opt-in (internal_method="exact"), 1-D only, matchingnp.percentile(a.compute(), q, method=method)including NaN propagation, leaving the default method alone. Before opening the PR I'd want to confirm the approach with you — particularly where the per-rank state should live in the task graph and whether you'd prefer the 16/32-bit special-cased pass counts or a uniform loop.