⚡ Sangsakala Proxy 🏠 Home
Skip to content

Provide an exact internal_method for da.percentile #12686

Open

Description

@matthiasschabel

Dask has no exact percentile for a chunked 1-D array, and da.median and da.quantile raise for axis=None on chunked arrays. da.percentile is approximate by design (#3099). There are existing algorithms that compute these statistics exactly out of core. In particular, multi-pass radix selection, which is what CUDA torch.kthvalue and the radix variants of RAFT's select_k use (Alabi et al., ACM JEA 2012), does it in at most four passes over the input for 64-bit dtypes, two for 32-bit and one for 16-bit, with one 65,536-bin int64 histogram per tracked rank per task. Each pass is a map over the chunks plus a sum, and the next pass depends only on the summed histogram, so chunks are re-read rather than retained. It should be possible to match np.percentile(a.compute(), q, method=method), with NaN propagating as in NumPy.

I'd propose this as an opt-in internal_method="exact" for da.percentile, leaving the default alone (#6566 is about changing it). It would be 1-D only like the existing methods; quantile(axis=None) on N-D input is the same computation on raveled chunks and could follow. If there's interest, I would be happy to put together a PR.

Activity

  1. AtharvRG commented on Oct 31, 2026

    @AtharvRG

    I'd be interested in putting together the exact internal_method="exact" for da.percentile. My plan: implement multi-pass radix selection (Alabi et al. 2012) as a graph of per-chunk histogram maps plus a sum per pass, with one 65,536-bin histogram per tracked rank per task, iterating until the pass bounds converge — at most four passes for 64-bit dtypes, two for 32-bit, one for 16-bit. It would be opt-in (internal_method="exact"), 1-D only, matching np.percentile(a.compute(), q, method=method) including NaN propagation, leaving the default method alone. Before opening the PR I'd want to confirm the approach with you — particularly where the per-rank state should live in the task graph and whether you'd prefer the 16/32-bit special-cased pass counts or a uniform loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageNeeds a response from a contributor

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions