Repository navigation
fix: preserve Unicode RawJSON values with ensure_ascii=True - #258
Conversation
|
I should focus on this, but one thing I wonder about is why we need to reimplement that encoding? Wouldn't it be more sensible to use |
No, forget that, it would be clearly wrong. |
|
@lelit , Thanks for taking a look. As you noted in your follow-up, The problem is that I chose this to preserve the JSON value while honoring the ASCII-output setting, without parsing and reserializing the fragment. The compatibility tradeoff is that non-ASCII spelling gets escaped under the default rather than remaining strictly verbatim. I'm happy to follow your preference on that. RawJSON still does not validate arbitrary raw input. |
|
Thank you! |
Preserve Unicode RawJSON values when ensure_ascii=True.
RawValue transcodes Unicode code points to an ASCII target without JSON escaping. Pass the ensure_ascii setting through the recursive encoder and escape non-ASCII RawJSON characters before that call, including UTF-16 surrogate pairs for astral characters. Existing ASCII raw spelling/whitespace and ensure_ascii=False literal behavior remain unchanged. Normal string encoding is unchanged. The shared stream-error check is retained.
Docs clarify that ensure_ascii=True escapes non-ASCII characters in RawJSON. This follows the encoder's ASCII-output guarantee rather than preserving corrupt output under a literal interpretation of 'verbatim'. RawJSON still accepts unvalidated content; this is not a general JSON validator or a fix for embedded-NUL input.
Validation on Linux CPython 3.14.8 with the native extension rebuilt from each source state:
Tests cover dumps/Encoder/text+binary dump, compact/pretty output, ASCII flag on/off, BMP/astral Unicode, unchanged raw whitespace, and stream exceptions. No speedup, hosted CI, sanitizer, debug-Python or cross-platform result is claimed.
Related but distinct: #169 discusses RawJSON performance. No competing corruption fix found in the current open PRs, targeted all-state searches, or relevant recent commit history.
Compatibility decision: the existing RawJSON docs promise verbatim insertion including whitespace, while dumps docs say all incoming non-ASCII characters are escaped under ensure_ascii=True. Current behavior satisfies neither for these inputs: it truncates Unicode or raises rather than inserting the original value. The proposal gives the global ASCII setting precedence for valid RawJSON string content. That deliberately changes literal non-ASCII spelling to escapes under the default; callers who require exact Unicode text must use ensure_ascii=False. An alternative would preserve Unicode verbatim regardless of ensure_ascii, but would weaken that documented global guarantee. ASCII content, existing escapes and whitespace are not normalized, and arbitrary raw content is still not validated. Maintainer preference on this compatibility tradeoff is welcome.
Closes #257