PL jDdZddlmZddlZddlZddlZddlmZeje Z ej dZ dd Z dd Zd dZd!dZd"d#dZddZd dZd$dZd dZddZgdZdS)%atMessage and tool-payload sanitization helpers. Pure functions extracted from ``run_agent.py`` so the AIAgent module can stay focused on the conversation loop. These walk OpenAI-format message lists and structured payloads, repairing or stripping problematic characters that would otherwise crash ``json.dumps`` inside the OpenAI SDK or be rejected by upstream APIs. All helpers are stateless and side-effect-free except for in-place mutation of their input (where documented). Backward-compatible re-exports from ``run_agent`` remain in place so existing imports ``from run_agent import _sanitize_surrogates`` keep working. ) annotationsN)Anyz[\ud800-\udfff]textstrreturncpt|rtd|S|S)zReplace lone surrogate code points with U+FFFD (replacement character). Surrogates are invalid in UTF-8 and will crash ``json.dumps()`` inside the OpenAI SDK. This is a fast no-op when the text contains no surrogates. �) _SURROGATE_REsearchsubrs >/home/kuhnn/.hermes/hermes-agent/agent/message_sanitization.py_sanitize_surrogatesrs5 D!!1  4000 Kpayloadrboolc0dfd|S)uuReplace surrogate code points in nested dict/list payloads in-place. Mirror of ``_sanitize_structure_non_ascii`` but for surrogate recovery. Used to scrub nested structured fields (e.g. ``reasoning_details`` — an array of dicts with ``summary``/``text`` strings) that flat per-field checks don't reach. Returns True if any surrogates were replaced. Fct|tr|D]|\}}t|tr;t|r td|||<dUt|ttfr |}dSt|trt|D]~\}}t|tr;t|r td|||<dUt|ttfr |}dSdS)Nr T) isinstancedictitemsrr r r list enumerate)nodekeyvalueidx_walkfounds rrz-_sanitize_structure_surrogates.._walk4sQ dD ! ! !"jjll ! ! UeS))!$++E22%$1$5$5h$F$FS $d|44!E%LLL  ! !d # # !'oo ! ! UeS))!$++E22%$1$5$5h$F$FS $d|44!E%LLL ! ! ! !rrrrs @@r_sanitize_structure_surrogatesr"*s= E!!!!!!& E'NNN Lrmessagesrcpd}|D]}t|ts|d}t|tr;t|r!td||d<d}nt|tr~|D]{}t|trd|d}t|tr:t|r td||d<d}||d}t|tr:t|r td||d<d}|d}t|trs|D]o}t|ts|d} t| tr:t| r td| |d<d}|d } t| tr| d} t| tr:t| r td| | d<d}| d } t| tr:t| r td| | d <d}q|D]\} }| d vr t|tr;t|r td||| <d}Zt|ttfrt|rd}|S) aSanitize surrogate characters from all string content in a messages list. Walks message dicts in-place. Returns True if any surrogates were found and replaced, False otherwise. Covers content/text, name, tool call metadata/arguments, AND any additional string or nested structured fields (``reasoning``, ``reasoning_content``, ``reasoning_details``, etc.) so retries don't fail on a non-content field. Byte-level reasoning models (xiaomi/mimo, kimi, glm) can emit lone surrogates in reasoning output that flow through to ``api_messages["reasoning_content"]`` on the next turn and crash json.dumps inside the OpenAI SDK. Fcontentr Trname tool_callsidfunction arguments>r&roler%r') rrgetrr r r rrr")r#rmsgr%partrr&r'tctc_idfnfn_namefn_argsrrs r_sanitize_messages_surrogatesr4Ksw E3!3!#t$$  '')$$ gs # # % (<(Repaired unescaped control chars in tool_call arguments for %sz ,\s*([}\]])z\1{}[]r2Nu8Repaired malformed tool_call arguments for %s: %s → %sPuARepaired control-char-laced tool_call arguments for %s: %s → %suPUnrepairable tool_call arguments for %s — replaced with empty object (was: %s))rrstriploggerwarningjsonloadsdumpsJSONDecodeError TypeError ValueErrorrer countrangeendswithrG) rIrJ raw_strippedparsed reserialisedfixed open_curly open_bracket_escapeds r_repair_tool_call_argumentsrms(2(C'@'@H8>>###bL CYOOOtvI9UUUt  L777z&Z@@@ < ' ' NNP     )Z 8      E F>5% 0 0ES!!EKK$4$44J;;s##ekk#&6&66LA~~ z!!a |## 2YY    Ju    E#   ~~c"" u{{3'7'7%++c:J:J'J'Jcrc $$ S)9)9EKK> e   Jw    NNS<,gcrcl   N   )Z 8      NN /<$ 4sL/AB>>CC<FB'H?>H?AJJJAK55LLcV|dddS)zRemove non-ASCII characters, replacing with closest ASCII equivalent or removing. Used as a last resort when the system encoding is ASCII and can't handle any non-ASCII characters (e.g. LANG=C on Chromebooks). asciiignore)errors)encodedecoder s r_strip_non_asciirts( ;;wx; 0 0 7 7 @ @@rcRd}|D] }t|ts|d}t|trt |}||kr||d<d}nut|t r`|D]]}t|trF|d}t|trt |}||kr||d<d}^|d}t|trt |}||kr||d<d}|d}t|t r|D]} t| trq| di} t| trF| d} t| trt | }|| kr|| d<d}|D];\} } | d vr t| trt | }|| kr||| <d}<"|S) aStrip non-ASCII characters from all string content in a messages list. This is a last-resort recovery for systems with ASCII-only encoding (LANG=C, Chromebooks, minimal containers). Returns True if any non-ASCII content was found and sanitized. Fr%Trr&r'r)r*>r&r+r%r')rrr,rrtrr)r#rr-r% sanitizedr.rr&r'r/r1r3rrs r_sanitize_messages_non_asciirw#s] E/!/!#t$$  '')$$ gs # # )(11IG##!*I  & & ) ) )dD)))88F++D!$,,)$4T$:$: $,,+4DL$(Ewwv dC  (..ID  'F WW\** j$ ' ' - - -b$''- B//B!"d++-"$&&"5"5%gs33-(8(A(AI(G332;;(,))++ ! !JC???%%% !,U33 %%(CH E ! Lrtoolsc t|S)z7Strip non-ASCII characters from tool payloads in-place.)_sanitize_structure_non_ascii)rxs r_sanitize_tools_non_asciir{^s ( / //rc2d}g}t|D]\}}t|ts|d}t|tsFg}|D]F}t|tr|ddvrd}1||Gt |t |kr<|r||d<|ddkrd|d<||t|D]}||=|S) uBRemove image_url content parts from all messages in-place. Called when a server signals it does not support images (e.g. "Only 'text' content type is supported."). Mutates messages so the next API call sends text only. Preserves message alternation invariants: * ``tool``-role messages whose content was entirely images are replaced with a plaintext placeholder, NOT deleted — deleting them would leave the paired ``tool_call_id`` on the prior assistant message unmatched, which providers reject with HTTP 400. * Non-tool messages whose content becomes empty are dropped. In practice this only hits synthetic image-only user messages appended for attachment delivery; real user turns always include text. Returns True if any image parts were removed. Fr%type>image image_url input_imageTr+toolu:[image content removed — server does not support images])rrrr,rr?r>reversed)r#r to_deleterDr-r% new_partsr.s r_strip_images_from_messagesrcsH$ EIH%%$$3#t$$  '')$$'4((    ' 'D$%% '$((6*:*:>c*c*c  &&&& y>>CLL ( ( $!*IF**"^I  ### i  QKK Lrc0dfd|S)zCStrip non-ASCII characters from nested dict/list payloads in-place.Fc&t|tru|D]^\}}t|trt |}||kr|||<d7t|tt fr |_dSt|t rnt |D]`\}}t|trt |}||kr|||<d7t|tt fr |_dSdS)NT)rrrrrtrr)rrrrvrrrs rrz,_sanitize_structure_non_ascii.._walks5 dD ! ! !"jjll ! ! UeS))! 0 7 7I E))$-S $d|44!E%LLL ! !d # # !'oo ! ! UeS))! 0 7 7I E))$-S $d|44!E%LLL ! ! ! !rr r!s @@rrzrzs; E!!!!!!* E'NNN Lr) r rr"r4rGrmrtrwr{rrz)rrrr)rrrr)r#rrr)r5rrr)rH)rIrrJrrr)rxrrr)__doc__ __future__rr[loggingratypingr getLogger__name__rYcompiler rr"r4rGrmrtrwr{rrz__all__r rrrsk  #"""""    8 $ $  -.. BAAAAH''''T^^^^^BAAAA8888v0000 ----`:   r