PL jt\UdZddlmZddlZddlZddlZddlmZmZm Z m Z m Z ddl m Z ddlmZejeZe rddlmZdaded <d)d ZGd dZeZd*dZd+dZd,dZd,dZd,dZd+dZd-dZ d.dZ!d-dZ"d/dZ#d0d!Z$d1d#Z%d2d&Z&Gd'd(e Z'dS)3uFirecrawl web search + extract — plugin form. Subclasses :class:`agent.web_search_provider.WebSearchProvider`. This is the largest provider migrated in this PR; it captures the full inline firecrawl implementation that previously lived in tools/web_tools.py: - :data:`Firecrawl` lazy proxy that defers the ~200ms SDK import to first use (re-exported by tools.web_tools for backward compat with existing tests that mock that name). - :func:`_get_firecrawl_client` with direct + managed-gateway dual mode, controlled by ``web.use_gateway`` config when both are configured. - :func:`check_firecrawl_api_key` re-exported (tests + tools_config setup hint depend on this name living in tools.web_tools). - :func:`_extract_web_search_results` / :func:`_extract_scrape_payload` response-shape normalizers that handle SDK / direct API / gateway response variants. - Per-URL extract loop with 60s timeout, redirect-aware SSRF re-check, website-policy gating, and format-aware content selection. Async note: the underlying SDK is sync. ``extract()`` is declared ``async def`` because it performs per-URL I/O that benefits from running in an executor; the implementation wraps each scrape in :func:`asyncio.to_thread` with :func:`asyncio.wait_for(timeout=60)` to guard against hung fetches. Config keys this provider responds to:: web: search_backend: "firecrawl" # explicit per-capability extract_backend: "firecrawl" # explicit per-capability backend: "firecrawl" # shared fallback (default) use_gateway: false # prefer managed gateway when both # direct + gateway credentials exist Env vars:: FIRECRAWL_API_KEY=... # direct cloud auth FIRECRAWL_API_URL=... # self-hosted Firecrawl FIRECRAWL_GATEWAY_URL=... # Nous tool-gateway (subscribers) TOOL_GATEWAY_DOMAIN=... # alternate gateway env TOOL_GATEWAY_SCHEME=... TOOL_GATEWAY_USER_TOKEN=... ) annotationsN)AnyDictListOptional TYPE_CHECKING)WebSearchProvider)check_website_access FirecrawlzOptional[type]_FIRECRAWL_CLS_CACHEreturntypectV ddlm}|ddn9#t$rYn-t$r!}tt |d}~wwxYwddlm}|atS)z)Import and cache ``firecrawl.Firecrawl``.Nr)ensurezsearch.firecrawlF)promptr )r tools.lazy_depsr ImportError Exceptionstr firecrawlr ) _lazy_ensureexc_clss B/home/kuhnn/.hermes/hermes-agent/plugins/web/firecrawl/provider.py_load_firecrawl_clsrLs# ( > > > > > > L+E : : : : :    D ( ( (c#hh'' ' (//////# s A AAAc.eZdZdZdZddZdd Zdd Zd S)_FirecrawlProxyzJCallable proxy that looks like ``firecrawl.Firecrawl`` but imports lazily.argsrkwargsrc*t|i|SN)r)selfr r!s r__call__z_FirecrawlProxy.__call__cs$"$$d5f555objboolc:t|tSr#) isinstancer)r$r's r__instancecheck__z!_FirecrawlProxy.__instancecheck__fs#244555r&rcdS)Nz rr$s r__repr__z_FirecrawlProxy.__repr__is11r&N)r rr!rrr)r'rrr(rr)__name__ __module__ __qualname____doc__ __slots__r%r+r.rr&rrr^s[TTI66666666222222r&rOptional[tuple]ctjdd}tjddd}|s|sdSi}|r||d<|r||d<|d|pd|pdffS) zHReturn explicit direct Firecrawl kwargs + cache key, or None when unset.FIRECRAWL_API_KEYFIRECRAWL_API_URL/Napi_keyapi_urldirect)osgetenvstriprstrip)r;r<r!s r_get_direct_firecrawl_configrBzsi+R006688Gi+R006688??DDG 7tF$#y$#y Hgow$? ??r&rc8ddlm}|dS)z,Return the configured Firecrawl gateway URL.rNr)tools.web_tools web_toolsbuild_vendor_gateway_url_wts r_get_firecrawl_gateway_urlrIs(!!!!!!  ' ' 4 44r&r(cJddlm}|d|jduS)aReturn True when gateway URL + Nous Subscriber token are available. Reads ``read_nous_access_token`` and ``resolve_managed_tool_gateway`` via :mod:`tools.web_tools` rather than direct imports, so unit tests that ``patch("tools.web_tools._read_nous_access_token", ...)`` see their patches honored. The names are re-exported on :mod:`tools.web_tools` for exactly this reason. rNr token_reader)rDrEresolve_managed_tool_gateway_read_nous_access_tokenrGs r_is_tool_gateway_readyrOsC"!!!!!  + +#"= ,    r&c"tduS)zBReturn True when direct Firecrawl config is explicitly configured.N)rBrr&r_has_direct_firecrawl_configrQs ' ) ) 55r&c:tp tS)zReturn True when Firecrawl backend (direct or gateway) is usable. Re-exported by :mod:`tools.web_tools` for backward compatibility with existing tests and the ``hermes tools`` setup flow. )rQrOrr&rcheck_firecrawl_api_keyrSs ( ) ) E-C-E-EEr&c@ddlm}|sdS dS)zAReturn optional managed-gateway guidance for Firecrawl help text.rNr8zc, or use the Nous Tool Gateway via your subscription (FIRECRAWL_GATEWAY_URL or TOOL_GATEWAY_DOMAIN))rDrEmanaged_nous_tools_enabledrGs r_firecrawl_backend_help_suffixrVs<!!!!!!  ) ) + +r 9r&Nonecbddlm}d}|r|dz }t|)z>Raise a clear error for unsupported web backend configuration.rNzWeb tools are not configured. Set FIRECRAWL_API_KEY for cloud Firecrawl or set FIRECRAWL_API_URL for a self-hosted Firecrawl instance.u With your Nous subscription you can also use the Tool Gateway — run `hermes tools` and select Nous Subscription as the web provider.)rDrErU ValueError)rHmessages r&_raise_web_backend_configuration_errorr[sT!!!!!! 0  %%''  S  W  r&rcddlm}t}||ds|\}}ne|d|j}|(t dt|j |j d}d|d |j f}t|d d}t|d d}|||kr|S|j d i||_ ||_|j S) u6Get or create the cached Firecrawl client. When ``web.use_gateway`` is set in config, the managed Tool Gateway is preferred even if direct Firecrawl credentials are present. Otherwise direct Firecrawl takes precedence when explicitly configured. Raises ValueError when neither path is usable. The cached client is stored on :mod:`tools.web_tools` (as ``_firecrawl_client`` and ``_firecrawl_client_config``) rather than on this plugin module so that unit tests that reset the cache via ``tools.web_tools._firecrawl_client = None`` keep working. Helper functions (``prefers_gateway``, ``resolve_managed_tool_gateway``, ``_read_nous_access_token``, ``Firecrawl``) are also looked up via :mod:`tools.web_tools` for the same reason — see :func:`_is_tool_gateway_ready`. rNwebrrKzTFirecrawl client initialization failed: missing direct config and tool-gateway auth.)r;r<z tool-gatewayr<_firecrawl_client_firecrawl_client_configr)rDrErBprefers_gatewayrMrNloggererrorr[nous_user_tokengateway_origingetattrr r^r_)rH direct_configr! client_configmanaged_gatewaycached cached_configs r_get_firecrawl_clientrks5$"!!!!!022M )<)z$_to_plain_object..&s0UUUTQ1<>H 1$((92E2EFFL $##(););E)B)BCC  N,^-?-? -J-JKK   xD%ghr&B&BCCC Ir& scrape_resultDict[str, Any]ct|}t|tsiS|d}t|tr|S|S)zINormalize Firecrawl scrape payload shape across SDK and gateway variants.r)rr*ryr)r result_plainnesteds r_extract_scrape_payloadrYsV#M22L lD ) )   f % %F&$ r&ceZdZdZeddZeddZddZddZdd Z dd Z dddZ ddZ d dZ d!dZdS)"FirecrawlWebSearchProviderz9Firecrawl search + extract provider with dual auth paths.rrcdS)Nrrr-s rnamezFirecrawlWebSearchProvider.namen{r&cdS)Nr rr-s r display_namez'FirecrawlWebSearchProvider.display_namerrr&r(ctS)zHReturn True when direct Firecrawl OR managed-gateway path is configured.)rSr-s r is_availablez'FirecrawlWebSearchProvider.is_availablevs&(((r&cdSNTrr-s rsupports_searchz*FirecrawlWebSearchProvider.supports_searchztr&cdSrrr-s rsupports_extractz+FirecrawlWebSearchProvider.supports_extract}rr&cdSrrr-s rsupports_crawlz)FirecrawlWebSearchProvider.supports_crawlrr&querylimitr{rcddlm}|rdddStd||t } |||}t |}tdt|d d |id S#t$r-}t d |dd |dcYd}~Sd}~wwxYw)uzExecute a Firecrawl search. Sync; matches the legacy ``_get_firecrawl_client().search(...)`` call directly. Normalizes the response across SDK/direct/gateway shapes via :func:`_extract_web_search_results`. Pre-flight errors (``ValueError`` from configuration check, ``ImportError`` from missing SDK) propagate to the dispatcher's top-level handler, which wraps them as ``tool_error(...)`` — matching the legacy ``{"error": "Error searching web: ..."}`` envelope. Only in-flight errors are caught and surfaced as ``{"success": False, "error": ...}``. ris_interruptedF Interrupted)successrbz!Firecrawl search: '%s' (limit=%d))rrz"Firecrawl: found %d search resultsTr])rrzFirecrawl search error: %szFirecrawl search failed: N) tools.interruptrrainforksearchrlenrwarning)r$rrrclientr web_resultsrs rrz!FirecrawlWebSearchProvider.searchs 322222 >   >$}== = 7FFF'(( R}}5}>>H5h??K KKN>N O O O#e[-ABB B R R R NN7 = = =$/P3/P/PQQ Q Q Q Q Q Q RsAB C  "CC C urls List[str]r!rrc Kddlm}|r d|DS|d}g}|dkrdg}n|dkrdg}nddg}g}|D]}|r||ddd 't |}|r`t d |d |d ||dd|d |d |d |ddd t d| tjtj tj ||dd{V} nK#tj $r9t d|||ddddY?wxYwt| } | di} | d} | d} t| t s?t#| dr| } nt#| dr| j} ni} | dd}| d|}t |}|rbt d|d |d |||dd|d |d |d |ddd|dks|| r| }n| p| pd}|||||| d #t($rO}t d!||||dddt-|d"Yd}~ d}~wwxYw|S)#aExtract content from one or more URLs via Firecrawl. Async; each URL is scraped in a background thread with a 60s timeout. After scraping, the final URL (post-redirect) is re-checked against website-access policy. Accepted kwargs (others ignored for forward compat): - ``format``: ``"markdown"`` or ``"html"``; default is both (request both, return markdown when available). Returns the legacy per-URL list-of-results shape. Per-URL failures (timeout, SSRF block, scrape error, policy block) become items with an ``error`` field rather than raising. rrcg|]}|ddd S)rr8urlrbtitler)ruus r z6FirecrawlWebSearchProvider.extract..s"RRRA CCRRRr&formatmarkdownhtmlrr8rz%Blocked web_extract for %s by rule %shostrulerZsourcerrr)rrcontentrbblocked_by_policyzFirecrawl scraping: %s)rformats<)timeoutNz!Firecrawl scrape timed out for %sucScrape timed out after 60s — page may be too large or unresponsive. Try browser_navigate instead.rrrrbmetadatarprqr sourceURLz0Blocked redirected web_extract for %s by rule %srrr raw_contentrbrrrrrrz"Firecrawl scrape failed for %s: %s)rrrrrb)rrrrr rarasynciowait_for to_threadrkscrape TimeoutErrorrrr*ryr}rprqrdebugr)r$rr!_is_interruptedrrrrblockedrscrape_payloadrcontent_markdown content_htmlr final_url final_blockedchosen_content scrape_errs rextractz"FirecrawlWebSearchProvider.extracts FEEEEE ?   SRRTRRR RH%% Z  !lGG v  hGG!6*G )+x x C   s]RPPQQQ+3//G  ;FOFO "!##%!(!3$+FO$+FO&-h&7..     \  4c:::*1*:)133: #$+ !# +++%%%%%%MM+   NN#FLLLNN#&%'')!Q    H "9!G!G)--j"==#1#5#5j#A#A -11&99 "(D11&x66&#+#6#6#8#8 :66&#+#4#% Wb11$LLc:: !5Y ? ?  KKJ%f-%f- NN#,%*')+-%29%=(5f(=(5f(=*7*A22     Z''FN?ON%5NN%1%K5E%KN(!'5$,     A3 SSS"!##%')!$Z  sF+LAE  L AFLFEL.L M AMM rc  K ddlm}|r d|ddddgiS|d}|dd }|rtd td |||d d gid}t jtjfd|i|d{V}g}t|drI|j r|j ng}tdt|ddt|n_t|trd|vr|dgpg}n-tdt!|jg} |D]} d} d} i} t| drU| }|d } |d} |di} nt| drt| d d} t| dd} t| di}t|dr|} nt|dr|j} npt|tr|} nXi} nUt| tr@| d } | d} | di} t| ts?t| dr| } nt| dr| j} ni} | d| dd}| dd}t)|}|rbtd|d|d| ||dd|d |d|d|d!d"d#e| p| pd}| ||||| d$d| iS#t,$r!}d|ddt/|dgicYd}~Sd}~wt0$r}d|ddd%|dgicYd}~Sd}~wt2$r2}td&|d|ddd'|dgicYd}~Sd}~wwxYw)(uCrawl a seed URL via Firecrawl's ``/crawl`` endpoint. Sync SDK call wrapped in ``asyncio.to_thread`` because the dispatcher in :func:`tools.web_tools.web_crawl_tool` is async and runs LLM post-processing on the response. The dispatcher gates the seed URL against SSRF + website-access policy before calling us; this method re-checks every crawled page's URL against the policy after the crawl returns to catch redirected pages that map to a blocked host. Accepted kwargs (others ignored for forward compat): - ``instructions``: str — logged then dropped. Firecrawl's /crawl endpoint does NOT accept natural-language instructions (that's an /extract feature), so we record the value for debugging and proceed without it. Tavily's crawl IS instruction-aware; this divergence is documented in both plugins' docstrings. - ``limit``: int — max pages to crawl (default 20). - ``depth``: str — accepted for API parity with Tavily; ignored by Firecrawl's crawl endpoint. Returns ``{"results": [...]}`` matching the shape that :func:`tools.web_tools.web_crawl_tool`'s shared LLM-summarization path expects. Per-page failures (policy block on redirected URL, bad response shape) are included as items with an ``error`` field rather than raising. rrrr8rr instructionsrzUFirecrawl crawl: 'instructions' parameter ignored (not supported by Firecrawl /crawl)zFirecrawl crawl: %s (limit=%d)rr)rscrape_optionsrNrz$Firecrawl crawl status: %s, %d pagesstatusunknownz*Firecrawl crawl: unexpected result type %rrprrrqrz Unknown URLrz"Blocked crawled page %s by rule %srrrZrrrrzFirecrawl SDK not installed: zFirecrawl crawl error: %szFirecrawl crawl failed: )rrrrarrrrkcrawlr}rrerr*ryrrr0rprqr rrYrrr)r$rr!rrr crawl_params crawl_result data_listpagesrrrr item_dict metadata_objpage_urlr page_blockedrrs rrz FirecrawlWebSearchProvider.crawlBs4X  6 6 6 6 6 6~ g!C"Vc$d$d#eff!::n55LJJw++E  : KK8#u E E E#,zl";L")!2%''-"""""L$&I|V,, 1=1BJL--  :L(I>> NN L$// Fl4J4J(,,VR88>B @&&/ +-E!K K #' # "4..8 $ 1 1I'0}}Z'@'@$#,==#8#8L(}}Z<#>L#*4R#@#@L|\::&#/#:#:#<#< z::&#/#8#L$77&#/#%d++8'+xx ';';$#'88F#3#3L#xx B77H"(D11&x66&#+#6#6#8#8 :66&#+#4#%#<<e]!C!C! Wb11 4H== KK<$V,$V, LL#+%*')+-%1)%<(4V(<(4V(<*6x*@22     *@l@b '!&#*'.$, u% % ^ ^ ^bRRUVYRZRZ [ [\] ] ] ] ] ] ]   "!##%!F!F!F            NN6 < < <"!##%!AC!A!A         sGO>OO>> RP$R$ R1 Q=R R'Q=7R=RcdddddddgdS) Nr upaid · optional gatewayzQFull search + extract + crawl; supports direct API and Nous tool-gateway routing.r7z2Firecrawl API key (or leave blank for self-hosted)z'https://docs.firecrawl.dev/introduction)keyrr)rbadgetagenv_varsrr-s rget_setup_schemaz+FirecrawlWebSearchProvider.get_setup_schemas6/- /RD   r&Nr/rr()r)rrrr{rr)rrr!rrr)rrr!rrr)rr)r0r1r2r3propertyrrrrrrrrrrrr&rrrks CC XX))))RRRRR@]]]]~rrrrh      r&r)rr)rr5r/r)rrW)rr)rnrrr)rrrr)rrrr)rrrr)(r3 __future__rrloggingr>typingrrrrragent.web_search_providerr tools.website_policyr getLoggerr0rarr FirecrawlSDKr __annotations__rrrBrIrOrQrSrVr[rkrmrrrrrrr&rrsi+++Z#""""" ;;;;;;;;;;;;;;777777555555  8 $ $4333333'+++++    $ 2 2 2 2 2 2 2 2 O   @@@@"5555     6666 FFFF    "5!5!5!5!p ( ( ( (".    >    $Z Z Z Z Z !2Z Z Z Z Z r&