LangChain Integration

Available

The official LangChain integration with Crawleo - a privacy-first web search and crawler API. Build powerful AI applications with real-time web intelligence.

PyPI Package GitHub Repository Get API Key

CrawleoSearch

Real-time web search with privacy-first approach and multiple output formats.

CrawleoCrawler

Extract content from URLs with HTML/markdown support (1-20 URLs per request).

Full Async Support

Async-compatible for high-performance applications and concurrent requests.

LangChain Compatible

Full tool calling compatible - works seamlessly with LangChain agents.

Privacy First

Zero data retention - your searches are never stored or tracked.

Multiple Formats

Get results as Markdown, HTML, enhanced HTML, or plain text.

Quick Start

Installation

pip install -U langchain-crawleo

Set API Key

import getpass
import os

if not os.environ.get("CRAWLEO_API_KEY"):
    os.environ["CRAWLEO_API_KEY"] = getpass.getpass("Crawleo API key:\n")

# Or set it in your shell:
# export CRAWLEO_API_KEY="your-api-key"

CrawleoSearch - Basic Usage

from langchain_crawleo import CrawleoSearch

# Initialize the search tool
search_tool = CrawleoSearch(
    max_pages=1,
    cc="US",
    setLang="en",
    markdown=True,
)

# Perform a search
result = search_tool.invoke({"query": "What is the future of AI?"})
print(result)

CrawleoSearch - All Parameters

from langchain_crawleo import CrawleoSearch

tool = CrawleoSearch(
    max_pages=1,           # Max result pages (each costs 1 credit)
    setLang="en",          # Language code: "en", "es", "fr", etc.
    cc="US",               # Country code: "US", "GB", "DE", etc.
    geolocation="random",  # Geo: random, pl, gb, jp, de, fr, es, us
    device="desktop",      # Device: "desktop", "mobile", "tablet"
    enhanced_html=True,    # Return AI-enhanced, cleaned HTML
    raw_html=False,        # Return original, unprocessed HTML
    page_text=False,       # Return extracted plain text
    markdown=True,         # Return content in Markdown format
)

# Override parameters during invocation
result = tool.invoke({
    "query": "local news",
    "cc": "DE"  # Overrides the "US" setting above
})

CrawleoCrawler - Extract Content from URLs

from langchain_crawleo import CrawleoCrawler

# Initialize the crawler tool
crawler_tool = CrawleoCrawler(
    markdown=True,
    raw_html=False,
)

# Crawl a single URL
result = crawler_tool.invoke({"urls": ["https://crawleo.dev"]})
print(result)

# Crawl multiple URLs (1-20 URLs supported)
result = crawler_tool.invoke({
    "urls": [
        "https://example.com",
        "https://another-site.com"
    ]
})

With LangChain Agent

from langchain_crawleo import CrawleoSearch, CrawleoCrawler
from langchain.agents import initialize_agent, AgentType
from langchain_openai import ChatOpenAI

# Initialize tools
search_tool = CrawleoSearch(max_pages=1, cc="US", markdown=True)
crawler_tool = CrawleoCrawler(markdown=True)

llm = ChatOpenAI(model="gpt-4", temperature=0)

# Create agent with Crawleo tools
agent = initialize_agent(
    tools=[search_tool, crawler_tool],
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Run the agent
response = agent.run("Search for SpaceX news and summarize the top result")
print(response)

CrawleoSearch Parameters

Parameter	Type	Required	Description
`query`	str	Yes	A natural language search query
`max_pages`	int	No	Max result pages to crawl. Each page costs 1 credit. Default: 1
`setLang`	str	No	Language code for search interface (e.g., "en", "es", "fr"). Default: "en"
`cc`	str	No	Country code for search results (e.g., "US", "GB", "DE")
`geolocation`	str	No	Geo location: random, pl, gb, jp, de, fr, es, us. Default: "random"
`device`	str	No	Device simulation: "desktop", "mobile", "tablet". Default: "desktop"
`enhanced_html`	bool	No	Return AI-enhanced, cleaned HTML. Default: True
`raw_html`	bool	No	Return original, unprocessed HTML. Default: False
`page_text`	bool	No	Return extracted plain text. Default: False
`markdown`	bool	No	Return content in Markdown format. Default: True

CrawleoCrawler Parameters

Parameter	Type	Required	Description
`urls`	list[str]	Yes	A list of URLs to crawl (1-20 URLs)
`raw_html`	bool	No	Whether to return raw HTML content. Default: False
`markdown`	bool	No	Whether to return content in markdown format. Default: False

Response Format

Search API Response Structure

{
  "status": "success",
  "data": {
    "query": "What is the future of AI?",
    "pages_fetched": 1,
    "time_used": 1.51,
    "pages": {
      "1": {
        "total_results": "About 523,000 results",
        "search_results": [
          {
            "title": "The Future of AI: Trends and Predictions",
            "link": "https://example.com/ai-future",
            "date": "Mar 12, 2024",
            "snippet": "Artificial Intelligence is rapidly evolving...",
            "domain": "example.com"
          }
        ],
        "page_content": {
          "page_ai_enhanced_html": "...",
          "page_text_markdown": "..."
        }
      }
    },
    "credits_used": 1
  }
}

Ready to Get Started?

Get your API key from your Crawleo dashboard and start building AI applications with real-time web intelligence.

Get API Key View on GitHub

Quick Start

Installation

pip install -U langchain-crawleo

Set API Key

import getpass
import os

if not os.environ.get("CRAWLEO_API_KEY"):
    os.environ["CRAWLEO_API_KEY"] = getpass.getpass("Crawleo API key:\n")

# Or set it in your shell:
# export CRAWLEO_API_KEY="your-api-key"

CrawleoSearch - Basic Usage

from langchain_crawleo import CrawleoSearch

# Initialize the search tool
search_tool = CrawleoSearch(
    max_pages=1,
    cc="US",
    setLang="en",
    markdown=True,
)

# Perform a search
result = search_tool.invoke({"query": "What is the future of AI?"})
print(result)

CrawleoSearch - All Parameters

from langchain_crawleo import CrawleoSearch

tool = CrawleoSearch(
    max_pages=1,           # Max result pages (each costs 1 credit)
    setLang="en",          # Language code: "en", "es", "fr", etc.
    cc="US",               # Country code: "US", "GB", "DE", etc.
    geolocation="random",  # Geo: random, pl, gb, jp, de, fr, es, us
    device="desktop",      # Device: "desktop", "mobile", "tablet"
    enhanced_html=True,    # Return AI-enhanced, cleaned HTML
    raw_html=False,        # Return original, unprocessed HTML
    page_text=False,       # Return extracted plain text
    markdown=True,         # Return content in Markdown format
)

# Override parameters during invocation
result = tool.invoke({
    "query": "local news",
    "cc": "DE"  # Overrides the "US" setting above
})

CrawleoCrawler - Extract Content from URLs

from langchain_crawleo import CrawleoCrawler

# Initialize the crawler tool
crawler_tool = CrawleoCrawler(
    markdown=True,
    raw_html=False,
)

# Crawl a single URL
result = crawler_tool.invoke({"urls": ["https://crawleo.dev"]})
print(result)

# Crawl multiple URLs (1-20 URLs supported)
result = crawler_tool.invoke({
    "urls": [
        "https://example.com",
        "https://another-site.com"
    ]
})

With LangChain Agent

from langchain_crawleo import CrawleoSearch, CrawleoCrawler
from langchain.agents import initialize_agent, AgentType
from langchain_openai import ChatOpenAI

# Initialize tools
search_tool = CrawleoSearch(max_pages=1, cc="US", markdown=True)
crawler_tool = CrawleoCrawler(markdown=True)

llm = ChatOpenAI(model="gpt-4", temperature=0)

# Create agent with Crawleo tools
agent = initialize_agent(
    tools=[search_tool, crawler_tool],
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Run the agent
response = agent.run("Search for SpaceX news and summarize the top result")
print(response)

CrawleoSearch Parameters

Parameter	Type	Required	Description
`query`	str	Yes	A natural language search query
`max_pages`	int	No	Max result pages to crawl. Each page costs 1 credit. Default: 1
`setLang`	str	No	Language code for search interface (e.g., "en", "es", "fr"). Default: "en"
`cc`	str	No	Country code for search results (e.g., "US", "GB", "DE")
`geolocation`	str	No	Geo location: random, pl, gb, jp, de, fr, es, us. Default: "random"
`device`	str	No	Device simulation: "desktop", "mobile", "tablet". Default: "desktop"
`enhanced_html`	bool	No	Return AI-enhanced, cleaned HTML. Default: True
`raw_html`	bool	No	Return original, unprocessed HTML. Default: False
`page_text`	bool	No	Return extracted plain text. Default: False
`markdown`	bool	No	Return content in Markdown format. Default: True

Parameter

Type

Required

Description

urls

list[str]

Yes

A list of URLs to crawl (1-20 URLs)

raw_html

bool

Whether to return raw HTML content. Default: False

markdown

bool

Whether to return content in markdown format. Default: False

Response Format

Search API Response Structure

{
  "status": "success",
  "data": {
    "query": "What is the future of AI?",
    "pages_fetched": 1,
    "time_used": 1.51,
    "pages": {
      "1": {
        "total_results": "About 523,000 results",
        "search_results": [
          {
            "title": "The Future of AI: Trends and Predictions",
            "link": "https://example.com/ai-future",
            "date": "Mar 12, 2024",
            "snippet": "Artificial Intelligence is rapidly evolving...",
            "domain": "example.com"
          }
        ],
        "page_content": {
          "page_ai_enhanced_html": "...",
          "page_text_markdown": "..."
        }
      }
    },
    "credits_used": 1
  }
}

LangChain Integration

CrawleoSearch

CrawleoCrawler

Full Async Support

LangChain Compatible

Privacy First

Multiple Formats

Quick Start

CrawleoSearch Parameters

CrawleoCrawler Parameters

Response Format

Ready to Get Started?

Legal

Connect

LangChain Integration

CrawleoSearch

CrawleoCrawler

Full Async Support

LangChain Compatible

Privacy First

Multiple Formats

Quick Start

CrawleoSearch Parameters

CrawleoCrawler Parameters

Response Format

Ready to Get Started?

Legal

Connect