Awesome Data Analysis
Section: Tools · Advanced web crawling framework designed for AI and data extraction tasks.
FreshScore 80
Entry
Crawl4AI
Appears in 5 awesome lists
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
Open github.comunclecode/crawl4ai
Found in these lists
Awesome Generative AI
Section: Everything to Markdown to LLMs · Open-source LLM Friendly Web Crawler & Scraper
SlowScore 62
Awesome Open Source AI
Section: 5. Retrieval-Augmented Generation (RAG) & Knowledge · LLM-friendly web crawler that turns websites into clean Markdown for RAG and agentic workflows.
FreshScore 89
Awesome Python
Section: Web Scraping · An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.
FreshScore 94
awesome-python
Section: Other · Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
FreshScore 81
Often listed next to it
LlamaIndex
(MIT) provides modules for structured outputs at different levels of abstraction, including output parsers for text completion endpoints, Pydantic programs for mapping prompts to structured outputs using function calling or output parsing, and pre-defined Pydantic programs for specific output types.
In 14 listsDetails
Milvus
Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.
In 14 listsDetails
Haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search,…
In 13 listsDetails
browser-use
Minimal browser-automation agent harness with clean separation of tool registration, DOM state injection, action loop, and error recovery. Small codebase, clear structure — the best "minimal viable harness" reference for understanding core loop mechanics.
In 10 listsDetails
MarkItDown
Python tool for converting files and office documents to Markdown. Supports PDF, PowerPoint, Word, Excel, images, audio, HTML, and more with OCR and transcription capabilities. MIT licensed.
In 9 listsDetails
Selenium
Portable software testing framework for web applications. License: Apache 2 , .
In 9 listsDetails