html pattern matching library and high-level interface concurrent requests lib for webscraping.
Scrappy is meant to be a fully expressive library for all aspects of webscraping. In this sense it is meant to be as undetectable as using Selenium but with a design specifically for webscraping (where Selenium was never intended for webscraping nor do it's maintainers seek to provide features that allow for more expressive scraping if it would not help testing). The Elem.* modules provide a wide range of expressive pattern matching functions while adding no more complexity in needing to learn this library over parsing.
In addition to expressive patterns to fit your specific patterns you hope to scrape, scrappy provides helper functions for complex control flows such as running multiple different parser-scrapers on respective sites based on a rotating ConcurrentStream when users have many target sites that they can rotate working on to not overload a given site and thus avoid detection
For simpler control flows such as scraping a large number of pages on a single site, scrappy currently provides functions like getHtml, which not only is a super simple interface to http requests, but a persistent function that gurantees retrieval of the HTML document to be scraped
This package is labelled as uncurated, and so suggestions are very much welcome and this package is expected to grow based on feedback. The primary focus will be on running Javascript to allow for greater access to information on sites