| 2011-09-11 |
Pablo Hoffman | added tests for SpiderState extension
|
commit | commitdiff | tree |
| 2011-09-09 |
Pablo Hoffman | added support for generating version from git revision...
|
commit | commitdiff | tree |
| 2011-09-04 |
Pablo Hoffman | scrapyd: updated schedule.json response format
|
commit | commitdiff | tree |
| 2011-09-04 |
Pablo Hoffman | added unittest for SpiderState extension
|
commit | commitdiff | tree |
| 2011-09-04 |
Pablo Hoffman | restored support for spider.DOWNLOAD_DELAY attribute...
|
commit | commitdiff | tree |
| 2011-09-02 |
Pablo Hoffman | replaced use of deprecated w3lib.url.urljoin_rfc by...
|
commit | commitdiff | tree |
| 2011-09-02 |
Pablo Hoffman | removed CONCURRENT_SPIDERS setting (use scrapyd maxproc...
|
commit | commitdiff | tree |
| 2011-09-02 |
Pablo Hoffman | added initial documentation about suspend and resume...
|
commit | commitdiff | tree |
| 2011-09-02 |
Pablo Hoffman | added SpiderState extension
|
commit | commitdiff | tree |
| 2011-09-02 |
Pablo Hoffman | fixed subtle bug in disk-based priority queues caused...
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | add setting to enable logging when unserializable requests...
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | PickleDiskQueue: use pickle protocol 2
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | minor fix to doc
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | no longer recommend using labmda's in the doc, as they...
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | remove redundant code
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | updated documentation and code to use -s instead of...
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | remove unneeded code to simplify
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | scrapy tool: added -s alias for --set option
|
commit | commitdiff | tree |
| 2011-09-01 |
Pablo Hoffman | persistent scheduler: use pickle (instead of marshal...
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | adapted test-scrapyd.sh to be compatible with older...
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | scrapyd: documented support for passing setting to...
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | added scrapyd system test script to extras/test-scrapyd.sh
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | moved scrapy.utils.sqlite to scrapyd.sqlite
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | removed (barely used) spider context extension, to...
|
commit | commitdiff | tree |
| 2011-08-27 |
Pablo Hoffman | scrapyd: added support for passing custom settings...
|
commit | commitdiff | tree |
| 2011-08-26 |
Pablo Hoffman | removed class method from_settings from ISpiderManager...
|
commit | commitdiff | tree |
| 2011-08-24 |
Pablo Hoffman | fixed unittest broken by previous commit
|
commit | commitdiff | tree |
| 2011-08-24 |
Pablo Hoffman | pass close reason to close() method of new DupeFilter
|
commit | commitdiff | tree |
| 2011-08-19 |
Pablo Hoffman | minor doc fix
|
commit | commitdiff | tree |
| 2011-08-19 |
Pablo Hoffman | fixed priority handling on the new scheduler so that...
|
commit | commitdiff | tree |
| 2011-08-16 |
Pablo Hoffman | added from_crawler class method to SpiderManager
|
commit | commitdiff | tree |
| 2011-08-11 |
Pablo Hoffman | some minor updates to documentation
|
commit | commitdiff | tree |
| 2011-08-11 |
Pablo Hoffman | Automated merge with ssh://hg.scrapy.org:2222/scrapy...
|
commit | commitdiff | tree |
| 2011-08-11 |
Pablo Hoffman | fixed import in doc
|
commit | commitdiff | tree |
| 2011-08-09 |
Pablo Hoffman | added new downloader middleware: ChunkedTransferMiddleware
|
commit | commitdiff | tree |
| 2011-08-09 |
Pablo Hoffman | Automated merge with ssh://hg.scrapy.org:2222/scrapy...
|
commit | commitdiff | tree |
| 2011-08-09 |
Pablo Hoffman | scrapy.utils.python: fixed bug introduced when adding...
|
commit | commitdiff | tree |
| 2011-08-09 |
Pablo Hoffman | scrapy.utils.python: fixed bug introduced when adding...
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | fixed backwards compability with images/media pipeline...
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | removed support for passing more than a single spider...
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | proper fix to what r2760 is supposed to fix
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | Some telnet console changes:
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | fixed bug in scheduler 'has_pending_requests' method...
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | Another step towards singleton removal: deprecated...
|
commit | commitdiff | tree |
| 2011-08-08 |
Pablo Hoffman | replaced DeprecationWarning by a new ScrapyDeprecationWarnin...
|
commit | commitdiff | tree |
| 2011-08-07 |
Pablo Hoffman | added note about engine_started signal
|
commit | commitdiff | tree |
| 2011-08-07 |
Pablo Hoffman | scrapy.utils.sitemap: added one more case of parsing...
|
commit | commitdiff | tree |
| 2011-08-07 |
Pablo Hoffman | scrapy.utils.sitemap: added support for parsing sitemaps...
|
commit | commitdiff | tree |
| 2011-08-07 |
Pablo Hoffman | moved module scrapy.core.downloader.responsetypes to...
|
commit | commitdiff | tree |
| 2011-08-05 |
Pablo Hoffman | added setting to support disabling DNS cache: DNSCACHE_ENABLED
|
commit | commitdiff | tree |
| 2011-08-05 |
Pablo Hoffman | added MarshalDiskQueue unittests
|
commit | commitdiff | tree |
| 2011-08-05 |
Pablo Hoffman | removed no longer working tests from get_engine_status()
|
commit | commitdiff | tree |
| 2011-08-05 |
Pablo Hoffman | MarshalDiskQueue bug fix
|
commit | commitdiff | tree |
| 2011-08-05 |
Pablo Hoffman | scheduler: bug fix to use in-memory queues when request...
|
commit | commitdiff | tree |
| 2011-08-04 |
Pablo Hoffman | removed wrong blocking api usage (socket.gethostbyname...
|
commit | commitdiff | tree |
| 2011-08-04 |
Pablo Hoffman | updated get_engine_status() after scheduler changes
|
commit | commitdiff | tree |
| 2011-08-03 |
Pablo Hoffman | fixed crawlspider bug introduced after scheduler refactoring
|
commit | commitdiff | tree |
| 2011-08-03 |
Pablo Hoffman | added marshal to formats supported by feed exports
|
commit | commitdiff | tree |
| 2011-08-03 |
Pablo Hoffman | Automated merge with ssh://hg.scrapy.org:2222/scrapy...
|
commit | commitdiff | tree |
| 2011-08-03 |
Pablo Hoffman | fix start_python_console() to work with IPython >=...
|
commit | commitdiff | tree |
| 2011-08-02 |
Pablo Hoffman | fixed bug with scheduler.__len__() when disk queues...
|
commit | commitdiff | tree |
| 2011-08-02 |
Pablo Hoffman | added __len__() method to scheduler
|
commit | commitdiff | tree |
| 2011-08-02 |
Pablo Hoffman | cleaned up lxml-based link extractors, and left one...
|
commit | commitdiff | tree |
| 2011-08-02 |
Pablo Hoffman | tie request to downloader failures, so that they can...
|
commit | commitdiff | tree |
| 2011-08-02 |
Pablo Hoffman | Initial support for a persistent scheduler, to support...
|
commit | commitdiff | tree |
| 2011-07-31 |
Pablo Hoffman | imported patch scheduler_single_spider.patch
|
commit | commitdiff | tree |
| 2011-07-31 |
Pablo Hoffman | removed obsolete profiling/ dir
|
commit | commitdiff | tree |
| 2011-07-28 |
Pablo Hoffman | added FAQ about preventing bots getting banned
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | restored support for download delays after downlaoder...
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | Big downloader refactoring to support real concurrency...
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | removed deprecation warning (scheduled to be removed...
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | added fragment attribute ot Link object
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | minor comments adjustments
|
commit | commitdiff | tree |
| 2011-07-27 |
Pablo Hoffman | Added cached DNS resolver based on old caching resolver...
|
commit | commitdiff | tree |
| 2011-07-26 |
Pablo Hoffman | downloader: removed unneeded code, and some minor refactoring
|
commit | commitdiff | tree |
| 2011-07-26 |
Pablo Hoffman | spidermanager: more detailed error message now that...
|
commit | commitdiff | tree |
| 2011-07-26 |
Pablo Hoffman | minor code rearrangement for consistency
|
commit | commitdiff | tree |
| 2011-07-26 |
Pablo Hoffman | removed rather useless (and some deprecated) docstrings
|
commit | commitdiff | tree |
| 2011-07-25 |
Pablo Hoffman | retry middleware: added TCPTimedOutError to exceptions...
|
commit | commitdiff | tree |
| 2011-07-25 |
Pablo Hoffman | fixed bug with scraper KeyError's on some ConnectionLost...
|
commit | commitdiff | tree |
| 2011-07-25 |
Pablo Hoffman | engine: make it more explicit that we don't need to...
|
commit | commitdiff | tree |
| 2011-07-25 |
Pablo Hoffman | more core refactoring including moving engine next...
|
commit | commitdiff | tree |
| 2011-07-25 |
Pablo Hoffman | updated settings.py for djangoitem tests to new django...
|
commit | commitdiff | tree |
| 2011-07-22 |
Pablo Hoffman | downloader: renamed SpiderInfo to Slot, for consistency...
|
commit | commitdiff | tree |
| 2011-07-22 |
Pablo Hoffman | scraper: renamed SpiderInfo to Slot, for consistency...
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | forked UnicodeDammit from BeautifulSoup to explicitly...
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | added missing import
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | removed SimpledbStatsCollector from scrapy code, it...
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | removed FAQ entry
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | engine: renamed slot.requests to slot.start_requests
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | speed up consumption of spider start requests, while...
|
commit | commitdiff | tree |
| 2011-07-20 |
Pablo Hoffman | removed unused import from 'crawl' spider template
|
commit | commitdiff | tree |
| 2011-07-19 |
Pablo Hoffman | added FAQ entry about scrapy deploy issue on Mac +...
|
commit | commitdiff | tree |
| 2011-07-19 |
Pablo Hoffman | moved spider queues to scrapyd
|
commit | commitdiff | tree |
| 2011-07-19 |
Pablo Hoffman | removed no longer used settings
|
commit | commitdiff | tree |
| 2011-07-19 |
Pablo Hoffman | removed SQSSpiderQueue from base scrapy code, it was...
|
commit | commitdiff | tree |
| 2011-07-15 |
Pablo Hoffman | More core changes:
|
commit | commitdiff | tree |
| 2011-07-15 |
Pablo Hoffman | fixed bug in engine.download() method
|
commit | commitdiff | tree |
| 2011-07-14 |
Pablo Hoffman | removed deprecated commands: queue, runserver
|
commit | commitdiff | tree |
| 2011-07-14 |
Pablo Hoffman | fixed python pass handling in cmdline/commands tests...
|
commit | commitdiff | tree |
