Ejecutar una araña Scrapy en una tarea de apio
Esto ya no funciona, la API de scrapy ha cambiado.
Ahora la documentación presenta una forma de "Ejecute Scrapy desde un script"pero consigo elReactorNotRestartable
error.
Mi tarea:
from celery import Task
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from scrapy.utils.project import get_project_settings
from .spiders import MySpider
class MyTask(Task):
def run(self, *args, **kwargs):
spider = MySpider
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()