Aranha raspada envia sinal spider_close antes de fechar
Eu tenho uma aranha que usa um arquivo como parâmetro, esse arquivo contém os xpaths.
A aranha analisa o arquivo, obtém os xpaths e começa a rastrear.
Tudo está funcionando bem
Agora, quero executar essa aranha várias vezes, então fiz isso:
script.py
def setup_crawler(file):
spider = MySpider(attributesXMLFilePath=file)
settings = get_project_settings()
crawler = Crawler(settings)
crawler.configure()
crawler.crawl(spider)
crawler.start()
for oneFile in myFiles:
setup_crawler(oneFile')
log.start()
reactor.run()
e emMySpider
Eu faço isso:
def __init__(self, attributesXMLFilePath):
dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
log.msg('The number of pages in the spider {1} are {0}'.format(self.numbers, self.attributesXMLFilePath))
log.msg('The number of details pages in the spider {1} are {0}'.format(self.numbers2, self.attributesXMLFilePath))
log.msg('The spider {0} with xml {2} finished working on {1}'.format(self.name, datetime.now(), self.attributesXMLFilePath), level=log.INFO)
mas no arquivo de log, vejo o seguinte:
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000
Como você vê:
toda linha é duplicada duas vezes,porque?muitas vezes a função spider_close é executadaNotaEu tenho muito esses dados de log no meu arquivo de log e apenas mostrei uma amostra para explicar meu problema
Nota 2Ofc eu não estou usandoMySpider
, file1.xml
efile2.xml
nomes, mas não consegui mostrar o nome real dos problemas de privacidade.