Eu tenho uma aranha que usa um arquivo como parâmetro, esse arquivo contém os xpaths.

A aranha analisa o arquivo, obtém os xpaths e começa a rastrear.

Tudo está funcionando bem

Agora, quero executar essa aranha várias vezes, então fiz isso:

script.py

def setup_crawler(file):
    spider = MySpider(attributesXMLFilePath=file)
    settings = get_project_settings()
    crawler = Crawler(settings)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()


for oneFile in myFiles:
    setup_crawler(oneFile')
log.start()
reactor.run()

e emMySpider Eu faço isso:

def __init__(self, attributesXMLFilePath):
    dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
        log.msg('The number of pages in the spider {1} are {0}'.format(self.numbers, self.attributesXMLFilePath))
        log.msg('The number of details pages in the spider {1} are {0}'.format(self.numbers2, self.attributesXMLFilePath))
        log.msg('The spider {0} with xml {2} finished working on {1}'.format(self.name, datetime.now(), self.attributesXMLFilePath), level=log.INFO)

mas no arquivo de log, vejo o seguinte:

2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000

Como você vê:

toda linha é duplicada duas vezes,porque?muitas vezes a função spider_close é executadaNota

Eu tenho muito esses dados de log no meu arquivo de log e apenas mostrei uma amostra para explicar meu problema

Nota 2

Ofc eu não estou usandoMySpider, file1.xml efile2.xml nomes, mas não consegui mostrar o nome real dos problemas de privacidade.