Tengo una araña que toma un archivo como parámetro, este archivo contiene los xpaths.

La araña analiza el archivo y obtiene los xpaths y comienza a gatear.

Todo funciona bien

Ahora, quiero ejecutar esa araña muchas veces, así que hice esto:

script.py

def setup_crawler(file):
    spider = MySpider(attributesXMLFilePath=file)
    settings = get_project_settings()
    crawler = Crawler(settings)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()


for oneFile in myFiles:
    setup_crawler(oneFile')
log.start()
reactor.run()

y enMySpider Hago esto:

def __init__(self, attributesXMLFilePath):
    dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
        log.msg('The number of pages in the spider {1} are {0}'.format(self.numbers, self.attributesXMLFilePath))
        log.msg('The number of details pages in the spider {1} are {0}'.format(self.numbers2, self.attributesXMLFilePath))
        log.msg('The spider {0} with xml {2} finished working on {1}'.format(self.name, datetime.now(), self.attributesXMLFilePath), level=log.INFO)

pero en el archivo de registro, veo esto:

2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000

Como ves:

cada línea se duplica dos veces,por qué?Hay muchas veces que se ejecuta la función spider_closeNota

Tengo muchos datos de registro en mi archivo de registro y acabo de mostrarle una muestra para explicar mi problema

Nota 2

Ofc no estoy usandoMySpider, file1.xml yfile2.xml nombres, pero no pude mostrarte el nombre real de los problemas de privacidad.