Error de transmisión de Python Hadoop "ERROR streaming.StreamJob: Job not Successful!" Y Stack trace: ExitCodeException exitCode = 134
Estoy tratando de ejecutar el script python en el clúster Hadoop usando Hadoop Streaming para el análisis de sentimientos. El mismo script que estoy ejecutando en la máquina local que se está ejecutando correctamente y está dando salida.
para ejecutar en la máquina local, uso este comando.
$ cat /home/MB/analytics/Data/input/* | ./new_mapper.py
y para ejecutar en clúster hadoop utilizo el siguiente comando
$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.5.0-mr1-cdh5.2.0.jar -mapper "python $PWD/new_mapper.py" -reducer "$PWD/new_reducer.py" -input /user/hduser/Test_04012015_Data/input/* -output /user/hduser/python-mr/out-mr-out
El código de muestra de mi script es
#!/usr/bin/env python
import sys
def main(argv):
## for line in sys.stdin:
## print line
for line in sys.stdin:
line = line.split(',')
t_text = re.sub(r'[?|$|.|!|,|!|?|;]',r'',line[7])
words = re.findall(r"[\w']+", t_text.rstrip())
predicted = classifier.classify(feature_select(words))
i=i+1
referenceSets[predicted].add(i)
testSets[predicted].add(i)
print line[7] +'\t'+predicted
if __name__ == "__main__":
main(sys.argv)
La traza de la pila de Exception es:
15/04/22 12:55:14 INFO mapreduce.Job: Task Id : attempt_1429611942931_0010_m_000001_0, Status : FAILED
Error: java.io.IOException: Stream closed at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:434)
...
Exit code: 134
Exception message: /bin/bash: line 1: 1691 Aborted
(core dumped) /usr/lib/jvm/java-7-oracle-cloudera/bin/java
-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Xmx525955249
-Djava.io.tmpdir=/yarn/nm/usercache/hduser/appcache/application_1429611942931_0010/container_1429611942931_0010_01_000016/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1429611942931_0010/container_1429611942931_0010_01_000016 -Dyarn.app.container.log.filesize=0
-Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.0.122 48725 attempt_1429611942931_0010_m_000006_1 16 > /var/log/hadoop-yarn/container/application_1429611942931_0010/container_1429611942931_0010_01_000016/stdout 2> /var/log/hadoop-yarn/container/application_1429611942931_0010/container_1429611942931_0010_01_000016/stderr
....
15/04/22 12:55:47 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Traté de ver los registros, pero en tono me muestra este error. Por favor sugiérame, qué está yendo mal.