Python script to run FME workbench
Question:
I have more than 500 xml files and each xml file should processed on FME workbench individually (iteration of FME workbench for each xml file).
For such a propose i have to run a python file (loop.py) to iterate FME workbench for each xml file.
The whole process was working in past on other PC without any problem. Now Once i run Module i got the following error:
Traceback (most recent call last):E:XML_Data
File "E:XML_Dataprocess 1_XML_Tile_1.py", line 28, in
if "Translation was SUCCESSFUL" in open(path_log + "" + data + ".log").read():
IOError: [Errno 2] No such file or directory: ‘E:XML_Datadata_outlog_01re_3385-5275.xml.log’
Attached the python code(loop.py).
Any help is greatly appreciated.
import os
import time
# Mainpath and Working Folder:
#path_main = r"E:XML_Data"
path_main = r"E:XML_Data"
teil = str("01")
# variables
path_in = path_main + r"data_in 3_Placesteil_" + teil # "Source folder of XML files"
path_in_tile10 = path_main + r"data_in 1_Tiling10x10.shp" # "Source folder of Grid shapefile"
path_in_commu = path_main + r"data_in 2_CommunitiesCommunities.shp" # "Source folder of Communities shapefile"
path_out = path_main + r"data_outteil_" + teil # "Output folder of shapefiles that resulted from XML files (tile_01 folder)"
path_log = path_main + r"data_outlog_" + teil # "Output folder of log files for each run(log_01 folder)"
path_fme = r"%FME_EXE_2015%" # "C:Program FilesFME2015fme.exe"
path_fme_workbench = path_main + r"processPY_FME2015.fmw" # "path of FME workbench"
datalists = os.listdir(path_in)
count = 0
# loop each file individually in FME
for data in datalists:
if data.find(".xml") != -1:
count +=1
print ("Run-No." + str(count) + ": with data " + data)
os.system (path_fme + " " + path_fme_workbench + " " + "--SourceDataset_XML"+ " " + path_in + "\" + data + " " + "--SourceDataset_SHAPE" + " " + path_in_tile10 + " " + "--SourceDataset_SHAPE_COMU" + " " + path_in_commu + " " + "--DestDataset_SHAPE" +" " +path_out + " " +"LOG_FILENAME" + " " + path_log + "\" + data + ".log" )
print ("Data processed: " + data)
shape = str(data[19:28]) + "_POPINT_CENTR_UTM32N.shp"
print ("ResultsFileName: " + shape)
if "Translation was SUCCESSFUL" in open(path_log + "\" + data + ".log").read():
# Translation was successful and SHP file exists:
if os.path.isfile(path_out + "\" + shape):
write_log = open(path_out + "\" + "result_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " " + shape + "n")
write_log.close()
print("Everything ok")
#Translation was successful, but SHP file does not exist:
else:
write_log = open(path_out + "\" + "error_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " Data: " + shape + " unavailable.n")
write_log.close()
# Translation was not successful:
else:
write_log = open(path_out + "\" + "error_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " Translation " + Data + " not successful.n")
write_log.close()
print ("Number of calculated files: " + str(count))
Answers:
Most likely, the script failed at the os.system
line, so the log file was not created from the command. Since you mentioned a different computer, it could be caused by many reasons, such as a different version of FME (so the environment variable %FME_EXE_2015% would not exist).
Use a workspace runner transformer to do this.
The FME version is outdated.so first check the version whether it is creating the problem.
subprocess.call(["C:/Program Files/fme/FMEStarter/FMEStarter.exe", "C:/Program Files/fme/fme20238/fme.exe", "/fmefile.fmw" "LOG_FILENAME","logfile"], stdin=None, stdout=None, stderr=None, shell=True, timeout=None)
I have more than 500 xml files and each xml file should processed on FME workbench individually (iteration of FME workbench for each xml file).
For such a propose i have to run a python file (loop.py) to iterate FME workbench for each xml file.
The whole process was working in past on other PC without any problem. Now Once i run Module i got the following error:
Traceback (most recent call last):E:XML_Data
File "E:XML_Dataprocess 1_XML_Tile_1.py", line 28, in
if "Translation was SUCCESSFUL" in open(path_log + "" + data + ".log").read():
IOError: [Errno 2] No such file or directory: ‘E:XML_Datadata_outlog_01re_3385-5275.xml.log’
Attached the python code(loop.py).
Any help is greatly appreciated.
import os
import time
# Mainpath and Working Folder:
#path_main = r"E:XML_Data"
path_main = r"E:XML_Data"
teil = str("01")
# variables
path_in = path_main + r"data_in 3_Placesteil_" + teil # "Source folder of XML files"
path_in_tile10 = path_main + r"data_in 1_Tiling10x10.shp" # "Source folder of Grid shapefile"
path_in_commu = path_main + r"data_in 2_CommunitiesCommunities.shp" # "Source folder of Communities shapefile"
path_out = path_main + r"data_outteil_" + teil # "Output folder of shapefiles that resulted from XML files (tile_01 folder)"
path_log = path_main + r"data_outlog_" + teil # "Output folder of log files for each run(log_01 folder)"
path_fme = r"%FME_EXE_2015%" # "C:Program FilesFME2015fme.exe"
path_fme_workbench = path_main + r"processPY_FME2015.fmw" # "path of FME workbench"
datalists = os.listdir(path_in)
count = 0
# loop each file individually in FME
for data in datalists:
if data.find(".xml") != -1:
count +=1
print ("Run-No." + str(count) + ": with data " + data)
os.system (path_fme + " " + path_fme_workbench + " " + "--SourceDataset_XML"+ " " + path_in + "\" + data + " " + "--SourceDataset_SHAPE" + " " + path_in_tile10 + " " + "--SourceDataset_SHAPE_COMU" + " " + path_in_commu + " " + "--DestDataset_SHAPE" +" " +path_out + " " +"LOG_FILENAME" + " " + path_log + "\" + data + ".log" )
print ("Data processed: " + data)
shape = str(data[19:28]) + "_POPINT_CENTR_UTM32N.shp"
print ("ResultsFileName: " + shape)
if "Translation was SUCCESSFUL" in open(path_log + "\" + data + ".log").read():
# Translation was successful and SHP file exists:
if os.path.isfile(path_out + "\" + shape):
write_log = open(path_out + "\" + "result_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " " + shape + "n")
write_log.close()
print("Everything ok")
#Translation was successful, but SHP file does not exist:
else:
write_log = open(path_out + "\" + "error_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " Data: " + shape + " unavailable.n")
write_log.close()
# Translation was not successful:
else:
write_log = open(path_out + "\" + "error_xml.log", "a")
write_log.write(time.asctime(time.localtime()) + " Translation " + Data + " not successful.n")
write_log.close()
print ("Number of calculated files: " + str(count))
Most likely, the script failed at the os.system
line, so the log file was not created from the command. Since you mentioned a different computer, it could be caused by many reasons, such as a different version of FME (so the environment variable %FME_EXE_2015% would not exist).
Use a workspace runner transformer to do this.
The FME version is outdated.so first check the version whether it is creating the problem.
subprocess.call(["C:/Program Files/fme/FMEStarter/FMEStarter.exe", "C:/Program Files/fme/fme20238/fme.exe", "/fmefile.fmw" "LOG_FILENAME","logfile"], stdin=None, stdout=None, stderr=None, shell=True, timeout=None)