Python in Execute Process Task

Question:

I have written a python code, then ran it through SSIS in the Execute Process Task, because I want to schedule it with table refresh dependency(which is not possible through windows scheduler or Alteryx).

The code reads and writes to SQL management studio to translate via Google API (pip install googletrans==4.0.0-rc1), with no output or input through SSIS. the input and output are done through sqlacademy in python.

The codes runs perfectly fine in Jupyter notebook, and I saved it to be .py file, but I face a lot of error in the Execute Process Task in SSIS.

Here are a couple of the solutions I tried:

  1. in EXCUTABLE (in command prompt written where python copied .exe path)
    in Arguments C:……file.py
    in WorkingDirectory C:…… (only removing the file name while keeping the backslash
    no white spaces a solution from stackflow)
    Error:

    "C:UsersmeAppDataLocalProgramsPythonPython311python.exe" "UC2.py" at "C:UsersmeDownloadsfoldertest", The process exit code was "1" while the expected was "0".

  2. in EXCUTABLE (in command prompt written where python copied .exe path)
    in Arguments ‘file.py’ (file name in quataion a solution from stackflow)
    in WorkingDirectory C:…… (only removing the file name while keeping the backslash
    no white spaces a solution from stackflow)
    Error:

    The process exit code was "2" while the expected was "0"

  3. move the .exe to a folder were no other files exists.

    The process exit code was "-1073741515" while the expected was "0"

I also tried opening a new project.

then I tried right click change success value no error when running in SSIS, but the following error pops up:

enter image description here

everything even the sql server is installed on the remote desktop.

Asked By: user267454

||

Answers:

HelloWorld.err.py

A trivial python script that will write a string to standard out, another to standard error and the exit with return code 1.

import sys
print("Hello world")
print("Egads, something bad happened!", file=sys.stderr)
exit(1)

SSIS package

A sample SSIS package would look something like the following. An Execute Process Task has a successor of a Script Task.

enter image description here

Double click the precedent constraint (the line between the two) and change that from the default of Success to Completion

enter image description here

EPT Python

An Execute Package Task that is configured as shown

enter image description here

  • Executable: C:Program FilesPython39python.exe
  • Arguments: C:ssisdataSO_75361110.err.py
  • StandardOutputVariable: User::StdOut
  • StandardErrorVariabele: User::StdErr

SCR Echo Back

This is a generic Script Task that I use and abuse to dump the values of SSIS Variables to the output log/results tab. It’s easy, lightweight and let’s you print values without popups or any hard work.

All you need to do is add the Variable(s) in the readonly/readWrite collection. In this case, I selected User::StdOut and User::StdErr

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;

namespace ST_ede4b38a087c4724a71149ffc5ed6be9
{
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
        public void Main()
        {
            bool fireAgain = false;
            foreach (Variable item in Dts.Variables)
            {
                Dts.Events.FireInformation(0, "SCR Echo Back", string.Format("{0}->{1}", item.QualifiedName, item.Value), "", 0, ref fireAgain);
            }

            Dts.TaskResult = (int)ScriptResults.Success;
        }

        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
    }
}

Results

When my package runs, as "expected" the SuccessValue from my run will be 1 given that I use exit(1) from the python script.

Let’s see what the Output tab looks like

SSIS package "C:UsersbfellowssourcereposSO_TrashSO_TrashSO_75361110.dtsx" starting.
Error: 0xC0029151 at EPT Python the thing, Execute Process Task: In Executing "C:Program FilesPython39python.exe" "C:ssisdataSO_75361110.err.py" at "", The process exit code was "1" while the expected was "0".
Task failed: EPT Python the thing
Warning: 0x80019002 at SEQC Do the thing: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED.  The Execution method succeeded, but the number of errors raised (1) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdErr->Egads, something bad happened!
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdOut->Hello world
Warning: 0x80019002 at SO_75361110: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED.  The Execution method succeeded, but the number of errors raised (1) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "C:UsersbfellowssourcereposSO_TrashSO_TrashSO_75361110.dtsx" finished: Failure.
The program '[68044] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).

Of note, are these two lines

Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdErr->Egads, something bad happened!

Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdOut->Hello world

That’s our Script Task echoing back the value of our Standard Out and Standard Error text streams. My suspicion is that given a "normal" run is 10 minutes and the runs from SSIS are measured in seconds, the process execution is throwing errors to the caller but nobody is listening. This will add a listener and give you better information to diagnose your root cause.

Answered By: billinkc

here is how I troubleshooted, in windows I did the following:

To trouble shoot I eliminated factors to find the root cause, thus I noticed SSIS runs on the command prompt so I directly ran the .py in the command prompt.

  1. I wrote in command prompt where python I got two .exe paths then I copied the python version I was using as a path in SSIS in the Execute Process Task.

  2. Next there is two options:

    • Option one, Create a batch file:

      • right click inside a folder press new then text document

      • write (make sure you replace the .exe path with your own executable python path, and .py path with your own python file path):

        " C:Users...python.exe " " C:Users...python_file.py " pause

      • save as name.bat (write .bat after the desired name)

      • then in cmd write cd C:Users...folder (replace the path with the .bat folder path)

      • then write in command prompt the .bat file name with .bat at the end name.bat

    • Option two, run .py directly:

      • Go to windows search bar write system click on system control panel then click advance system settings, a window will pop up click environment variables.

      • Then under User variables for username, double-click path make sure your exactable are added like the following:(note I have Python311 because I have python 3.11)

        C:Users… Python311python.exe

        C:Users..Python311Scripts

        C:Users..Python311

      • Make sure all of the three paths are first by clicking them then selecting move up so this executable will be your default .exe path.

      • To check once writing where python it will now be the first one to come up.

      • Then run by writing in command prompt the following python C:Users...python_file.py

  3. trouble shoot based on the error message, mine was even though the code was running perfectly fine in some of the studios, the installation of packages was not done on the executable I provided in SSIS in the Execute Process Task(which is not the same executable that the studios used), so for installations I did the following:

    • write C:Users...python.exe -m pip install --upgrade pip(you may skip this but my pip was not working)
    • write C:Users...python.exe -m pip install packagename
Answered By: user267454