Building an AMI using Packer with Python & Python modules (using pip) installed via powershell script

Question:

Using Packer, I am trying to create a Windows AMI with Python + the cryptography module installed. Here is the installation command I’m using for Python:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"

Standalone that works fine. If I launch an EC2 instance from the resulting AMI, I can open Powershell and execute python --version and it returns the Python version. This is to be expected since, according to Python documentation, PrependPath=1 will "Add install and Scripts directories to PATH"
In addition, however, I want to install cryptography module so I add the following to the install script:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"
pip install --upgrade pip
pip install cryptography

Now Packer will fail when it gets to the pip command saying The term 'pip' is not recognized as the name of a amazon-ebs.windows: cmdlet, function, script file, or operable program.
I tried adding pip’s location to the system path in multiple different ways but nothing helped. What did work (as well as the addition to the system path) was adding a sleep after the Python install command. Seemingly Packer/Powershell doesn’t wait for the Python installer to finish. So now my install script looks like this:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"
sleep 30
$env:Path += ";C:Program FilesPython36Scripts"
pip install --upgrade pip
pip install cryptography

Now Packer executes no problem and creates the new AMI but when I launch the resulting AMI and run python --version I get 'python' is not recognized as the name of a cmdlet, function, script file, or operable program. Adding commands to the script to append the system path has not helped.

Can anyone shed any light on this predicament?

Asked By: Jay

||
Source

Answers:

As an aside:

There are two problems with your code:

  • Your ./python-3.6.8-amd64.exe installer appears to be GUI-subsystem application, which therefore runs asynchronously by default.

    • Your sleep 30 approach to await completion of the installer isn’t reliable, as there’s no guarantee that it will complete in that time frame, given that execution time may vary based on external factors (system load). Below are two reliable alternatives.

    • A simple trick for making such a call synchronous – i.e. to wait for its completion – is to append an additional pipeline segment (what specific command you use in that last segment is irrelevant; here, Out-Host is used, which would print any stdout output (which GUI applications rarely produce) to the console; use Out-Null to suppress any such output); this approach has the additional advantage that the exit code of the process is reflected in the automatic $LASTEXITCODE variable (see this answer for background information):

      # Note the ... | Out-Host at the end.
python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 | Out-Host
# Inspect $LASTEXITCODE to see if the installer signaled
# failure via a nonzero value.

    • Alternatively, use Start-Process with the -Wait parameter:

      Start-Process -Wait python-3.6.8-amd64.exe '/quiet InstallAllUsers=1 PrependPath=1 Include_test=0'

      • To obtain the process exit code in this case, use the -PassThru switch and examine the .ExitCode property value of the process-info object that -PassThru emits:

        $exitCode = (Start-Process -PassThru -Wait python-3.6.8-amd64.exe '/quiet InstallAllUsers=1 PrependPath=1 Include_test=0').ExitCode

  • Whatever modifications the installer makes to the persistent, registry-based Path environment variable ($env:Path) are not immediately seen in the same session (see this answer for background information).

    • You can use the following command to explicitly refresh your session’s Path value – this should make the pip Python executable discoverable:

      $env:PATH = [Environment]::GetEnvironmentVariable('Path', 'Machine'),
            [Environment]::GetEnvironmentVariable('Path', 'User') -join ';'

    • However, there may be a better, Packer-specific approach, mentioned in a since-deleted answer by Paolo: use of a separate shell provisioner in which the modified Path value is automatically seen (I don’t know Packer, so I cannot spell out this solution.)

Answered By: mklement0