Write multi-line secret to windows runner in GitHub workflow

Question:

Summary

What specific syntax must be changed in the code below in order for the multi-line contents of the $MY_SECRETS environment variable to be 1.) successfully written into the C:\Users\runneradmin\somedir\mykeys.yaml file on a Windows runner in the GitHub workflow whose code is given below, and 2.) read by the simple Python 3 main.py program given below?

PROBLEM DEFINITION:

The echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\Users\runneradmin\somedir\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.

We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. … This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.

EXPECTED RESULT:

The multi-line secret should be printed into the C:\Users\runneradmin\somedir\mykeys.yaml file and read by main.py.

The resulting printout of the contents of the C:\Users\runneradmin\somedir\mykeys.yaml file should look like:

***  
***  
***  
***

LOGS THAT DEMONSTRATE THE FAILURE:

The result of running main.py in the GitHub Actions log is:

ccc item is:  $MY_SECRETS

As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.

REPO FILE STRUCTURE:

Reproducing this error requires only 2 files in a repo file structure as follows:

.github/
    workflows/
        test.yml
main.py

WORKFLOW CODE:

The minimal code for the workflow to reproduce this problem is as follows:

name: write-secrets-to-file
on:
  push:
    branches:
      - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          pathlib.Path("C:\Users\runneradmin\somedir\").mkdir(parents=True, exist_ok=True)
          print('About to: echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml')
          output = subprocess.getoutput('echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml')
          print(output)
          os.chdir('D:\a\myRepoName\')
          mycmd = "python myRepoName\main.py"
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
            # returns None while subprocess is running
            retcode = p.poll() 
            line = p.stdout.readline()
            print(line)
            if retcode is not None:
              break

MINIMAL APP CODE:

Then the minimal main.py program that demonstrates what was actually written into the C:\Users\runneradmin\somedir\mykeys.yaml file is:

with open('C:\Users\runneradmin\somedir\mykeys.yaml') as file:
  for item in file:
    print('ccc item is: ', str(item))
    if "var1" in item:
      print("Found var1")

STRUCTURE OF MULTI-LINE SECRET:

The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:

var1:value1
var2:value2
var3:value3
var4:value4

These 4 lines should be what gets printed out when main.py is run by the workflow, though the print for each line should look like *** because each line is a secret.

Asked By: CodeMed

||
Source

Answers:

You need to use yaml library:

import yaml

data = {'MY_SECRETS':'''
var1:value1
var2:value2
var3:value3
var4:value4
'''}#add your secret 

with open('file.yaml', 'w') as outfile: # Your file
    yaml.dump(data, outfile, default_flow_style=False)

This is result:
Result
I used this.

Answered By: George

I tried the following code and it worked fine :

LIST_OF_SECRETS

key1:val1
key2:val2

Github action (test.yml)

name: write-secrets-to-file
on:
  push:
    branches:
      - main
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agentt
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import base64, subprocess, sys
          import os
          secrets = os.environ["MY_SECRETS"]
          
          def powershell(cmd, input=None):
              cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
              stdin = None if input is None else subprocess.PIPE
              process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
              if input is not None:
                  input = input.encode(sys.stdout.encoding)
              output, stderr = process.communicate(input)
              output = output.decode(sys.stdout.encoding).replace('rn', 'n')
              return output
          
          command = r"""$secrets = @'
          {}
          '@
          $secrets | Out-File -FilePath .mykeys.yaml""".format(secrets)
          
          command1 = r"""Get-Content -Path .mykeys.yaml"""
          
          powershell(command)
          print(powershell(command1))

Output

***
***

As you also mention in the question, Github will obfuscate any printed value containing the secrets with ***

EDIT : Updated the code to work with multiple line secrets. This answer was highly influenced by this one

Answered By: JAAAY

Edit: updated with fixed main.py and how to run it.

You can write the key file directly with Python:

      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\Users\runneradmin\somedir\').mkdir(parents=True, exist_ok=True)
          with open('C:\Users\runneradmin\somedir\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

To avoid newline characters in your output, you need a main.py that removes the newlines (here with .strip().splitlines()):

main.py

with open('C:\Users\runneradmin\somedir\mykeys.yaml') as file:
    for item in file.read().strip().splitlines():
        print('ccc item is: ', str(item))
        if "var1" in item:
            print("Found var1")

Here’s the input:

LIST_OF_SECRETS = '
key:value
key2:value
key3:value
'

And the output:

ccc item is:  ***
Found var1
ccc item is:  ***
ccc item is:  ***
ccc item is:  ***

Here is my complete workflow:

name: write-secrets-to-file
on:
  push:
    branches:
      - master
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\Users\runneradmin\somedir\').mkdir(parents=True, exist_ok=True)
          with open('C:\Users\runneradmin\somedir\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

Also, a simpler version using only Windows shell (Powershell):

      - name: Create key file
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          mkdir C:\Users\runneradmin\somedir
          echo "$env:MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py
Answered By: duthils

The problem is – as it is so often – the quirks of Python with byte arrays and strings and en- and de-coding them in the right places…

Here is what I used:

test.yml:

name: write-secrets-to-file
on:
  push:
    branches:
    - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          import os
          # using os.path.expanduser() instead of hard-coding the user's home directory
          pathlib.Path(os.path.expanduser("~/somedir")).mkdir(parents=True, exist_ok=True)
          secrets = os.getenv("MY_SECRETS")
          with open(os.path.expanduser("~/somedir/mykeys.yaml"),"w",encoding="UTF-8") as file:
              file.write(secrets)
          mycmd = ["python","./main.py"]
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
              # returns None while subprocess is running
              retcode = p.poll()
              line = p.stdout.readline()
              # If len(line)==0 we are at EOF and do not need to print this line.
              # An empty line from main.py would be 'n' with len('n')==1!
              if len(line)>0:
                # We decode the byte array to a string and strip the
                # new-line characters r and n from the end of the line,
                # which were read from stdout of main.py
                print(line.decode('UTF-8').rstrip('rn'))
              if retcode is not None:
                  break

main.py:

import os
# using os.path.expanduser instead of hard-coding user home directory
with open(os.path.expanduser('~/somedir/mykeys.yaml'),encoding='UTF-8') as file:
    for item in file:
        # strip the new-line characters r and n from the end of the line
        item=item.rstrip('rn')
        print('ccc item is: ', str(item))
        if "var1" in item:
            print("Found var1")

secrets.LIST_OF_SECRETS:

var1: secret1
var2: secret2
var3: secret3
var4: secret4

And my output in the log was

ccc item is:  ***
Found var1
ccc item is:  ***
ccc item is:  ***
ccc item is:  ***
Answered By: Frank