Finding and replacing symbols in list elements
Question:
I need to take a list of items (that list is not static and will change every time the script is run) that may or may not have multiple certain symbols in the list elements, for example:
a = ['C:\Temp\Folder1',
'C:\Temp\Folder2',
'C:\Temp\Folder3',
'C:\Temp\Folder4',
'%ALLUSERSPROFILE%\Temp',
'%WINDIR%\Temp']
and I need to change the item that would have the % symbol to output the following:
a = ['C:\Temp\Folder1',
'C:\Temp\Folder2',
'C:\Temp\Folder3',
'C:\Temp\Folder4',
'${allusersprofile}\Temp',
'${windir}\Temp']
There could be more than one entry in the list as well so I need to make sure I hit every one that may exist. I’ve been wracking my brain trying to come up with something and looked at match-case, if/elif/else, and the suggestions are elegant but I’m not sure how to code something like this as it’s not a simple 1:1 replacement. Any help is appreciated.
EDIT:
I’ve tried adding @shadowtalkers code to mine but can’t get the desired output. I’m sure I’m missing something innocuous, but I don’t know what. The last part of "for" where I’m adding the double-slash, is for API requirements and folder names. Here’s a snippet of my current code:
# importing the module
import requests, json, os, re
# Pattern to recognize when preforming the Folder Exceptions
var_pattern = re.compile(r"%([^%]+)%")
# Function to swap out the '%' symbols in environment variables
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
# Function to write Folder Exceptions
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for p in excluded_folder_list:
var_pattern.sub(process_var_sub, p)
for i in range(len(excluded_folder_list)):
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list
}
payload = json.dumps(excluded_folder_list, indent=4)
print(payload)
url = baseurl + "directorylists"
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
del response
del url
del excluded_folder_list
del payload
else:
return
EDIT 2:
Ok, it’s a hack job, I know, but I got it to work. Any suggestions on cleaning it up?
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for i in range(len(excluded_folder_list)):
excluded_folder_list_cleaned = [
var_pattern.sub(process_var_sub, p)
for p in excluded_folder_list
]
excluded_folder_list = excluded_folder_list_cleaned
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list_cleaned
}
payload = json.dumps(excluded_folder_list, indent=4)
print(payload)
EDIT 3:
I was able to get it working (and made a few more optimizations to my code) thanks to shadowtalkers input. I’m sure it can be cleaned up and optimized more, but here are the snippets of the relative code if someone else has to deal with this in the future.
# Pattern to recognize when performing the Folder Exceptions
var_pattern = re.compile(r"%([^%]+)%")
# Function to swap out the '%' symbols in environment variables from Apex One
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
# Populate elements from the JSON File
def populate_elements(policy_filename):
with open(policy_filename) as jsonFile: # Open the file as a JSON file
data = json.load(jsonFile) # Read the file
policyname = (data['policy'][0]['policyName']) # Get the name of the policy as seen in Apex One
dictionary = (data['policy'][0]['managedSettings']) # Read the managedSettings section (which holds the users custom data)
settings = json.loads(dictionary) # Read the settings as it's own JSON string
return policyname, [settings]
# Function to write Folder Exceptions using the API
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for i in range(len(excluded_folder_list)):
excluded_folder_list_cleaned = [
var_pattern.sub(process_var_sub, p)
for p in excluded_folder_list
]
excluded_folder_list = excluded_folder_list_cleaned
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list_cleaned
}
payload = json.dumps(excluded_folder_list, indent=4)
url = baseurl + "directorylists"
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
del response
del url
del excluded_folder_list
del payload
else:
return
# Open JSON policy file(s) within the current working directory the script is run in
for file in os.listdir(): # Get a list of files in the directory
policy_filename = os.fsdecode(file) # Pull the file name
if policy_filename.endswith(".policy"): # Make sure we're only working with the .policy files
policyName, settings = populate_elements(policy_filename) # Call the function to populate the managed_settings information
for settings in settings: # Iterate the settings to populate fields
# RTS Exclusions
RTSExcludedFolders = (settings["managed_settings"][0]["settings"][1]["value"]['ExcludedFolder'])
RTSExcludedFiles = (settings["managed_settings"][0]["settings"][3]["value"]['ExcludedFile'])
RTSExcludedExtensions = (settings["managed_settings"][0]["settings"][5]["value"]['ExcludedExt'])
write_folder_exception_list(RTSExcludedFolders, policyName, "RTS Scan")
write_extensions_exception_list(RTSExcludedExtensions, policyName, "RTS Scan")
write_filename_exception_list(RTSExcludedFiles, policyName, "RTS Scan")
Answers:
In general, you are looking to replace any pattern of the form %...%, which seems like a great job for regular expressions.
First, you need to match the desired pattern, and capture the text within. This is easy enough with the regex %([^%]+)%:
% literal "%"
( start a capture group
[^%] any character other than "%"
+ repeat the previous character 1 or more times
) end the capture group
% literal "%"
See here to explore this regex further:
- https://regex101.com/r/Ps9FGA/1
- Replace a string located between (other question linked in comments)
Now we can use a nice Python feature and use a function to perform regex substitution on the captured result:
import re
var_pattern = re.compile(r"%([^%]+)%")
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
paths_in = [
r"C:TempFolder1",
r"C:TempFolder2",
r"C:TempFolder3",
r"C:TempFolder4",
r"%ALLUSERSPROFILE%Temp",
r"%WINDIR%Temp",
]
paths_out = [
var_pattern.sub(process_var_sub, p)
for p in paths_in
]
assert paths_out == [
r"C:TempFolder1",
r"C:TempFolder2",
r"C:TempFolder3",
r"C:TempFolder4",
r"${allusersprofile}Temp",
r"${windir}Temp",
]
This feature is documented in https://docs.python.org/3/library/re.html#re.sub:
If repl is a function, it is called for every non-overlapping occurrence of pattern.
Finally, note the use of r"" to prevent Python from interpreting
I need to take a list of items (that list is not static and will change every time the script is run) that may or may not have multiple certain symbols in the list elements, for example:
a = ['C:\Temp\Folder1',
'C:\Temp\Folder2',
'C:\Temp\Folder3',
'C:\Temp\Folder4',
'%ALLUSERSPROFILE%\Temp',
'%WINDIR%\Temp']
and I need to change the item that would have the % symbol to output the following:
a = ['C:\Temp\Folder1',
'C:\Temp\Folder2',
'C:\Temp\Folder3',
'C:\Temp\Folder4',
'${allusersprofile}\Temp',
'${windir}\Temp']
There could be more than one entry in the list as well so I need to make sure I hit every one that may exist. I’ve been wracking my brain trying to come up with something and looked at match-case, if/elif/else, and the suggestions are elegant but I’m not sure how to code something like this as it’s not a simple 1:1 replacement. Any help is appreciated.
EDIT:
I’ve tried adding @shadowtalkers code to mine but can’t get the desired output. I’m sure I’m missing something innocuous, but I don’t know what. The last part of "for" where I’m adding the double-slash, is for API requirements and folder names. Here’s a snippet of my current code:
# importing the module
import requests, json, os, re
# Pattern to recognize when preforming the Folder Exceptions
var_pattern = re.compile(r"%([^%]+)%")
# Function to swap out the '%' symbols in environment variables
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
# Function to write Folder Exceptions
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for p in excluded_folder_list:
var_pattern.sub(process_var_sub, p)
for i in range(len(excluded_folder_list)):
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list
}
payload = json.dumps(excluded_folder_list, indent=4)
print(payload)
url = baseurl + "directorylists"
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
del response
del url
del excluded_folder_list
del payload
else:
return
EDIT 2:
Ok, it’s a hack job, I know, but I got it to work. Any suggestions on cleaning it up?
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for i in range(len(excluded_folder_list)):
excluded_folder_list_cleaned = [
var_pattern.sub(process_var_sub, p)
for p in excluded_folder_list
]
excluded_folder_list = excluded_folder_list_cleaned
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list_cleaned
}
payload = json.dumps(excluded_folder_list, indent=4)
print(payload)
EDIT 3:
I was able to get it working (and made a few more optimizations to my code) thanks to shadowtalkers input. I’m sure it can be cleaned up and optimized more, but here are the snippets of the relative code if someone else has to deal with this in the future.
# Pattern to recognize when performing the Folder Exceptions
var_pattern = re.compile(r"%([^%]+)%")
# Function to swap out the '%' symbols in environment variables from Apex One
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
# Populate elements from the JSON File
def populate_elements(policy_filename):
with open(policy_filename) as jsonFile: # Open the file as a JSON file
data = json.load(jsonFile) # Read the file
policyname = (data['policy'][0]['policyName']) # Get the name of the policy as seen in Apex One
dictionary = (data['policy'][0]['managedSettings']) # Read the managedSettings section (which holds the users custom data)
settings = json.loads(dictionary) # Read the settings as it's own JSON string
return policyname, [settings]
# Function to write Folder Exceptions using the API
def write_folder_exception_list(excluded_folder_list, policy_name, section):
if len(excluded_folder_list):
for i in range(len(excluded_folder_list)):
excluded_folder_list_cleaned = [
var_pattern.sub(process_var_sub, p)
for p in excluded_folder_list
]
excluded_folder_list = excluded_folder_list_cleaned
excluded_folder_list[i] = excluded_folder_list[i] + "\"
excluded_folder_list = {
'name': policy_name + " "+section+" Exclusions",
'description': "",
'items': excluded_folder_list_cleaned
}
payload = json.dumps(excluded_folder_list, indent=4)
url = baseurl + "directorylists"
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
del response
del url
del excluded_folder_list
del payload
else:
return
# Open JSON policy file(s) within the current working directory the script is run in
for file in os.listdir(): # Get a list of files in the directory
policy_filename = os.fsdecode(file) # Pull the file name
if policy_filename.endswith(".policy"): # Make sure we're only working with the .policy files
policyName, settings = populate_elements(policy_filename) # Call the function to populate the managed_settings information
for settings in settings: # Iterate the settings to populate fields
# RTS Exclusions
RTSExcludedFolders = (settings["managed_settings"][0]["settings"][1]["value"]['ExcludedFolder'])
RTSExcludedFiles = (settings["managed_settings"][0]["settings"][3]["value"]['ExcludedFile'])
RTSExcludedExtensions = (settings["managed_settings"][0]["settings"][5]["value"]['ExcludedExt'])
write_folder_exception_list(RTSExcludedFolders, policyName, "RTS Scan")
write_extensions_exception_list(RTSExcludedExtensions, policyName, "RTS Scan")
write_filename_exception_list(RTSExcludedFiles, policyName, "RTS Scan")
In general, you are looking to replace any pattern of the form %...%, which seems like a great job for regular expressions.
First, you need to match the desired pattern, and capture the text within. This is easy enough with the regex %([^%]+)%:
% literal "%"
( start a capture group
[^%] any character other than "%"
+ repeat the previous character 1 or more times
) end the capture group
% literal "%"
See here to explore this regex further:
- https://regex101.com/r/Ps9FGA/1
- Replace a string located between (other question linked in comments)
Now we can use a nice Python feature and use a function to perform regex substitution on the captured result:
import re
var_pattern = re.compile(r"%([^%]+)%")
def process_var_sub(match):
var_name = match.group(1)
return "${" + var_name.lower() + "}"
paths_in = [
r"C:TempFolder1",
r"C:TempFolder2",
r"C:TempFolder3",
r"C:TempFolder4",
r"%ALLUSERSPROFILE%Temp",
r"%WINDIR%Temp",
]
paths_out = [
var_pattern.sub(process_var_sub, p)
for p in paths_in
]
assert paths_out == [
r"C:TempFolder1",
r"C:TempFolder2",
r"C:TempFolder3",
r"C:TempFolder4",
r"${allusersprofile}Temp",
r"${windir}Temp",
]
This feature is documented in https://docs.python.org/3/library/re.html#re.sub:
If
replis a function, it is called for every non-overlapping occurrence ofpattern.
Finally, note the use of r"" to prevent Python from interpreting as a special character inside the quotes.