config

How to set `spark.driver.memory` in client mode – pyspark (version 2.3.1) Question: I’m new to PySpark and I’m trying to use pySpark (ver 2.3.1) on my local computer with Jupyter-Notebook. I want to set spark.driver.memory to 9Gb by doing this: spark = SparkSession.builder .master(“local[2]”) .appName(“test”) .config(“spark.driver.memory”, “9g”) .getOrCreate() sc = spark.sparkContext from pyspark.sql import SQLContext …

Credentials in pip.conf for private PyPI Question: I have a private PyPI repository. Is there any way to store credentials in pip.conf similar to .pypirc? What I mean. Currently in .pypirc you can have such configuration: [distutils] index-servers = custom [custom] repository: https://pypi.example.com username: johndoe password: changeme From what I’ve found that you can put …

How to validate structure (or schema) of dictionary in Python? Question: I have a dictionary with config info: my_conf = { ‘version’: 1, ‘info’: { ‘conf_one’: 2.5, ‘conf_two’: ‘foo’, ‘conf_three’: False, ‘optional_conf’: ‘bar’ } } I want to check if the dictionary follows the structure I need. I’m looking for something like this: conf_structure = …

Python ConfigParser Checking existence of both Section and Key Value Question: Using ConfigParser’s has_section() method I can check if a section exists in a file, such as: config.has_section(section_name) What would be a command to check if a key exists as well? So it would be possible to verify that both a section and key exists …

Using logging in multiple modules Question: I have a small python project that has the following structure – Project — pkg01 — test01.py — pkg02 — test02.py — logging.conf I plan to use the default logging module to print messages to stdout and a log file. To use the logging module, some initialization is required …

Most Pythonic way to provide global configuration variables in config.py? Question: In my endless quest in over-complicating simple stuff, I am researching the most ‘Pythonic’ way to provide global configuration variables inside the typical ‘config.py‘ found in Python egg packages. The traditional way (aah, good ol’ #define!) is as follows: MYSQL_PORT = 3306 MYSQL_DATABASE = …