Defined function is miscounting number of upper/lowercase characters Question: def letcheck(a): upper = 0 lower = 0 for letter in a: if a.islower(): lower += 1 else: upper += 1 print(‘The number of lowercase letters is’, lower) print(‘The number of uppercase letters is’, upper) return letcheck(‘My name is Slugcat’) Hi there. I imagine this is …
How to use Scala UDF accepting Map[String, String] in PySpark Question: Based on the discussion from How to use Scala UDF in PySpark?, I am able to execute the UDF from a scala code for Primitive types, but I want to call scala UDF from PySpark which accepts a Map[String, String]. package com.test object ScalaPySparkUDFs …
How to get element wise boolean array if array elements of column1 exists in array column2? [Pyspark/ Python] Question: My pyspark dataframe has two columns of Array(StringType). I want to check if items in column1 are present in column2. Based on that I want to get bool column which gives me index-wise 1 or 0 …
Snowflake UDF returns "Unknown user-defined function" For Existing UDF Question: I have a UDF that I can call within my snowflakecomputing.com console. SELECT DECODE_UTF8(‘some string’) Works great, until I try to call it programmatically from a Python script. I receive this… snowflake.connector.errors.ProgrammingError: 002141 (42601): or: Unknown user-defined function CS_QA.CS_ANALYTICS.DECODE_UTF8 I am even fully qualifying it …
Pyspark alternative to UDF function which loops an array Question: I’ve searched and can’t find a suitable answer for my Pyspark issue. I’m looking for an alternative approach which is more efficient and doesn’t use a UDF. I have a simple equation in a UDF which has inputs from (a)literal constant, (b)column values, and (c)values …
How to create a udf in PySpark which returns an array of strings? Question: I have a udf which returns a list of strings. this should not be too hard. I pass in the datatype when executing the udf since it returns an array of strings: ArrayType(StringType). Now, somehow this is not working: the dataframe …
Counting consecutive numbers in a list Question: I couldn’t find a question that was similar enough to mine to where I could develop a satisfactory answer. I’m pretty new to Python (3.4.3). I am trying to add elements to an output list using a for loop by comparing each element of an input list to …
Apache Spark — Assign the result of UDF to multiple dataframe columns Question: I’m using pyspark, loading a large csv file into a dataframe with spark-csv, and as a pre-processing step I need to apply a variety of operations to the data available in one of the columns (that contains a json string). That will …
