Create multiple dataframes in loop

Question:

I have a list, with each entry being a company name

companies = ['AA', 'AAPL', 'BA', ....., 'YHOO']

I want to create a new dataframe for each entry in the list.

Something like

(pseudocode)

for c in companies:
     c = pd.DataFrame()

I have searched for a way to do this but can’t find it. Any ideas?

Answers:

You can do this (although obviously use exec with extreme caution if this is going to be public-facing code)

for c in companies:
     exec('{} = pd.DataFrame()'.format(c))
Answered By: maxymoo

Just to underline my comment to @maxymoo’s answer, it’s almost invariably a bad idea (“code smell“) to add names dynamically to a Python namespace. There are a number of reasons, the most salient being:

  1. Created names might easily conflict with variables already used by your logic.

  2. Since the names are dynamically created, you typically also end up using dynamic techniques to retrieve the data.

This is why dicts were included in the language. The correct way to proceed is:

d = {}
for name in companies:
    d[name] = pd.DataFrame()

Nowadays you can write a single dict comprehension expression to do the same thing, but some people find it less readable:

d = {name: pd.DataFrame() for name in companies}

Once d is created the DataFrame for company x can be retrieved as d[x], so you can look up a specific company quite easily. To operate on all companies you would typically use a loop like:

for name, df in d.items():
    # operate on DataFrame 'df' for company 'name'

In Python 2 you are better writing

for name, df in d.iteritems():

because this avoids instantiating a list of (name, df) tuples.

Answered By: holdenweb

Adding to the above great answers. The above will work flawless if you need to create empty data frames but if you need to create multiple dataframe based on some filtering:

Suppose the list you got is a column of some dataframe and you want to make multiple data frames for each unique companies fro the bigger data frame:-

  1. First take the unique names of the companies:-

    compuniquenames = df.company.unique()
    
  2. Create a data frame dictionary to store your data frames

    companydict = {elem : pd.DataFrame() for elem in compuniquenames}
    

The above two are already in the post:

for key in DataFrameDict.keys():
    DataFrameDict[key] = df[:][df.company == key]

The above will give you a data frame for all the unique companies with matching record.

Answered By: ak3191

Below is the code for dynamically creating data frames in loop:

companies = ['AA', 'AAPL', 'BA', ....., 'YHOO']

for eachCompany in companies:
    #Dynamically create Data frames
    vars()[eachCompany] = pd.DataFrame()

For difference between vars(),locals() and globals() refer to the below link:

What's the difference between globals(), locals(), and vars()?

Answered By: Chandan

The following is reproducable -> so lets say you have a list with the df/company names:

companies = ['AA', 'AAPL', 'BA', 'YHOO']

you probably also have data, presumably also a list? (or rather list of lists) like:

 content_of_lists = [
 [['a', '1'], ['b', '2']],
 [['c', '3'], ['d', '4']],
 [['e', '5'], ['f', '6']],
 [['g', '7'], ['h', '8']]
]

in this special example the df´s should probably look very much alike, so this does not need to be very complicated:

dic={}
for n,m in zip(companies, range(len(content_of_lists))):
   dic["df_{}".format(n)] = pd.DataFrame(content_of_lists[m]).rename(columns = {0: "col_1", 1:"col_2"}) 

Here you would have to use dic["df_AA"] to get to the dataframe inside the dictionary.
But Should you require more "distinct" naming of the dataframes I think you would have to use for example if-conditions, like:

dic={}
    for n,m in zip(companies, range(len(content_of_lists))):
if n == 'AA':
    special_naming_1 = pd.DataFrame(content_of_lists[m]).rename(columns = {0:     
    "col_1", 1:"col_2"})
elif n == 'AAPL':
    special_naming_2 ...

It is a little more effort but it allows you to grab the dataframe object in a more conventional way by just writing special_naming_1 instead of dic['df_AA'] and gives you more controll over the dataframes names and column names if that´s important.

Answered By: Artur

you can do this way:

for xxx in yyy:
   globals()[f'dataframe_{xxx}'] = pd.Dataframe(xxx)
Answered By: Joao Nogaroli
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.