How to Store correlation matrix's values in dataframe

Question

I have a df. I want to apply a correlation matrix between the columns and store the result values in another dataframe, however, I do not want to keep the upper part of the matrix, which is useless given that it’s duplicate from the other part, I also don’t want the diagonal (1,1,1,1). So i applied a mask, and then append the result to a dataframe, It

my basic df :

	A	B	C	name
0	0.5	0	8	test1
1	5	1	5	test2
2	3	2	6	test3
3	6	3	10	test4
4	18	14	12	test5

And I would like to do the correlation between those parameters and return the result in another dataframe, but only keeping the upper part to not have the duplicate values. The code is :

dft=pd.DataFrame()
for i in df['name']:
    corrmat = df[df['name']==i].corr()
    corr_triu = corrmat.where(~np.tril(np.ones(corrmat.shape)).astype(np.bool))
    corr_triu = corr_triu.stack()
    corr_triu["name","name"]=i
    dft=dft.append(corr_triu,ignore_index=True)

and that’s the result of my code :

	(A,B)	(B,C)	(C,A)	(name,name)
0	0.5	0.2	0.2	test1
1	0.6	0.8	0.3	test2
2	0.7	0.23	0.25	test3
3	0.5	0.9	0.65	test1
4	0.6	0.4	0.12	test2

then I am trying to use .melt in order to restructure the dataframe but I can’t do it :

dft.melt(id_vars=['(name, name)'], 
        var_name="Var1_Var2", 
        value_name="Value")

the error is :

KeyError: "The following ‘id_vars’ are not present in the DataFrame:
[‘(name, name)’]"

but when I looked for the column name got :

Index([('A', 'B'),
       ('A', 'C'),
       ('B', 'C'),
       ('name', 'name')], dtype='object')

It seems weird that the name is not found, I do not get it.

Here is what I would like to get as a result :

	Var1_Var2	Value	name
0	(A,B)	0.2	test1
1	(A,B)	0.8	test2
2	(A,B)	0.23	test3
3	(B,C)	0.9	test1
4	(B,C)	0.4	test2
5	(B,C)	0.4	test3
6	(C,A)	0.4	test1
7	(C,A)	0.4	test2
8	(C,A)	0.4	test3

There is probably an easier way leading to the result but I can’t find it

Asked By: Art

||

Source

Answer 1

Tuples in columns are problematic, here is solution for change tuple for scalar name:

dft.columns = ['name' if x == ('name', 'name') else x for x in dft.columns]

df = dft.melt(id_vars='name', 
          var_name="Var1_Var2", 
          value_name="Value")
print (df)
     name Var1_Var2  Value
0   test1    (A, B)   0.50
1   test2    (A, B)   0.60
2   test3    (A, B)   0.70
3   test1    (A, B)   0.50
4   test2    (A, B)   0.60
5   test1    (A, C)   0.20
6   test2    (A, C)   0.80
7   test3    (A, C)   0.23
8   test1    (A, C)   0.90
9   test2    (A, C)   0.40
10  test1    (B, C)   0.20
11  test2    (B, C)   0.30
12  test3    (B, C)   0.25
13  test1    (B, C)   0.65
14  test2    (B, C)   0.12

Answered By: jezrael

How to Store correlation matrix's values in dataframe

Question:

Answers: