Row wise cosine similarity calculation in pandas

This can be done without apply (faster):

import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randint(0, 5, (3, 5)), columns=['id', 'commit_date', 'feature1', 'feature2', 'feature3'])


# Calculate L2 norm of features in row
df["l2norm"] = np.linalg.norm(df.loc[:, "feature1":"feature3"], axis=1)

# Create shifted dataframe
df2 = df.shift(1, fill_value=0)


# Dot product of current with previous row
dot_product = (df.loc[:, "feature1":"feature3"] * df2.loc[:, "feature1":"feature3"]).sum(axis=1)

# L2 norm product of current and previous row
norm_product = df["l2norm"] * df2["l2norm"]

# Divide and print
print(dot_product / norm_product)

Try this approach using cosine_similarity

from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd

df['Distance'] = (df.iloc[:, 2:]
                    .apply(lambda row: cosine_similarity([row],
                    [df.iloc[row.name - 1, 2:]])[0][0]
                    if row.name > 0 else None, axis=1))
print(df)

You can also use for-loop but considering the size of your dataframe

similarity_df = cosine_similarity(df.iloc[:, 2:])
df['Distance'] = ([None] + [similarity_df[i, i-1] for i in range(1, len(df))]
print(df)

Note If the provided code do not produce your desired output, you might have to update your question by including the exact values of the Distance column.

TLDR; The goal is to determine the cosine similarity between every row that follows another in a Pandas DataFrame that has numerous feature columns. The cosine_similarity function from sklearn.metrics.pairwise may be used to accomplish this, and the new column created will contain the similarity values. To accomplish this, each successive pair of rows is iterated over, their feature values are converted to a 2D shape, and the cosine_similarity between them is calculated. The resulting similarity values are then put to a list and added to the DataFrame as a new column.

To get the cosine similarity between each succeeding row in your DataFrame, use the cosine_similarity function from the sklearn.metrics.pairwise package.

from sklearn.metrics.pairwise import cosine_similarity

# Calculate cosine similarity for consecutive rows
similarity_list = []
for i in range(len(df) - 1):
    row1 = df.iloc[i, 2:].values.reshape(1, -1)
    row2 = df.iloc[i+1, 2:].values.reshape(1, -1)
    similarity = cosine_similarity(row1, row2)[0][0]
    similarity_list.append(similarity)

# Add similarity values to DataFrame
df['Distance'] = [np.nan] + similarity_list

The cosine_similarity function is used in this code to iteratively calculate the cosine similarity between each pair of subsequent rows in the DataFrame. These similarity values are then added to a list and added as a new column with the name Distance to the original DataFrame.

To ensure that the feature values have a 2D shape, which is necessary for the cosine similarity function, we reshape them using reshape(1, -1).