Creating a csv file with measurement data and meta data in the first line

Question

I want to save measurement data I receive via the CAN bus in a csv file. Apart from the measurement data I want to also store some meta data I need to convert the measurement data I receive via CAN.

The measurement data is for three axis and has 1024 values each. Due to timing issues I store the data in a list while I receive it and store it after the transmission of measurement data I finished. For the reception of the can messages I use the python-can library. The meta data I receive in the first message with measurement data.

I want to store everything in one file with a format like this

x data	y data	z data	meta data 1	meta data 2
x(0)	y(0)	z(0)	2000	3
x(1)	y(1)	x(1)
…	…	…
x(1022)	y(1022)	z(1022)
x(1023)	y(1023)	z(1023)

I am struggling with writing the meta data and first bin of the measurement data in the first line. After writing the first bin and meta in the first line the cells for the meta data should be empty.

There is probably an easy solution to that but I am not searching with the right keywords.

Currently I am just writing the complete list into the csv with the write row function

with open(path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["x_data", "y_data", "z_data"])
    for msgData in msgBuffer:
        writer.writerow([((msgData[1] <<8) | msgData[2]), ((msgData[3] <<8) | msgData[4]) , ((msgData[5] <<8) | msgData[6])])

Here I am struggling with just writing the first line of measurement data and the meta data and then proceed with just writing the measurement data.

Asked By: melfie

||

Source

Answer 1

I’m not entirely sure how you’re getting your metadata, or how to access it, but you’ve said that it’s saved in a variable. For purposes of this answer, I’ll assume there are all the metadata is in a list, metadata (for example: [2000, 3]).

Right now, you’re only writing a header file with the rows for your data, but you’ll also want to include the headers for the metadata. To future-proof this, you can create a generator expression that creates just enough columns for however many elements are in metadata. Then, you can change your for-loop to use enumerate() to check if this is the first iteration or not. If it is not, you can expand metadata into the line, otherwise you can fill it with nothing:

import csv

msgBuffer = [[1, 2, 3, 4, 5, 6, 7], [7, 8, 9, 10, 11, 12, 13]]  # just example data
metadata = [2000, 3]  # just example data

with open('test.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["x_data", "y_data", "z_data", *(f'meta data {i+1}' for i in range(len(metadata)))])
    for i, msgData in enumerate(msgBuffer):
        data = metadata if i == 0 else ['' for i in range(len(metadata))]
        writer.writerow([((msgData[1] <<8) | msgData[2]), ((msgData[3] <<8) | msgData[4]) , ((msgData[5] <<8) | msgData[6]), *data])

This will create a CSV file that looks like this:

x_data,y_data,z_data,meta data 1,meta data 2
515,1029,1543,2000,3
2057,2571,3085,,

Answered By: Michael M.

Creating a csv file with measurement data and meta data in the first line

Question:

Answers: