read a multi tier csv file in python

Question

I need to read the following data out of a text file;

[L02]
g,g,g,g,g,g,g,g,g,g,w,w,w,w,g,g
g,g,g,g,g,g,g,g,g,w,w,w,w,w,g,g
g,g,g,g,g,g,g,g,w,w,w,w,w,g,g,g
g,g,g,g,g,g,g,g,w,w,w,w,g,g,g,g
g,g,g,g,g,g,g,g,g,w,w,w,w,g,g,g
g,g,g,g,g,g,g,g,g,g,w,w,w,w,g,g
g,g,g,g,g,g,g,g,g,g,g,w,w,w,g,g
g,g,g,g,g,g,g,g,g,g,g,w,w,g,g,g
g,g,g,g,g,g,g,g,g,g,g,w,w,g,g,g
g,g,g,g,g,g,g,g,g,g,w,w,w,g,g,g
g,g,g,g,g,g,g,g,g,w,w,w,g,g,g,g
g,g,g,g,g,g,g,g,w,w,w,w,g,g,g,g
g,g,g,g,g,g,g,w,w,w,w,g,g,g,g,g
g,g,g,g,g,g,g,w,w,w,g,g,g,g,g,g
g,g,g,g,g,g,w,w,w,w,w,g,g,g,g,g
g,g,g,g,g,g,g,w,w,w,w,g,g,g,g,g
[L01]
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d
d,d,d,d,d,d,d,d,d,d,d,d,d,d,d,d

I can read a single block as a csv file but I don’t know how to read each file as a separate list

The output I want is to have arrays/lists for each block with the block contents as the list elements. Any ideas?

Asked By: Rishav Sharan

||

Source

Answer 1

If you have numpy, you could read the file into a numpy array. comments='[' tells np.genfromtxt to ignore lines that begin with [. The reshape method places each 16×16 block in its own "layer".

import numpy as np
arr=np.genfromtxt('data.csv',comments='[',delimiter=',',dtype=None)
arr=arr.reshape(-1,16,16)

You can access the nth layer with arr[n].

Answered By: unutbu

Answer 2

Here’s a script that demonstrates how to break down the problem into reusable steps (functions) and performs the transformation your need.

import itertools
import operator
import re
import csv
import pprint

class TaggedLine(str):
    """
    Override str to allow a tag to be added.
    """
    def __new__(cls, val, tag):
        return str.__new__(cls, val)

    def __init__(self, val, tag):
        super(TaggedLine, self).__init__(val)
        self.tag = tag

def sections(stream):
    """
    Tag each line of the stream with its [section] (or None)
    """
    section_pattern = re.compile('[(.*)]')
    section = None
    for line in stream:
        matcher = section_pattern.match(line)
        if matcher:
            section = matcher.group(1)
            continue
        yield TaggedLine(line, section)

def splitter(stream):
    """
    Group each stream into sections
    """
    return itertools.groupby(sections(stream), operator.attrgetter('tag'))

def parsed_sections(stream):
    for section, lines in splitter(stream):
        yield section, list(csv.reader(lines))

if __name__ == '__main__':
    with open('data.csv') as stream:
        for section, data in parsed_sections(stream):
            print 'section', section
            pprint.pprint(data[:2])

Save your file as ‘data.csv’ and the script will run on your data with this output:

section L02
[['g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'w',
  'w',
  'w',
  'w',
  'g',
  'g'],
 ['g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'g',
  'w',
  'w',
  'w',
  'w',
  'w',
  'g',
  'g']]
section L01
[['d',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd'],
 ['d',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd',
  'd']]

Answered By: Jason R. Coombs

read a multi tier csv file in python

Question:

Answers: