Why am I getting "Rate Exceeded" errors for this code in Amazon Kinesis

Question

For the following code, I’m getting “Rate exceeded for shard…” errors.

from boto import kinesis 
import time

kinesis = kinesis.connect_to_region('us-east-1')
response = kinesis.describe_stream('BenTest')

if response['StreamDescription']['StreamStatus'] == 'ACTIVE':
    print '[x] Stream is active'

    shards = response['StreamDescription']['Shards']
    for shard in shards:
        shard_id = shard['ShardId']
        print "[x] Shard ID: %s"    % shard_id

        shard_it = kinesis.get_shard_iterator('BenTest', shard_id, 'LATEST')['ShardIterator']
        print "[x] Shard iterator: %s" % shard_it
while True:
        out = kinesis.get_records(shard_it, limit=2)
        shard_it = out["NextShardIterator"]
            for o in out['Records']:
                print "[x] %s" % o['Data']
                time.sleep(0.5)

It seems to be erroring on the first get_records call.

What am I doing wrong?

Asked By: mr-sk

||

Source

Answer 1

unindent the time.sleep(0.5) one level. you poll the api like crazy when no data is available.

Answered By: hellomichibye

Why am I getting "Rate Exceeded" errors for this code in Amazon Kinesis

Question:

Answers: