How to connect to AWS elasticache cluster nodes through a bastion using Python and redis-py-cluster?

Question

I’ve been hitting my head against a wall all day today with this. I’m tunneling into a EC2 instance Bastion server that has access to my Elasticache cluster. I am able to create the connection in my code to the cluster that I’ve tunneled to, but when I execute anything against the cluster, I get a "ClusterError: TTL exhausted".

I think the problem is that the nodes in the connection pool are still the DNS that AWS provides, so when it tries to connect to those, it gets a connection refused (at least that is what it seems like). I’ve tried to use the host_port_remap property but it doesn’t support any host that isn’t an IP address, so I can’t map the AWS DNS to localhost. Any ideas how I can connect and execute against the cluster using this method?

redis_cluster = RedisCluster(
                        host='localhost',
                        port=6379,
                        ssl=True,
                        ssl_cert_reqs=None,
                        decode_responses=True,
                        skip_full_coverage_check=True,
                        password=password)
redis_cluster.set("foo", "bar")`

I’ve tried to use the host_port_remap to map the AWS DNS to my localhost. I’ve tried creating a local Redis cluster in a docker container and connect to that through the AWS SAM lambda running locally, but that also seems to have connection issues.

Asked By: Kyle

||

Source

Answer 1

For anyone who runs into the same issue, did kind of a hacky workaround.

def _map_local_cluster(redis_cluster):
    # need to map the redis cluster nodes to the docker localhost so it can point back to tunneled connection
    # also need to set the remap rule applies function because it expects the host to be an IP but the RedisCluster
    # creates the nodes with the AWS DNS
    host_port_remap = [
        {
            'from_host': '<one nodes DNS on AWS>',
            'from_port': 6379,
            'to_host': 'host.docker.internal',
            'to_port': 6380,
        },
        {
            'from_host': '<other nodes DNS on AWS>',
            'from_port': 6379,
            'to_host': 'host.docker.internal',
            'to_port': 6381,
        },
    ]

    def _remap_rule_applies(remap_rule, node_obj):
        return node_obj[0] == remap_rule['from_host']

    redis_cluster.connection_pool.nodes.host_port_remap = host_port_remap
    redis_cluster.connection_pool.nodes._remap_rule_applies = _remap_rule_applies

Answered By: Kyle

How to connect to AWS elasticache cluster nodes through a bastion using Python and redis-py-cluster?

Question:

Answers: