How to detect zstd compression?

Question

I am currently working on a python application, that works with facebook api’s. As we all know, facebook loves their own technology and is working with zstd for data compression.

The problem: facebook is returning either a uncompressed response with normal json or if the response is longer, it is responding with a zstd compressed json.

My current code is something like this:

import zstd
import json


def handle_response(response)
    json = None
    try:
        json = json.loads(zstd.decompress(response.content))
    except:
        json = json.loads(response.text)

    return json

I am currently wondering, if there is a more clean way to do this and even detect zstd.

Asked By: mynameisgod

||

Source

Answer 1

What you’re doing is fine.

You could, I suppose, check to see if the stream starts with the four bytes 28 b5 2f fd. If it doesn’t, it’s not a zstd stream. If it does, it may be a zstd stream. In the latter case, you would try to decompress and if it fails, you would fall back to just copying the input.

That turns out to be exactly the same as what you’re already doing, since the first thing that zstd.decompress is doing is to look for that signature.

Answered By: Mark Adler

Answer 2

After excute python clause ‘b = zstd.compress(str)’,the byte array always starts with ‘b'(xb5/xfd x03x19x00’ .
So you can use ‘index()’ function to judge.

——————
Does anyone know how to decompress the content which is the http response output type ‘applicaton/zstd’ when using internet browser?
like below:

ache-control: no-store
cf-cache-status: DYNAMIC
cf-ray:
7a51c7dbad5efb2c-SJC
content-length: 1591
content-type: application/zstd
date: Thu, 09 Mar 2023 07:59:49 GMT
nel:
{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
report-to:
{"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=K3U4RKcx6XOP1ekNoJGTilOIZ%2FR4f43q%2BBsvbmmbEkQFHWMJQ5JvhDFDbZFHqVczdsR0rzY24pO9h4kjeehrn3fs0H76%2FO2F612s%2F7%2FjQ%2F6LRjYIb%2BOsPsFOEzIzJTU0NwSyUEmTPW0%3D"}],"group":"cf-nel","max_age":604800}
server: cloudflare

Answered By: James Wang

How to detect zstd compression?

Question:

Answers: