Can you safely read utf8 and latin1 files with a naïve try-except block?

I believe that any valid latin1 character will either be interpreted correctly by Python’s utf8 encoder or throw an error. I, therefore, claim that if you work with only either utf8 files or latin1 files, you can safely write the following code to read those files, without ending up with Mojibake:

from pathlib import Path

def read_utf8_or_latin1_text(path: Path, args, kwargs):
    try:
        return path.read_text(encoding="utf-8")
    except UnicodeDecodeError:
        return path.read_text(encoding="latin1")

I tested this hypothesis out on this large character data set and found that it holds up to scrutiny. Is this always the case?

Input:

import requests

insanely_many_characters = requests.get(
    "https://github.com/bits/UTF-8-Unicode-Test-Documents/raw/master/UTF-8_sequence_unseparated/utf8_sequence_0-0x10ffff_including-unassigned_including-unprintable-asis_unseparated.txt"
).text


print(
    f"n=== test {len(insanely_many_characters)} utf-8 characters for same-same misinterpretations ==="
)
for char in insanely_many_characters:
    if (x := char.encode("utf-8").decode("utf-8")) != char:
        print(char, x)


print(
    f"n=== test {len(insanely_many_characters)} latin1 characters for same-same misinterpretations ==="
)
latinable = []
nr = 0
for char in insanely_many_characters:
    try:
        if (x := char.encode("latin1").decode("latin1")) != char:
            print(char, x)
        latinable.append(char)
    except UnicodeEncodeError:
        nr += 1
if nr:
    print(f"{nr} characters not in latin1 set")

print('found the following valid latin1 characters: """n' + "".join(latinable) + 'n"""')

print(
    f"n=== test {len(latinable)} latin1 characters for utf-8 Mojibake ==="
)
for char in latinable:
    try:
        if (x := char.encode("latin1").decode("utf-8")) != char:
            print(char, x)
    except UnicodeDecodeError:
        pass

Output:

=== test 1111998 utf-8 characters for same-same misinterpretations ===

=== test 1111998 latin1 characters for same-same misinterpretations ===
1111742 characters not in latin1 set
found the following latin1 characters: """
    
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ
"""

=== test 256 latin1 characters for utf-8 Mojibake ===

Addendum:

I see I totally forgot to test for sequences of latin1 characters, and only tested for individual characters. By adding this test:

print(
    f"n=== test {len(latinable)} latin1 sequences wrongly interoperable by utf-8 ==="
)
for char1 in latinable:
    for char2 in latinable:
        try:
            if (x := (char1 + char2).encode("latin1").decode("utf-8")) != char1 + char2:
                print(char1 + char2, x)
        except UnicodeDecodeError:
            pass

I ended up generating many utf-8 Mojibake (1920 instances in total), which is a counterexample to my hypothesis!:

=== test 256 latin1 sequences wrongly interoperable by utf-8 ===
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
   
¡ ¡

 ⋮