Error messages are hard
Error messages are hard. When a program logs an error message sometimes both of these things can be true:
- The error message can be 100% accurate.
- The error message can be completely unhelpful, if not outright misleading.
Let’s look at some examples:
import json import sys with open(sys.argv) as f: contents = f.read() print(json.loads(contents))
What happens if we run that against an invalid file:
$ python3 python_json.py test_one.txt Traceback (most recent call last): File "python_json.py", line 5, in <module> print(json.loads(contents)) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What is the root cause here? The file doesn’t contain JSON, it is empty.
Go and JS do a little better in this case, they both say
unexpected end of JSON input
This is true, but the message could still be improved to check for the empty case.
You may be thinking at this point that while the message is bad, you can assume that specific error means that the file was empty. You would be wrong.
Let’s run this same program again against another invalid file, and see what we get:
$ python3 python_json.py test_two.txt ... json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
That file was not empty, it was an HTML file:
$ cat test_two.txt <!doctype html> ...
Go and JS do a bit better here:
invalid character '<' looking for beginning of value Uncaught SyntaxError: Unexpected token '<', "<!doctype html>" is not valid JSON
Why might you run into this error? These errors are common in python programs that
use the requests library to fetch a JSON file from a web server, but don’t
raise_for_status() or check the
r = requests.get("https://example.com/file.json") data = r.json()
r = requests.get("https://example.com/file.json") r.raise_for_status() data = r.json()
Without the call to
raise_for_status that program will raise the same
With the call to
raise_for_status it will raise:
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://example.com/file.json
How could this be improved?
The error messages from Go and JS show that there are some easy wins here. Multiple languages could benefit from special casing the empty string case.
OpenSSL wrong version number
Let’s run curl and see if we can understand what the problem is from the error message:
$ url=https://... $ curl $url curl: (35) OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
This error message may lead you to believe that there is an issue with the TLS version being used. For example a client that wants to use TLS 1.3 connecting to an old server that only supports 1.2.
If we run curl with
--trace -, we can see what the real issue is:
... <= Recv SSL data, 5 bytes (0x5) 0000: 48 54 54 50 2f HTTP/ == Info: OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
The problem is that the server is speaking plain-text HTTP, not TLS, on that port. Where it was expecting to receive a SSL handshake, it received
HTTP/1.1 400 Bad Request Server: nginx ...
Normally it’s supposed to get something like this:
<= Recv SSL data, 5 bytes (0x5) 0000: 16 03 03 00 7a ....z
HTTP/ in ASCII is not a proper version, OpenSSL complains. The error
message is accurate, but not helpful.
The assumption the user might make is that there is some issue with the supported TLS version, and not that the server is not even speaking TLS in the first place.
How could this be improved?
OpenSSL could include the value in the error message, or make it available to the caller somehow.
This would help, but logging
485454502f without converting it to ASCII first
isn’t as helpful as it could be. A web search for
485454502f has results
full of people who have that value in an error message, but don’t know why.
If the value was available to the caller, libcurl could check for this case.
OpenSSL could check if the invalid version is
485454502f and if so, handle
this specific case. Since OpenSSL is used for more than just HTTP this might
not be appropriate.
Connection reset by peer
Connection reset by peer, AKA
In theory, this error is straightforward. It is generated when an
is received from the remote side of a TCP connection. If the client is logging
this error message, the log on the server side should indicate what the problem
In the real world, you may find that both sides of a connection are
ECONNRESET at the same time. The root cause for this is
that there is a middlebox somewhere
resetting the connection. This device can be an “inline IPS”, or a “Next Gen
To further complicate things, admins of such devices are often unaware that the
underlying mechanism used to block connections may involve injecting spoofed TCP
resets. I was once troubleshooting this exact issue on a customer network and
asked “Do you have an IPS or a Next Gen Firewall such as a Palo Alto”. They
responded “yes”, and we got their Palo admin on the call. Their Palo admin
proceeded to tell me that the Palo couldn’t be the cause of this problem,
because “The Palo doesn’t do that”. We then did a search in the Palo UI for our
IP address and the result was a screen full of entries that had an action not
of something like
allow, but of
reset-both. The action
corresponds to sending spoofed TCP resets to both the client and the server at
the same time.
How could this be improved?
This is a hard one. I suppose the error message could be more accurate as
“Connection reset by peer or (because RST packets are not authenticated in any way) an RST packet was spoofed by a next gen firewall”