A (not so) pleasant surprise from NodeJS

June 2016 ยท 3 minute read

Perhaps it is me who lacks knowledge in basic networking as I learned everything I know mostly by accident and never actually studied it. Maybe it is a lack in NodeJS documentation. Maybe I was expected to know this… but I didn’t. I am currently working on a project called TChaP which is intended to be highly secure. I say intended because it has multiple minor security flaws which could be abused. But I digress. The way TChaP works, is that a client uploads a bunch of encrypted JSON data to the TChaP server which then gets forwarded to another client over TCP.

The problem arose when I started doing random testing with different transfer types which started to fail with large strings… strangely. The error occured specifically with strings of 724 and up. I say strange for two reasons; firstly, it was not an issue where I expected it to be and the pure oddness of the number, it was not familiar.

The moment I saw an error, I didn’t even read it and assumed it was a problem with my encryption algorithms but upon further inspection, I knew that it wasn’t that. So I reluctantly read the error message - It wasn’t valid JSON.

This made no sense to me. Why would my JSON become invalid all of the sudden?

After a few hours of debugging, I noticed something: the last } was missing. In fact, the more data I sent, the more data was missing. This was not packet loss, which is suppose to be random and this was very demonstrable.

I traced a few steps back and realised I had the wrong number. 724 was the number of decrypted characters, 1538 was the total transmitted. Ended up being a very trivial detail, didn’t help a thing.

The code:

socket.on('data', function(data) {
        data = data.toString(); //This is 1538 characters
        plain = decrypt(data).toString(); //This is 724 characters. It is JSON in text format
        json = JSON.parse(plain); //This throws error

       /* SyntaxError: Unexpected end of input
          JSON is *obviously invalid as the last curly bracket is missing (as well as extra data)

At this point, I was very confused. I started looking up documentation on everything that my project used until I realised something: the data event is not triggered when data has been transfered, it is triggered when data is being transfered. After an arbitrary point, it would stop just reading data and leave everything else to be handled by the data event seperately, again.

The solution I ended up taking is ugly. Data is stored and appended into a variable and not processed until a special character is sent at the end.

This solution is not perfect, but it works.