Solving Post JSON to Python CGI FieldStorage Error

Estimated reading time: 5 minutes

 

 

 

Download Guideline

• Prepare HTTP server such as XAMPP or WAMP in your windows environment.
• Download and unzip into a folder that http server can access.

 DOWNLOAD SOURCE

 

 

POST Data by Content-Type of TEXT

If you send both TEXT string str_data and JSON data json_data to Python WebApp that uses cgi.FieldStorage(), the server site can get only TEXT string. Note that dataType should be for TEXT, so that the header would include “Content-Type: text/html”.

var str_data = 'EXECUTE';
var json_data = {'commands': {'a00abf1ddb37': [{'devices': ['FAN'], 'execution': [{'params': {'on': true}, 'command': 'OnOff'}]}]}};

$("#btn1").on("click", function () {
    console.log("\nSEND TO cgi.FieldStorage => ");
    console.log(json_data);
    $.ajax({
        url: "webapp-fieldstorage.py",
        type: "POST",
        data : {'string': str_data, 'json': json_data},
        dataType: "text", // Default. This line can be ignored.
        success: function(r) {
            console.log("RECEIVE =>");
            console.log(r);
            $("#rtext").append(r+"\n").scrollTop($('#rtext')[0].scrollHeight);
        },
    });
});

Here, we use a complex JSON data for interpretation.

The server site, webapp-fieldstorage.py, retrieves TEXT string and the complex JSON data, separately. However, JSON data is None type.

{'string': 'EXECUTE', 'json': None}

For TEXT string, cgi.FieldStorage() did it correctly for ‘EXECUTE’.

import json, cgi

form = cgi.FieldStorage()

print('Content-Type: text/html\n')
print(form)
print("\n")

string = form.getvalue('string')
payload = form.getvalue('json')
print({'string': string, 'json': payload})
print("\n")

key = 'json[commands][a00abf1ddb37][0][devices][]'
print(form.getvalue(key))

 

Why Mistakes Happen

We guess there is invalid data structure inside Python cgi.FieldStorage(), and find that using an abnormal key as below, you can get the desired value, FAN, out of JSON data.

key = 'json[commands][a00abf1ddb37][0][devices][]'

Look at the screen layout, clicking on the left button will test the cgi.FieldStorage() error which results in Python None type for JSON data. Also, you can find that cgi.FieldStorage() treat an abnormal string as key in its internal data structure. Indeed, even applying the wrong key we can get the real value, FAN.

 

 

POST Data by Content-Type of JSON

At this time, we send both TEXT string str_data and JSON data json_data to another Python WebApp that uses module sys. Fortunately, the server site can get entire data, even complex JSON data.

Note that dataType should be for JSON, dataType: "json", so that the header would include “Content-Type: application/json”. In addition, what you send should be made by JSON.stringify().

$("#btn2").on("click", function () {
    console.log("\nSEND TO sys.stdin => ");
    console.log(json_data);
    $.ajax({
        url: "webapp-stdin.py",
        type: "POST",
        data : JSON.stringify({'string': str_data, 'json': json_data}),
        dataType: "json",  // HTTP Header will be modified for JSON type automatically.
        success: function(r) {
            console.log("RECEIVE =>");
            console.log(r);
            r = JSON.stringify(r);
            $("#rtext").append(r+"\n\n").scrollTop($('#rtext')[0].scrollHeight);
        },
    });
});

For the server part, webapp-stdin.py, let us inspect inside the JSON POST data and retrieve commands['a00abf1ddb37']. The action indicates that the received is of Python’s Dictionary type, rather than TEXT only. Finally, make commands['a00abf1ddb37'] a TEXT string and send it back to the client site with header protocol of “Content-Type: application/json”.

import json, sys

dict1 = json.load(sys.stdin)

print('Content-Type: application/json\n')
json_data = dict1['json']
commands = json_data['commands']
print(json.dumps(commands['a00abf1ddb37']))

 

Truly Solution to JSON Data POST for Python WebApp

The screen layout below shows that response data are part of request data, and both of them are of JSON format, as indicated in browser’s console logs. Actually, this is a complete solution for Python WebApp to accept JSON POST data without mistakes.

 

 

Suggested Reading

• Python Webapp Demo on Apache CGI for Windows
• Scraping Three Global Weather Channels by Tricks