Python Webapp Demo on Apache CGI for Windows

Estimated reading time: 9 minutes

 

 

 

Download Guideline

• Install Python on Windows by clicking Python Downloads, or search a Python setup pack for Linux.
• The installation package for Windows also contains pip install, which allow you to obtain more Python libraries in the future.

 DOWNLOAD SOURCE

 

 

PHP Scripts and Apache

PHP language dominates almost all server-site scripts on HTTP services like Apache. These scripts serve every web request from browser clients implemented by client-site script JavaScript.

Even though PHP and JS cooperates with each other around http protocol, sometimes, Python takes more advantages than PHP in specific topics. Python cooperating with JS directly should give extra benefits to http services.

 

Python Scripts and Apache CGI

Languages other than PHP like Python and Perl depend on CGI mechanism in order to cooperate with Apache or other web service softwares. That is, CGI can help Python to communicate with Apache. How to configure CGI on Apache is what we will discuss in the article.

In the article, the way to enable Python webapp focuses on Apache2 CGI configurations, rather than older versions. Among OS platforms, Apache CGI settings for Python are a little bit different between Windows, Ubuntu Linux and CentOS Linux.

Apart from CGI mechanism, there are more interfaces to web servers such as Flask and WSGI. Flask can also create a independent web server without Apache installed. WSGI is the interface specification for forwarding requests from web servers to web applications, but installing and setting WSGI on Windows is not easy.

 

Advantages of Python Webapp

If JS scripts in browser clients can directly access to the competence of Python data science, web applications get higher value than before. The Python language itself is capable of web scrawling, data science by libraries such as numpy, and even artificial intelligence.

 

 

Enable Apache CGI on Windows

We demonstrate configuration using the most popular web service, xampp, in Windows. It enables the CGI setting on default. To check the default setting, you just make sure that the following line in httpd.conf is not commented.

LoadModule cgi_module modules/mod_cgi.so

 

Handle Python Script Extension .py

Another configuration point is to let Apache knows the extension .py is of Python scripts, not just of text contents. There are two steps to do.

First, in the file httpd.conf, the directory set by DocumentRoot should specify .py by AddHandler directive, and add an option of ExecCGI. Look at the configuration part as below.

DocumentRoot "C:/xampp/htdocs"
<Directory "C:/xampp/htdocs">
    .....
    Require all granted
    Options +ExecCGI
    AddHandler cgi-script .py
</Directory>

Next, if you want an alternative CGI directory to place your Python scripts, instead of the directory DocumentRoot defined, do the following. For example, you can associate CGI directory with the directory C:/xampp/cgi-bin, which is even not under RootDocument C:/xampp/htdocs. Use the ScriptAlias directive to do it as below.

<IfModule alias_module>
    .....
    ScriptAlias /cgi-bin/ "C:/xampp/cgi-bin/"
</IfModule>
<Directory "C:/xampp/cgi-bin">
    .....
    Require all granted
</Directory>

Remember that the CGI directory should have be granted with access rights by using Require all granted. Finally, restart xampp to get new settings.

 

 

Enable Apache CGI on Ubuntu

Apache CGI on Ubuntu is not default set on. Issue the command line to enable CGI.

$ sudo a2enmod cgi

Based on the Virtual Host structure of Apache, there are two files you have to modify for handling the extension .py. The file 000-default.conf defines a default virtual host, and the file apache2.conf customizes the DocumentRoot directory. Similar to Windows, add ExecCGI in it.

<VirtualHost *:80>
    .....
    DocumentRoot /var/www/html
</VirtualHost>

<Directory /var/www/html>
    .....
    Require all granted
    Options +ExecCGI
    AddHandler cgi-script .py
</Directory>

In addition, if a separated CGI directory is required, set the following in file serve-cgi-bin.conf by using ScriptAlias directive.

<IfModule mod_alias.c>
    .....
    <IfDefine ENABLE_USR_LIB_CGI_BIN>
        ScriptAlias /cgi-bin/ /var/www/cgi-bin/
        <Directory "/var/www/cgi-bin">
        .....
        </Directory>
    </IfDefine>
</IfModule>

Restart Apache to refresh new settings.

$ sudo service apache2 restart

 

Enable Apache CGI on CentOS

Unlike Ubuntu, CentOS presets the CGI mechanism available on default.

Next, you want to specify .py by AddHandler directive, and add an option by Options +ExecCGI directive.

<Directory /var/www/html>
    .....
    Require all granted
    Options +ExecCGI
    AddHandler cgi-script .py
</Directory>

Moreover, when a separated CGI directory is required, set the following. The configuration is different from that on Ubuntu but similar to that on Windows.

<IfModule alias_module>
    .....
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<Directory "/var/www/cgi-bin/">
    .....
    Require all granted
</Directory>

Restart http service to renew the configuration.

$ sudo service httpd restart

 

 

An Example – Crawling Stock Price

As illustrated below, our example JS scripts display price of the selected stock. The price is crawled from web pages by Python scripts. Also, the demos let you know how Python scripts get HTTP environment variables such as HTTP_HOST, REQUEST_URI, and so on.

Traditionally, JS scripts request PHP scripts for information. In contrast, these example JS scripts request server-site Python scripts through Apache CGI mechanism here.

 

HTTP Environment Variables

Click the orange button to get environment variables packed in JSON format. You should view console logs to find variables, because displaying JSON data directly in HTML is impossible to see.

Scripts for getting http environment variables of Apache2 are as below. The object environ comes from Python default library os.

import json
from os import environ
.....
def get_env() :
    print('Content-Type: application/json\n')
    # HTTP environment variables
    env = dict()
    for key in environ :
        env[key] = environ[key]
    # Convert to JSON data
    json_str = json.dumps(env)
    print(json_str)

 

Request with GET and POST Parameters

In server sites, retrieving GET or POST parameters is in the same way. You can get values by using method .getvalue() from the library cgi, which is also Python default library.

Here we get POST data to determine which stock price users want to search for, and then download the web page content by requests. If done, beautifulsoup scrape the content to find what we want.

def get_stock(params) :
    print('Content-Type: text/html\n')
    # POST data
    stock = params.getvalue('stock')
    print("<h3>Inquery {} ...</h3>".format(stock))
    # Get the Content of web page
    the_url = "https://finance.yahoo.com/quote/{}/".format(stock)
    r = requests.get(the_url)
    if r.status_code != 200:
        print('Download Err: '+r.url)
        exit()
    # Crawl and analyze
    soup = BeautifulSoup(r.text, "html.parser")
    info = soup.find("div", attrs={"id": "quote-header-info"})

    items = list()
    items.append(info.find("h1", attrs={"data-reactid": "7"}).string + "<br>")
    items.append(info.find("span", attrs={"data-reactid": "9"}).string + "<br>")
    price = info.find("fin-streamer", attrs={"data-symbol": stock})
    increase = price.next_sibling.span.string
    percent = price.next_sibling.next_sibling.next_sibling.next_sibling.next_sibling.span.string
    items.append(f"{price.string} {increase} {percent} <br>")
    items.append(info.find("div", attrs={"id": "quote-market-notice"}).span.string + "<br>")
    notification = "\n".join(items)
    print(notification)

 # Start up
if __name__ == "__main__":
    params = cgi.FieldStorage()
    action = params.getvalue('action')
    # Two actions with different Content-Type
    if action == "stock" : get_stock(params)
    if action == "env" : get_env()

For the browser client stockprice.html, it sends POST data by jQuery. Similarly, you can sends GET data. We use jQuery Mobile for layout decoration and responsive web design, so the screen layout looks well in mobile phones, too.

<!-- jQuery Mobile -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
<link rel="stylesheet" href="http://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.css">
<script src="http://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"></script>
.....
// User click
$('.ui-btn').on("click", function() {
    send($(this).attr("id"));
});
// Send out
function send(act) {
    stock = $("#selected").text();
    $.ajax({
        url: "stockprice.py",
        type: "POST",
        data: { action: act, stock: stock },
        success: function(r) {
            if (act == "env") {
                console.log(r);
                $("#response").html("Look at JSON data in console log");
            } else {
                $("#response").html(r);
            }
        },
    });
}

Back to server sites. In Linux, you have to change file mode of Python scripts to be executable as the following.

$ chmod +x stockprice.py

Before running correctly, you should install these two Python libraries.

C:\> pip install requests
C:\> pip install beautifulsoup4

Or on Linux,

$ pip3 install requests
$ pip3 install beautifulsoup4

To learn more details in scrawling web pages using beautifulsoup, refer to Python Web Scraping using BeautifulSoup in 3 Steps.

 

Response Leading with Content-Type

It is essential for Python scripts to response with the leading content type of text/html or application/json as below. Note that the "\n" is a must.

For example, we send back stock price using leading content type of text/html, but send out HTTP environment variables using application/json.

def get_stock(params) :
    print('Content-Type: text/html\n')
.....
def get_env() :
    print('Content-Type: application/json\n')

 

Specifying Python Executable Binary File

Executing Python scripts in CGI must specify the path of Python executable binary file such as python.exe in Windows or python3 in Linux.

#!C:/Users/user/AppData/Local/Programs/Python/Python39/python.exe

or

#!/usr/bin/python3

Moreover, suppose that the xampp installation is on D: drive, or you want to prevent errors from other ambiguous reasons, you had better add drive names like C: and D: in Apache configuration on Windows.

Provided you don’t know where Python executable binary file is, find the path by using Windows Start Menu in Windows or, in Linux, by issuing the command line that will result in where Python is installed.

$ which python3
/usr/bin/python3

 

 

Suggested Reading

• WSGI App in Python under Apache mod_wsgi
• Python Web Scraping using BeautifulSoup in 3 Steps
• NLP Chatbot Deep Learning by Tensorflow and NLTK