How to Pass HTTP Request into Queue using PHP

Estimated reading time: 14 minutes

 

 

 

DOWNLOAD GUIDELINE

• Prepare HTTP server such as XAMPP or WAMP in your windows environment.
• Prepare HTTP server such as Apache on Linux. For someone without Linux, it still works to build Linux virtual machine on Windows by freeware like Oracle VM VirtualBox.
• Download and unzip into a folder that HTTP server can access.

 DOWNLOAD SOURCE

 

 

Apache MPMs on Linux

Apache has three MPMs shown in the following, while Prefork is default. Up to Apache 2.4, all modules has been stable.

You can find in configuration files, or issue commands to check which module is currently installed.

joe@hr7:/etc/apache2/mods-enabled$ ls -l mpm*
lrwxrwxrwx 1 root root 34 Mar 15  2019 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root 34 Mar 15  2019 mpm_prefork.load -> ../mods-available/mpm_prefork.load
joe@hr7:~$ apachectl -M | grep mpm
mpm_prefork_module (shared)

 

How Does a HTTP Request Route?

The diagram shows normal path of each request coming from browser. When a child process serves a request and finish the job, it returns the result. After that, the child process waits for another request.

As default MPM Prefork module is installed, Apache always forks child processes previously, and each child process serves assigned jobs sequentially. If getting more busy, Apache generates additional child processes gradually. However, if staying idle, Apache kills spare children one by one until reducing total to be MaxSpareServers.

In Ubuntu, configuration file /etc/apache2/mods-available/mpm_prefork.conf define MaxSpareServers and more related parameters to tell Apache how to control the number of spare child processes at idle time, and moreover, to limit the maxnum number at peak time.

 

 

Apache Child Processes

HTTP global counter means that browsers from everywhere make requests to a single counter system. Each Apache child process accept a request and get counter resource for it through a Queue.

<?php
// Convert a path name and a project identifier to a System V IPC key
$key1 = ftok(dirname(__FILE__) . '/ex1-token.php', 'A');
$key2 = ftok(dirname(__FILE__) . '/ex1-token.php', 'B');
// Creating a message queue with a key, we need to use an integer value.
$queue1 = msg_get_queue($key1);
$queue2 = msg_get_queue($key2);
// Send a message.
$mypid = posix_getpid();
msg_send($queue1, $mypid, 'dummy');
while(1) {
    if ( msg_receive($queue2, $mypid, $msgtype, 4096, $message, false, MSG_IPC_NOWAIT) ) {
        echo parse_msg($message);
        exit();
    }
}
// Parse message
function parse_msg($message) {
    // e.g. $message => s:6:"000007";
    $items = explode(":", $message);
    return substr($items[2], 1, (int)$items[1]);
}
?>

Preforked Apache child process executes the script ex1-token.php as above. At first, ftok() generate System V IPC keys for both $queue1 and $queue1. Next, msg_send() send into $queue1 a dummy message with PID (Process ID) $mypid.

We call $queue2 a priority queue because $msg_receive() listens only the atom with $mypid, no matter what time the atom came in. In other words, each process promises to get message of its own.

 

Daemon Process Dealing with File

At opposite ends of $queue1 and $queue2, a daemon process is listening and serving the jobs in queue.

Let us illustrate by the global counter example in two ways, File or Database, respectively. The first case of daemon process, ex1-qsys-f.php, deal with File.

<?php
// Convert a path name and a project identifier to a System V IPC key
$key1 = ftok(dirname(__FILE__) . '/ex1-token.php', 'A');
$key2 = ftok(dirname(__FILE__) . '/ex1-token.php', 'B');
// Creating a message queue with a key, we need to use an integer value.
$queue1 = msg_get_queue($key1);
$queue2 = msg_get_queue($key2);
// Loop through all, read messages are removed from the queue.
// Here we find a constant MSG_IPC_NOWAIT, without it all will hang forever.
$counter_file = dirname(__FILE__) . "/cnt.txt";
echo "*** qsys-f starting\n";
while (1) {
    if( msg_receive($queue1, 0, $msgtype, 4096, $message, false, MSG_IPC_NOWAIT) ) {
        $cnt_str = get_cnt();
        echo "msgtype: {$msgtype}, message: {$message}, return {$cnt_str}\n";
        msg_send($queue2, $msgtype, $cnt_str );
    }
}
/*
  get counter value
*/
function get_cnt() {
    global $counter_file;
    $fp = fopen($counter_file, "r+");
    $val = (int)fgets($fp);
    $val += 1;
    rewind($fp);
    fwrite($fp, strval($val));
    fclose($fp);
    return sprintf("%06d", $val);
}
?>

Remember that $queue1 incoming $message is dummy and can be ignored, so what daemon process does is to get counter from cnt.txt file for sending back. It returns data to $queue2 with $msgtype, which is actually the PID, to indicate the owner. Only the Apache child process of this PID can get it from $queue2.

$ chgrp www-data cnt.txt
$ chmod g+w cnt.txt

To make sure no privilege problem occurs, issue the above command lines to allow Apache have right to access cnt.txt file, where www-data represents Apache2 user id on Linux.

 

Daemon Process Dealing with Database

For the second case, daemon process ex1-qsys-d.php deal with Database. We leverage a small PHP CRUD class db.php to speed up operations. If interesting in detailed usage of it, you can refer to our article Small PHP CRUD Class for OOP using MySQL.

DROP TABLE IF EXISTS `a0006`;
CREATE TABLE `a0006` (
  `val` int(11) DEFAULT 0 NOT NULL
) ENGINE=InnoDB;
INSERT INTO `a0006` (`val`) VALUES (0);

<?php
require(dirname(__FILE__) . '/db.php');
$dbconfig=array(
    'host' => 'localhost',
    'dbname' => 'mydb',
    'username' => 'myuser',
    'password' => '12345678',
    'charset' => 'utf8',
);
$tblname = "a0006";
// Convert a path name and a project identifier to a System V IPC key
$key1 = ftok(dirname(__FILE__) . '/ex1-token.php', 'A');
$key2 = ftok(dirname(__FILE__) . '/ex1-token.php', 'B');
// Creating a message queue with a key, we need to use an integer value.
$queue1 = msg_get_queue($key1);
$queue2 = msg_get_queue($key2);
// Loop through all, read messages are removed from the queue.
// Here we find a constant MSG_IPC_NOWAIT, without it all will hang forever.
echo "*** qsys-d starting\n";
while (1) {
    if( msg_receive($queue1, 0, $msgtype, 4096, $message, false, MSG_IPC_NOWAIT) ) {
        $cnt_str = get_cnt();
        echo "msgtype: {$msgtype}, message: {$message}, return {$cnt_str}\n";
        msg_send($queue2, $msgtype, $cnt_str );
    }
}
/*
  get counter value
*/
function get_cnt() {
    global $dbconfig, $tblname;
    $conn = new DB($dbconfig);
    $stmt = $conn->fetch("SELECT val FROM {$tblname}");
    if(!$stmt) { echo "fetch() error {$conn->error} \n"; return "_ERROR"; }
    $val = $stmt['val'];
    $val += 1;
    if(!$conn->execute("UPDATE {$tblname} SET val=?", [$val])) {
        echo "execute() error {$conn->error}\n";
        return "_ERROR";
    }
    return sprintf("%06d", $val);
}
?>

By now, get_cnt() get global counter in Database, instead of File, for each HTTP request. Likewise, messages are sent back to $queue2 with owner PID for priority queue approach.

 

 

Normal HTTP Request

<?php
require(dirname(__FILE__) . '/db.php');
$dbconfig=array(
    'host' => 'localhost',
    'dbname' => 'mydb',
    'username' => 'myuser',
    'password' => '12345678',
    'charset' => 'utf8',
);
$tblname = "a0006";
echo get_cnt();
/*
  get counter value
*/
function get_cnt() {
    global $dbconfig, $tblname;
    $conn = new DB($dbconfig);
    $stmt = $conn->fetch("SELECT val FROM {$tblname}");
    if(!$stmt) { echo "fetch() error {$conn->error} \n"; return "_ERROR"; }
    $val = $stmt['val'];
    $val += 1;
    if(!$conn->execute("UPDATE {$tblname} SET val=?", [$val])) {
        echo "execute() error {$conn->error}\n";
        return "_ERROR";
    }
    return sprintf("%06d", $val);
}
?>

Apache child processes serve HTTP requests directly by connecting database, and then reply result to browsers. After connection, do READ and WRITE actions to finish the global counter retrieval.

 

Why FILE LOCK?

<?php
$counter_file = dirname(__FILE__) . "/cnt.txt";
echo get_cnt();
/*
  get counter value
*/
function get_cnt() {
    global $counter_file;
    $fp = fopen($counter_file, "r+");
    if(flock($fp, LOCK_EX)) {
        $val = (int)fgets($fp);
        if($val == 999999) $val = 1; else $val += 1;
        rewind($fp);
        fwrite($fp, strval($val));
        flock($fp, LOCK_UN);
        fclose($fp);
        return sprintf("%06d", $val);
    } else {
        fclose($fp);
        return "_LOCK_";
    }
}
?>

It is necessary to combine READ and WRITE actions together with LOCK and UNLOCK actions by using flock(). Therefore, a series of LOCK-READ-WRITE-UNLOCK actions will avoid race condition between two Apache child processes while they are getting global counter in a specific file at the very exact same moment.

 

 

A Simulated Client

Assume benchmarking host is hr1 and operational host is hr5. A simulated client can be running in hr1 by issuing command lines like the following. The second argument representing URL could be one of the following.

• http://hr5:90/wpp/exams/a0006/ex1-token.php
• http://hr5:90/wpp/exams/a0006/ex2-token-d.php
• http://hr5:90/wpp/exams/a0006/ex2-token-f.php

joe@hr1:~/simcli$ php simcli.php 1 http://hr5:90/wpp/exams/a0006/ex1-token.php

<?php
error_reporting(E_ERROR & ~E_NOTICE);
$url = $argv[2];
$timeout = 30; // seconds
$reqcnt = 0;
$sendfp = fopen('sendlog/' . $argv[1], 'a');
$recvfp = fopen('recvlog/' . $argv[1], 'a');
while($reqcnt < 100) {
    fwrite($sendfp, sprintf("%s", 'send' . PHP_EOL));
    $t1 = microtime(true);
    $r = usecurl(['msg'=>'dummy'], $url);
    $t2 = microtime(true);
    if (($t2-$t1) > $timeout )
        fwrite($recvfp, sprintf("%s %s\n", 'timeout', sprintf("%.4f", $t2-$t1) ) );
    else
        fwrite($recvfp, sprintf("%s %s %s\n", 'elapsed', sprintf("%.4f", $t2-$t1), $r ) );
    $reqcnt += 1;
}
fclose($sendfp);
fclose($recvfp);
function usecurl($post_data, $url){
...
?>

Let us make a test by establishing two more tty connecting on hr5. Looking at the command lines below. On one tty, start daemon ex1-qsys-d.php listening Queue1. It is suggested to run daemon in foreground, so you can observe it.

eric@hr5:~/a0006$ php ex1-qsys-d.php
*** qsys-d starting
msgtype: 13045, message: s:5:"dummy";, return 015091
msgtype: 13050, message: s:5:"dummy";, return 015092

One another tty, the tool ex1-tools.php monitoring real-time status show how busy the system is. After that, we are going to start mass benchmarking in next paragraph.

eric@hr5:~/a0006$ php ex1-tools.php status
q1=>0  q2=>0  apache2 child processes=>10
q1=>0  q2=>0  apache2 child processes=>10

 

Mass Simulated Clients

At experimental testing, we will raise 10 clients at a moment by the tool mgmt.php. Each of them repeats 100 iterated HTTP requests. In other words, there are 10×100 tests. For example, the command line can raise 10×100 cases.

joe@hr1:~/simcli$ php mgmt.php start 10 http://hr5:90/wpp/exams/a0006/ex1-token.php
Duration: 13:53:34 to 13:54:12

Once all concurrent running clients stop, we can check results for both count and average time.

joe@hr1:~/simcli$ php mgmt.php stat
sendlog count: 1000
recvlog count: 1000
elapsed count: 1000
timeout count: 0
elapsed average: 0.3803

<?php
error_reporting(E_ERROR & ~E_NOTICE);
$func=$argv[1]; // start | stop | stat
$maxcnt=(int)$argv[2]; // the number of client programs to be generated
$url = $argv[3];
switch ($func) {
    default:
        echo "INVALID ARGUMENT\n";
        break;
    case "start":
        actionStart($maxcnt);
        break;
    case "stop":
        actionStop();
        break;
    case "stat":
        actionStat();
        break;
}
function actionStart($n) {
    global $url;
    /* remove history data */
    exec("rm sendlog/*");
    exec("rm recvlog/*");
    $err_cnt = 0;
    $prev_pid = 0;
    for ($i = 1; $i <= $n; $i++) {
        $cmd1="php simcli.php {$i} {$url}";
        exec($cmd1 . " > /dev/null &");
        usleep(1000);
    }
    $beginning = date("H:i:s");
    while(1) {
        unset($out);
        $chk1="ps aux | grep 'php simcli.php' | grep -v grep | awk '{ print $2 }' ";
        exec($chk1, $out);
        //echo date("i:s") . " concurrent php simcli.php is " . count($out) . PHP_EOL;
        if(count($out) == 0) break;
        usleep(1000);
    }
    echo "Duration: {$beginning} to " . date("H:i:s") . "\n";
}
function actionStop() {
    unset($out);
    $chk1="ps aux | grep 'php simcli.php' | grep -v grep | awk '{ print $2 }' ";
    exec($chk1, $out);
    foreach($out as $pid) {
        exec('kill ' . $pid);
        echo "killing " . $pid . PHP_EOL;
    }
        exec("rm sendlog/*");
        exec("rm recvlog/*");
}
function actionStat() {
    $folder='sendlog/';
    $files = glob($folder . '*', GLOB_MARK);
    $cnt = 0;
    foreach ($files as $file) {
        unset($out);
        exec("cat " . $file . " | wc -l", $out);
        if (isset($out[0])) $cnt += (int)$out[0];
    }
    echo 'sendlog count: ' . $cnt . PHP_EOL;
    $folder='recvlog/';
    $files = glob($folder . '*', GLOB_MARK);
        $stat = (float)0;
        $cnt = 0;
        $elapsed_cnt = 0;
        $timeout_cnt = 0;
    foreach ($files as $file) {
                $fp = fopen($file, "r");
                while(!feof($fp)) {
                        $ary = explode(' ', fgets($fp));
                        if($ary[0] == 'elapsed') {
                                $stat += (float)$ary[1];
                                $elapsed_cnt += 1;
                        }
                        if($ary[0] == 'timeout') {
                                $timeout_cnt += 1;
                        }
                        $cnt += 1;
                }
                $cnt -= 1;
        }
        echo 'recvlog count: ' . $cnt . PHP_EOL;
        echo 'elapsed count: ' . $elapsed_cnt . PHP_EOL;
        echo 'timeout count: ' . $timeout_cnt . PHP_EOL;
        echo 'elapsed average: ' . sprintf("%.4f", $stat / $elapsed_cnt) . PHP_EOL;
}
?>

 

 

1. Using Queue and Database

Below starts the 100×100 tests in hr1. However, from the result, the average elapsed time for each HTTP request seems too long.

joe@hr1:~/simcli$ php mgmt.php start 100 http://hr5:90/wpp/exams/a0006/ex1-token.php
Duration: 10:28:47 to 10:36:24
joe@hr1:~/simcli$ php mgmt.php stat
sendlog count: 10000
recvlog count: 10000
elapsed count: 10000
timeout count: 0
elapsed average: 4.3972

To optimize the result, another test done below with 3 daemons do reduce the average elapsed time.

joe@hr1:~/simcli$ php mgmt.php start 100 http://hr5:90/wpp/exams/a0006/ex1-token.php
Duration: 10:38:05 to 10:42:31
joe@hr1:~/simcli$ php mgmt.php stat
sendlog count: 10000
recvlog count: 10000
elapsed count: 10000
timeout count: 0
elapsed average: 2.6419

Meanwhile, you can optionally open a terminal in hr5 to observe the real-time status.

eric@hr5:~/a0006$ php ex1-tools.php status
q1=>0  q2=>0  apache2 child processes=>14
q1=>2  q2=>1  apache2 child processes=>14
q1=>3  q2=>2  apache2 child processes=>15
q1=>3  q2=>2  apache2 child processes=>21
q1=>24  q2=>0  apache2 child processes=>29
q1=>1  q2=>19  apache2 child processes=>44
q1=>0  q2=>17  apache2 child processes=>56
q1=>0  q2=>20  apache2 child processes=>75
q1=>0  q2=>38  apache2 child processes=>87
q1=>2  q2=>32  apache2 child processes=>102
q1=>0  q2=>0  apache2 child processes=>109

 

2. Using Queue and File

joe@hr1:~/simcli$ php mgmt.php start 100 http://hr5:90/wpp/exams/a0006/ex1-token.php
Duration: 11:27:42 to 11:27:50
joe@hr1:~/simcli$ php mgmt.php stat
sendlog count: 10000
recvlog count: 10000
elapsed count: 10000
timeout count: 0
elapsed average: 0.2594

 

3. Using Database Directly

joe@hr1:~/wpp/exams/a0006/simcli$ php mgmt.php start 100 http://hr5:90/wpp/exams/a0006/ex2-token-d.php
Duration: 11:32:09 to 11:32:27
joe@hr1:~/wpp/exams/a0006/simcli$ php mgmt.php stat
sendlog count: 10000
recvlog count: 10000
elapsed count: 10000
timeout count: 0
elapsed average: 0.1801

eric@hr5:~/a0006$ php ex2-tools.php status
apache2 child processes=>14
apache2 child processes=>14
apache2 child processes=>15
apache2 child processes=>17
apache2 child processes=>21
apache2 child processes=>29
apache2 child processes=>45
apache2 child processes=>77
apache2 child processes=>109
apache2 child processes=>109

 

4. Using File Directly

joe@hr1:~/wpp/exams/a0006/simcli$ php mgmt.php start 100 http://hr5:90/wpp/exams/a0006/ex2-token-f.php
Duration: 11:54:38 to 11:54:39
joe@hr1:~/wpp/exams/a0006/simcli$ php mgmt.php stat
sendlog count: 10000
recvlog count: 10000
elapsed count: 10000
timeout count: 0
elapsed average: 0.0041

eric@hr5:~/a0006$ php ex2-tools.php status
apache2 child processes=>14
apache2 child processes=>14
apache2 child processes=>15
apache2 child processes=>15
apache2 child processes=>16
apache2 child processes=>18
apache2 child processes=>18
apache2 child processes=>17
apache2 child processes=>16
apache2 child processes=>15
apache2 child processes=>14
apache2 child processes=>14

 

 

Suggested Reading

• PHP Codes Access Shared Memory in Linux with Lock
• 5 Steps to AJAX Login Form with Small PHP DB Class