HTTP::Parser::XS#

📦 std

Parse an HTTP request or response head out of a byte buffer - the parser Plack, Starman and Twiggy sit on.

Two entry points, both non-blocking and both operating on a buffer the caller owns:

  • parse_http_request($buf, \%env) fills %env with a PSGI environment and returns the number of bytes the request head occupies, -2 if the head is not complete yet, -1 if it is malformed. A server calls it again with more bytes on -2.

  • parse_http_response($buf, $format, \%special) returns ($ret, $minor_version, $status, $message, $headers) for a client.

use HTTP::Parser::XS qw(parse_http_request parse_http_response HEADERS_AS_HASHREF);

my $req = "GET /search?q=perl%20glob HTTP/1.1\r\nHost: example.com\r\n"
        . "X-Trace: 1\r\nX-Trace: 2\r\n\r\nbody";
my %env;
my $len = parse_http_request($req, \%env);
print "$len bytes of head; $env{REQUEST_METHOD} $env{PATH_INFO} ? $env{QUERY_STRING}\n";
# 81 bytes of head; GET /search ? q=perl%20glob
print "host $env{HTTP_HOST}, trace '$env{HTTP_X_TRACE}'\n";   # host example.com, trace '1, 2'
print parse_http_request("GET / HTTP/1.1\r\nHost:", {}), "\n";  # -2: not complete yet

my ($ret, $minor, $status, $msg, $headers) =
    parse_http_response("HTTP/1.0 404 Not Found\r\nContent-Length: 0\r\n\r\n", HEADERS_AS_HASHREF);
print "$ret: HTTP/1.$minor $status $msg, length $headers->{'content-length'}\n";
# 45: HTTP/1.0 404 Not Found, length 0

$format is HEADERS_AS_HASHREF (a hash of lower-cased names; a repeated header’s values in an array reference), HEADERS_AS_ARRAYREF (name, value pairs in order) or HEADERS_NONE. This is a native implementation of HTTP::Parser::XS 0.17; its XS.pm is used unchanged, and nothing is exported by default.

The PSGI environment#

parse_http_request writes REQUEST_METHOD, REQUEST_URI, SCRIPT_NAME (always ""), PATH_INFO, QUERY_STRING and SERVER_PROTOCOL, then one key per header. Header names are upper-cased, - becomes _, and everything except Content-Type and Content-Length is prefixed HTTP_; those two land as CONTENT_TYPE and CONTENT_LENGTH with no prefix, as CGI and PSGI require. A header sent twice is joined with ", ".

REQUEST_URI keeps the request target verbatim. PATH_INFO is the part before ? with %XX escapes decoded, QUERY_STRING the part after; a # fragment and everything after it is dropped from both but not from REQUEST_URI.

Limits#

At most 128 headers, and a header name of at most 1024 bytes. The two parsers differ on what happens past the name limit: the request parser fails the whole request (-1, and %env is cleared), the response parser silently skips the offending header and carries on.

$BACKEND#

'xs', set by XS.pm once XSLoader::load has loaded this native code. PERL_ONLY=1 or PERL_HTTP_PARSER_XS=pp make XS.pm load the distribution’s pure-Perl HTTP::Parser::XS::PP instead, and then it is 'pp'.

Vectorised tokenizer#

The byte scanners run AVX2 (32B/iteration) or SSE2 (16B) blocks before falling back to picohttpparser’s own eight-way unrolled loop and then a byte tail. Upstream drives PCMPESTRI instead, but only under -msse4.2, which its Makefile.PL never passes - the shipped .so has no vector instruction in it at all - so every CPAN install runs the scalar loop.

Speed relative to the reference XS on a Skylake Xeon E3-1505M (--release, processes interleaved, min of 5 - sequential runs drift enough between processes to invent a 10% result):

workload

speedup vs perl5 XS (>1 = pperl faster)

64 KB header value, HEADERS_NONE

6.01x

64 KB header value

4.23x

4 KB header value

2.15x

1 KB header name

1.95x

4 KB request target

1.25x

realistic 10-header GET

1.00x

response, HEADERS_AS_HASHREF

0.88x

The long-value rows are where the vector scan pays. A realistic request is not bound by scanning: most of its cost is creating the hash entry and the string for each PSGI key, so there the two run at the same speed.

PPERL_HTTP_PARSER_SIMD=avx2|sse2|scalar forces a kernel, for example to compare them on one machine; all three give identical results.

Differences from upstream#

None in what is parsed; only the speed differs.

Functions#

Request parsing#

parse_http_request#

parse_http_request($buf, \%env) - XS.xs parse_http_request.

Response parsing#

parse_http_response#

parse_http_response($buf, $header_format, \%special_headers) - XS.xs parse_http_response.