---
date: 2020-05-28T20:00:11+02:00
description: "To deliver compressed static content (in the deflate, gzip, brotli format) by httpd+slowcgi, natives servers on OpenBSD"
draft: false
include_toc: true
show_comments: false
slug: "httpd-delivers-compressed-files"
tags: ['httpd','slowcgi','OpenBSD','CGI','deflate','gzip','brotli']
title: "httpd: deliver compressed static files (+slowcgi)"
translationKey: "httpd-compressed-static-files"
---

## Description

**OpenBSD** has, by default, in basesystem: 

* a webserver, named **httpd**, since 5.7
* a server CGI, named **slowcgi**, since 5.4

* OpenBSD : **6.6, 6.7**

---

**Principle**: if web client accepts datas compression in the deflate, gzip, 
brotli format, then **httpd** redirects to **slowcgi** to deliver the 
compressed static content.

{{< note warning >}}
This is not means compression dynamicly, or "compression on the fly!"
{{</note>}}

**httpd** is not able to manage delivery about compressed static content. 

The tip is to use the server **slowcgi** CGI for. 

In facts, by CGI script shell, we'll assume the delivery of those compressed
static content:  
* for the format: **atom**, **css**, **html**, **js**, **json**, **svg**, **txt**, **xml**
* to both compression: **gzip**, and **brotli**. 

## Installation

You do to [download my script **sbw.cgi**][1].

When it's done, you need to put into the `cgi-bin` folder on web chroot. 
For this, use the command `install` as: <br>
`install -o www -g bin -m 0550 sbw.cgi /var/www/cgi-bin/sbw.cgi` <br>

Indeed, we install correctly with minimals/needed rights to the web user 
`www` and the group `bin`. 


### Dependencies

This script need the installation in the web chroot of several binaries
and some libraries, to run correctly:

Copy from:

* first: **sh**, and binaries: **cat**, **date** and **sha256** — 
	*which are all in the `/bin` system directory*.
* all others binaries: **basename**, **logger** <sup>{{<color orange>}}1{{</color>}}</sup>, 
	**stat** - *which are all in the `/usr/bin` system directory*.
* the shared libc: `/usr/lib/libc.so.xx.0` <sup>{{<color orange>}}2{{</color>}}</sup>
* the library to execution: `/usr/libexec/ld.so`

to the respective folders into web chroot `/var/www/`.

All need `root` user and `bin` group rights:
* with 0555 mode for the binaries, e.g.:
	* **sh**: <br> `install -o root -g bin -m 0555 /bin/sh /var/www/bin/`
	* **stat**: <br> `install -o root -g bin -m 0555 /usr/bin/stat /var/www/usr/bin/`
* and 0444 mode for the libraries, as:
	* **libc**: <br> `install -o root -g bin -m 0444 /usr/lib/libc.so.xx.0 /var/www/usr/lib/`
	* **ld.so**:  <br> `install -o root -g bin -m 0444 /usr/libexec/ld.so /var/www/usr/libexec/`

Before, you need to create the corresponding directories in the web chroot!

See my [dependencies script][2]. 

---

<sup>{{<color orange>}}1{{</color>}}</sup> about the interest of the binary `logger`: 
Normaly, ideally, we don't need the logger. It is about logging some actions, 
which in case of failure, have written in the logs `/var/log/{daemon,messages}`.

Also, if the `debug` variable is set to `1`, on the main function, then 
the logger will return the values corresponding to the different variables, 
for analysis: ensure that variable receive one value, and what value‽

<sup>{{<color orange>}}2{{</color>}}</sup> The **libc.so** change name, 
at each version of OpenBSD: 
* v6.7 : `libc.so.96.0`
* v6.6 : `libc.so.95.1`

This detail is important, can be hardly scripted. You need to modify the
script, at the new version of OpenBSD, to change the name of the library, 
otherwise, it will not work!

---

{{< note tip >}}
To known, what are the dependencies from a binary, use the command `ldd`. 
{{</note>}}

## Configuration

### httpd

Add at your context `server`, all following needed `location` statements:

{{< code "web-httpd-delivre-fichiers-compresses-location-examples" httpd >}}

The file `sbw.conf` contains the following `fastcgi` statements:

{{< file "web-httpd-delivre-fichiers-compresses-fastcgi-examples" httpd "/etc/httpd.d/sbw.conf" >}}

**Explainations** : 

It's important to define, at least:
* **root**: the relative path of script CGI, into the web chroot. 
* **realroot**: a parameter for your web root
* and optionals paramaters, specially of the cache.


### slowcgi

The server **slowcgi** does not require any configuration. Only, you enable
and start with the tool `rcctl`.


## History

### Vulnerabilities

In fact, the story of the {{< abbr HTTP "HyperText Transfert Protocol" >}} protocol 
reveals us two majors vulnerabilities related to the on-the-fly compression:
**CRIME** and **BREACH** — *the second is forked on the first*.  <br>
Thoses vulnerabilities can even impact {{< abbr TLS "Transport Layer Secure" >}}. 

Among the countermeasures has been the adoption since HTTP 1.1 of block 
encoding transfer — *the famous `Transfer-Encoding: chunked` header*; 
similarly, it is strongly recommended to implement a **Referrer** policy 
to allow delivery of compressed content ONLY from the current domain, 
and to refuse it from any other domain.


### about httpd

Reyk Floeter, the httpd author/developer, [denies support for the compressed
content][3]. Even, one [request][4] had be done to support pre-compressed 
content, it's not ready to be integrated.

### brotli

**brotli** is a compression format invented by a team from the company Google.
It is considered to be the successor to gzip because it is faster and has 
a better compression ratio.

For more informations, see: 
* https://brotli.org/
* Does your web client support it: https://caniuse.com/#search=brotli

**curl**, since v7.57.0, support brotli, by adding the option `--compressed`, 
or `-H` — *this manage finely HTTP headers*. (see the [manpage][7])*

#### unsupported clients

* On OpenBSD, **curl** seems not supported brotli.
* Egual, on all OS, **lynx**, **w3m** does not support it. 


### Firefox 

Since v64, [Firefox does not support Atom or RSS feed][5].

Actually, it's more subtle than it sounds:
* if you deliver Atom or RSS with mime type **text/xml**, both are XML files,
	Firefox accepts to read and native display. 
* if you deliver it with their mime type, respectively **application/atom+xml** 
	and **application/rss+xml**, a RFC standard, then Firefox ask you what
	to do with. 

"A nameless aberration!!!"

### the little story


For the little story, Xavier Cartron @prx is the original author of this
genious idea to deliver compressed static content, by adding the header 
`ETag`. *This is an id for the delivered ressource*. 

It is in this context, that the binary **sha256** is useful. 

---

My work, based on his first version, was to add several things:

About {{< anchor brotli brotli >}}, so I resumed/continued writing in order 
to be able to deliver static content previously compressed with this format. 

Next, I added code to manage others needed headers:
* `Content-Length`: to send weight of delivered document. <br> 
	*It is in this context, that the binary **stat** is useful*. 
* `Last-Modified`: to get the modification date of the delivered document; 
	someone considers this header is more relevant than **ETag**.  <br> 
	*It is in this context, that the binaries **date** and **stat** are useful*. 
* `Transfer-Encoding`: to send the delivered document with the good compression
	format, if necessary. 

Then, I wrote the necessary code to detect if the useful dependencies were 
in the web chroot, otherwise the script can't work. If it's the case, the 
serveur send an error 500, with an explicit HTML message. 

**ATTENTION**: the script not installs and can not install the dependancies; 
because, it on the web chroot, it can not "view" the OS filesystem. 

Next, after some research on the web, I understant that the format **deflate**, 
that may be requested by some web clients, is managed by the format **gzip**; 
then, the script supports too.

---

But I was confronted with dysfunctions that I couldn't understand, let alone solve.
*I stopped the project* :(

But, two "things" helped seriously to continue:
* Solène Rapenne, from the OpenBSD team, helped to understand I was making 
	the mistake of sending too many line breaks, when I need only one, 
	at the right time, one between the sending of the different headers 
	and the document itself, whether it is compressed or not.
* the idea to implement a variable `debug` and use the binary `logger` to 
	ensure some differents returns. 

One tips that Solène gave me is the local use of this command: <br>
`env HTTP_ACCEPT_ENCODING=br realroot=/var/www/htdocs/domaine.tld/dev/ PATH_INFO=index.html /var/www/cgi-bin/sbw.cgi | less` <br>
explaining me that it is possible to query the CGI server locally directly, 
by sending it the different possible values before the call. <br>
Amazing! :D

About this, since we installed the CGI shell script **sbw.cgi** with the
user rights to **0550**, we had this erreur:  <br>
`env: /var/www/cgi-bin/sbw2.cgi: Permission denied` <br>
you need to change the other rights to `+x` *(or, `0551`)*. ;-)

---

I improved the detection of the mime type by getting it from the file 
called on the filesystem — *this need to use the binary **basename*** -
and the management of the mime type about feed Atom or RSS.

Finally, I wrote the necessary code to detect if the user agent was {{< anchor Firefox firefox >}}:
* If yes, to obtain his version number. A little hack to deliver feed Atom
	and RSS to the "false" mime-type **text/xml**. Also, Firefox accepts 
	to read the feed instead asking to open with another application!

I wrote the first draft of this hack requiring the addition of the binaries 
`grep` and `awk` — *but, this functional solution did not satisfy me*. 
When, an user on the forum about the french community "OpenBSD pour tous",
@eol makes me [the think to use shell expansion][6]. 

"Et, voilà!"

---

Actualy, @prx rewrote his script in C: 
* the advantage is that there is no need for any dependency; too, it's 
	"protected" by the system call security measures pledge(2) and unveil(2). 
* however, it ONLY supports gzip compression; egual, no header **Last-Modified**,
	or brotli, and deflate support, at least not directly|automatically.


## Documentation

For more documentation about [referrer policy][8].

### manpage

* {{< man install >}}
* {{< man slowcgi 8 >}}, {{< man rcctl 8 >}}
* {{< man pledge 2 >}}, {{< man unveil 2 >}}

### Wikipédia

* About the **BREACH** vulnerability: https://en.wikipedia.org/wiki/BREACH
* and the **CRIME** exploit: https://en.wikipedia.org/wiki/CRIME_(security_exploit)


---

[1]: https://framagit.org/hucste/tools/-/raw/master/OpenBSD/slowcgi/sbw.cgi
[2]: https://framagit.org/hucste/tools/-/blob/master/OpenBSD/chroot_deps
[3]: https://github.com/reyk/httpd/issues/21
[4]: https://github.com/reyk/httpd/issues/80
[5]: https://support.mozilla.org/en-US/kb/feed-reader-replacements-firefox
[6]: https://forum.openbsd.fr.eu.org/showthread.php?tid=2620&pid=20898#pid20898
[7]: https://curl.haxx.se/docs/manpage.html
[8]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy