---
categories: ['']
date: 2021-07-30T15:35:45+02:00
description: "Install to use Goaccess on OpenBSD (with configuration snippets for both webserver, httpd and nginx)"
draft: false
include_toc: true
show_comments: false
tags: ['Monitoring','dataviz','log','goaccess','OpenBSD','httpd','nginx']
title: "Goaccess / OpenBSD"
translationKey: 'monitoring-goaccess-openbsd'
---
## Description
**Goaccess** is a FLoSS, known to be light, fast, in order to to analyze
in real time or not the activity on a web server, either directly within
a Unix terminal, or on the HTTPS protocol.
It's able to produce statistics on HTML, JSON even CSV format.
---
⇒ Environnement:
* OpenBSD: 6.9 → 7.1
## Installation
Usual: `# pkg_add goaccess`
## Configuration
The main file config is on: `/etc/goaccess/goaccess.conf`.
---
⇒ You can use goaccess without any modifications on its file config.
This need to pass all options on the CLI, but few options are clearly
commons and can be "frozen" on the file config.
⇒ Too, it's also possible to create numerous files config, each specific
to a web domain, to set particularly segun this domain. In this case, you
need to specify it…
*In this article, I use one configuration file, and put some options on
the CLI.*
Now, let's see the main configuration options:
### Time and Date formats
Formatting the time!
⇒ If you use **httpd**, you need to be careful to set correctly the style
of the log, and into in the goaccess configuration file in the
**Time Format Options** section.
- see: [httpd.conf](https://man.openbsd.org/httpd.conf.5#style)(5)#style
⇒ For **nginx**, it will be enough to uncomment the following options:
- **time-format %H:%M:%S**
- **date-format %d/%b/%Y** - this being into **Date Format Options**
section.
### Log format
- **log-format**: prefer the **COMBINED** format — *for httpd, you can
choose the **common** format, too*.
*If you use **nginx**, it is imperative to register all statuses,
not to filter the 2xx and 3xx — if you had created the **map** directive
you will have at least to comment it and restart the web service*.
### File Options
If you have only one web domaine on the server to analyze, it's usefull
to set the **log-file** option with the absolute directory of the **access.log**
relient.
### Parse Options
- **exclude-ip**: use this option to exclude adresses IP or network segments,
*like you personal network at home*
- **444-as-404**: the **444** error is nginx's specific.
If you want to analyze as an **404** error, change to **true**.
It's up to you!
- **ignore-crawlers**: to ignore all crawlers robots; change to **true**
- **ignore-panel** is one panel that can be disabled. On Europe, with the
RGPD, it is better to disable, at least, the **REMOTE_USER** panel.
- **anonymize-ip** : change to **true** - *egual, because RGPD, disable!*
### Persistence Options
- **db-path**: the absolute directory of the goaccess database — *configure
only one domain.*
- **persist**: to backup the analyzed datas.
- **restore**: load the data to be visualized from the saved data.
⇒ Enable both last options to **true**, if you want to save the data into
the goaccess database.
---
Of course, there are numerous options.
After, {{< anchor "just use" "basic use" >}} goaccess.
---
Now, let's push the configuration a further:
### User system
I prefer to create and use a dedicated user system. Among the advantages
are the search in log messages and even cron.
```sh
# useradd -s /sbin/nologin -d /var/db/goaccess _goaccess
```
---
⇒ to search in log messages:
`$ grep goaccess /var/log/messages` or
`$ doas grep _goaccess /var/cron/log`
#### doas
⇒ I preferred to add the permission to use the **goaccess** binary into
`/etc/doas.conf`:
```cfg
permit nopass _goaccess cmd /usr/local/bin/goaccess
```
*(I'm not absolutly sure about the necessity)*.
#### database directory
⇒ Go to create the main directory for the goaccess database:
```sh
# mkdir -p /var/db/goaccess
# chown _goaccess:daemon /var/db/goaccess
```
*(personally, I preferred another absolute folder)*.
**Think to add at yours backups!**
---
Then if like me, you have several web domains on your server, create
many subdirectories with the name of the web domain, such as:
```sh
$ domain=
$ doas -u _goaccess mkdir "/var/db/goaccess/${domain}"
```
Thus, future statistics will be really dedicated to a domain.
#### crontab
⇒ You need to set cron rules:
```sh
$ doas -u _goaccess crontab -e
```
*(Egual, it's possible to do by: `doas crontab -u _goaccess -e`)*.
Add:
```cfg
*/15 * * * * -ns goaccess -a --db-path "/var/db/goaccess/domain/" -f /var/www/logs/domain/access.log -o /var/www/goaccess/domain/stats.html
```
Littles explainations:
- one crontab to execute every 15 minutes. *Keep on mind, it's an example.*
- replace the *domain* string by the domain name.
### Authenticate
It's up to you if you want web authentication before viewing; some people
think you should, others don't; personally I prefer it.
After using **htpasswd**, you need to configure on the server directive
on the virtual host:
- for http, use **authenticate**.
- on nginx, both **auth_basic** and **auth_basic_user_file** directives.
#### httpd configuration
Basically:
```httpd
server "domain.tld" {
(…)
root "/htdocs/domain.tld/www"
location "/stats" {
authenticate with "/file_htpwd"
directory auto index
}
(…)
}
```
**Do not forget: the path of the htpasswd file is relative to the web chroot!**
#### nginx configuration
```nginx
(…)
location /stats/ {
auth_basic "Auth Area";
auth_basic_user_file /file_htpwd;
autoindex on;
}
(…)
```
---
Voila for the "configuration" part!
---
## Use
### Basic use
Execute the binary, as:
```sh
$ goaccess -o /var/www/htdocs/domain.tld/stats/index.html
```
### use _goaccess
⇒ Use goaccess with the **_goaccess** user:
For instance, the first time, to precise datation with month and year:
```sh
$ domain=
$ date="$(date +'%Y-%m')"
$ doas -u _goaccess goaccess -a --db-path "/var/db/goaccess/${domain}/" -f "/var/www/logs/${domain}/access.log" -o "/var/db/goaccess/${domain}/stats-${domain}-${date}.html"
```
---
I created a shell script, named **goaccess.sh**:
```file goaccess.sh
#!/bin/sh
###
#
# manage statistics by domain
#
##
date="$(date +'%m-%Y')"
dir_db="/var/db/goaccess"
domain="$1"
if [ -z "${domain}" ]; then
printf '%s %s\n' "KO" "No domain. Script stops!"
logger "$0: no domain found as option; script stops!"
exit 1
fi
if [ ! -d "${dir_db}/${domain}/" ]; then
printf '%s %s\n' "KO" "The destination directory '${dir_db}/${domain}/' seems not exist!"
logger "$0: The destination directory for goaccess not exists; script stops!"
exit 2
fi
goaccess -a --db-path "${dir_db}/${domain}/" -f "/var/www/logs/${domain}/access.log" -o "${dir_db}/${domain}/stats-${domain}-${date}.html"
```
And, after setting up the **_goaccess** crontab:
```cfg
*/15 * * * * -ns /directory/goaccess.sh domain-x.tld
0 * * * * -ns /directory/goaccess.sh domain-y.tld
0 0 * * * -ns /directory/goaccess.sh domain-z.tld
```
*(at differents times)*.
---
No! It's not finish…
In fact, the generated HTML file to dataviz is wrote on the database directory.
The **_goaccess** user cant write on the web directory, and does not have
access.
On the other hand, it is possible to ask the web user **www** to copy
the HTML statistics file on the corresponding web directory.
Here, I use this following shell script, named **cp_stats.sh**:
```file cp_stats.sh
#!/bin/sh
set -e
#set -x
###
#
# copy statistics files into the domain web directory
#
##
date="$(date +'%m-%Y')"
domain="$1"
dir_db="/var/db/goaccess"
dir_stats="/var/www/htdocs/${domain}/www/stats/"
if [ -z "${domain}" ]; then
printf '%s %s\n' "KO" "No domain. Script stops!"
logger "$0: no domain found as option; script stops!"
exit 1
fi
if [ ! -f "${dir_db}/${domain}/stats-${domain}-${date}.html" ]; then
printf '%s %s\n' "KO" "The needed file '${dir_db}/${domain}/stats-${domain}-${date}.html' not exists. Script stops!"
logger "$0: The needed file '${dir_db}/${domain}/stats-${domain}-${date}.html' not exists; script stops!"
exit 2
fi
if [ ! -d "${dir_stats}" ]; then mkdir -p "${dir_stats}"; fi
cp "${dir_db}/${domain}/stats-${domain}-${date}.html" "${dir_stats}"
chown -R www "${dir_stats}"
```
You need to modify the crontab of the web user:
```sh
$ doas -u www crontab -e
```
Like, for instance:
```cfg
*/15 * * * * -ns /directory/cp_stats.sh domain-x.tld
5 * * * * -ns /directory/cp_stats.sh domain-y.tld
5 0 * * * -ns /directory/cp_stats.sh domain-z.tld
```
### Time-real
The real time use is done in two possible ways.
Into those contexts, we will not need the **_goaccess** user.
#### Terminal
⇒ at least, tape:
```sh
$ goaccess -f /var/www/logs/${domain}/access.log
```
Enjoy the aesthetic view based on monokai colors, by default.
#### Web proxy
Here, it's really interesting, but a "bit complicated":
##### relay httpd
I have not found solution for the httpd service. Maybe, with relayd ;-)
##### proxy nginx
Lets to set the server configuration to add the **location** directive:
```nginx
location /ws {
proxy_pass http://localhost:7890;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
```
##### websocket
Now, you need to execute goaccess with the **www** web user rights:
```sh
$ doas -u www goaccess -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html --ws-url=wss://huc.fr.eu.org:443/ws --port 7890
[PARSING /var/www/logs/doc.huc.fr.eu.org/access.log] {0} @ {0/s}
WebSocket server ready to accept new client connections
```
Note: for this test, I have to:
- create a new configuration file named **goaccess.realtime.conf**
- to create a new HTML file
- and to listen the server on the secure websocket.
Also, we can monitoring, in real time, the user, into a SSH console, with
binaries like **fstat**, **ps**; see:
```sh
$ fstat -u www -n
USER CMD PID FD DEV INUM MODE R/W SZ|DV
www goaccess 76729 wd 4,15 725760 40755 r 512
www goaccess 76729 0 4,0 78329 20620 rw 5,0
www goaccess 76729 1 4,0 78329 20620 rw 5,0
www goaccess 76729 2 4,0 78329 20620 rw 5,0
www goaccess 76729 3 4,3 175 10644 rw 0
www goaccess 76729 4 4,3 176 10644 rw 0
www goaccess 76729 5 4,3 175 10644 w 0
www goaccess 76729 6 4,3 176 10644 w 0
www goaccess 76729 7 pipe 0x0 state:
www goaccess 76729 8 pipe 0x0 state:
www goaccess 76729 9* internet stream tcp 0x0 *:7890
www goaccess 76729 10 pipe 0x0 state:
www goaccess 76729 11 pipe 0x0 state:
www goaccess 76729 12* internet stream tcp 0x0 127.0.0.1:7890 <-- 127.0.0.1:6459
$ ps aux -U www
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
www 4071 0.0 0.1 1572 3488 ?? S 10:12AM 0:00.88 sshd: www@notty (sshd)
www 76729 0.0 0.3 10096 12752 p0 S+ 7:56PM 0:01.49 goaccess -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html /var/www/logs/do
```
Now, go to view the **realtime.html** file on your web browser.
{{< img a="dashboard goaccess real time" s="monitor/goaccess-realtime.png" w="250" >}}
Note the little dash below the parameter icone, at the top left on the
screen.
*It seems necessary to reload the page; after some time, it lose the connector.
Press F5…*
---
Finally, we cant execute goaccess as service, either by the **--daemonize**
option, or by configuring the dedicated configuration file.
```sh
$ doas -u www goaccess --daemonize -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html --ws-url=wss://huc.fr.eu.org:443/ws --port 7890
Daemonized GoAccess: 48646
```
This option **only works with real time option** enabled.
Egual, it's **imperative that the web user is given access to the path that
will write the PID process file** — *otherwise, you will fail!*
You need to set the **pid-file** option into the configuration file.
As reminder, on OpenBSD, by default, it's on the web chroot: `/var/www/run`.
Prefer to create a sub-directory dedicated to the user **www**.
---
Of course, all of this, it's for the FUN, and the example! :D
---
Voila!
*(It's my XP… and yours‽)*
## Troubleshooting
Here are some errors encountered:
### Permission denied
- `Couldn't open file /var/db/goaccess/xxx/I32_DATES.db: Permission denied`
- `Unable to open the specified pid file. Permission denied`
- `Unable to open the specified pid file. Permission denied`
1. goaccess can not write into the directory!
2. check that directory exists.
3. check rights user; they must match the one of the user who runs
goaccess, like : `_goaccess:daemon`
This is the same problem during the generation of HTML files.
Example:
```sh
GoAccess - version 1.5.1 - Sep 26 2021 14:08:19
Config file: /etc/goaccess/goaccess.conf
Fatal error has occurred
Error occurred at: src/output.c - output_html - 1183
Unable to open HTML file: Permission denied.
```
### Error opening the specified MaxMind DB file
```sh
GoAccess - version 1.5.5 - Apr 8 2022 09:03:43
Config file: /etc/goaccess/goaccess.conf
Fatal error has occurred
Error occurred at: src/geoip2.c - init_geoip - 89
Unable to open GeoIP2 database /var/db/GeoIP/GeoLite2-Country.mmdb: Error opening the specified MaxMind DB file
```
You have certainly activate the `geoip-database` option.
But did you download the necessary files and install them in the directory
`/var/db/GeoIP/`?
### No home directory
- **Make sure that the home directory for the dedicated user really exists !**
Then check the existence of the path in your filesystem, without forgetting
the user rights on!
- make sure that the home directory of the dedicated user AND the `dir_db`
variable on the `goaccess.sh` script matches.
## Documentations
- https://www.geeek.org/goaccess-analyser-access-log/
### Wikipédia
- {{< wp2 n="List_of_HTTP_status_codes" a="4xx_client_errors" l="en" >}}
---
***Enjoy-ID!***
***Enjoy-IT!***