31st December 2024, 148 min read

Website Analytics

I had written on website analytics with regard to this blog before:

I do not use cookies or any JavaScript libraries to track users. Instead I only analyze the web-server logs using a Perl script. To doublecheck the accuracy of this Perl script I occasionally inserted analytics code from

These analytics provider used cookies, and JavaScript. Therefore my site was not cookie-free, when I employed them.

Nowadays I no longer employ those bulky JavaScript libraries. I resort to the web-server logfile. There are a number of advantages and disadvantages.

Advantages:

The user does not have to be concerned about cookies
The user does not need to download bulky JavaScript libraries
The user does not need to make yet another connection to any other server

Disadvantages:

The detection of bots and nonsense access is a little more cumbersome; there seems to be no ready-to-use software to filter out all the bots, so I had to write it myself, see accesslogFilter
There seems to be no off-the-shelf software to fully analyze weblogs and generate diagrams, so I had to write it myself, see blogurlcnt

Below are the statistics for the year 2024. Some key figures using filtered data.

Ca. 18,000 "real" accesses as can be seen by looking at pagefind-ui.js, i.e., someone not loading this JavaScript is in many cases not a real reader
On average there are ca. 1,000 monthly accesses for "real" posts
Most intensive access is HetrixTools
Best post was Hosting Static Content with GitLab, which had more than 4,500 views in July 2024, and more than 1,300 views in August
Second best was Performance Comparison C vs. Java vs. Javascript vs. LuaJIT vs. PyPy vs. PHP vs. Python vs. Perl having 100-200 views per month

1. URL statistics

Below is the output of the Perl script blogurlcnt.

The generated output uses JavaScript DataTables. DataTables allow easy filtering and sorting within the table.

Combine that with Apache ECharts to show histograms to get a visual representation of the development of various URLs over time.

Year	Month	Week

Above table and charts where generated with below command:

time blogconcatlog 67 | pv | tee /tmp/a2 | accesslogFilter -o >(tee /tmp/a1.stat | blogstatcnt -m3000 > /srv/http/statcnt.html) | tee /tmp/a1 | blogurlcnt -m470 > /srv/http/urlstat2-m100.html

I.e., the "uncount" statistics are dropped if they count less than 3,000 entries. Likewise the statistics and charts are only shown for entries, which have more than 470 views.

2. Uncount statistics

The "uncount" statistics is called "uncounted", as these number were not counted in above statistics, i.e., they were filtered out. This filtering was done using accesslogFilter.

A key issue with analyzing the raw web-server log is to filter out all irrelevant entries. Examples of irrelevant entries are:

My own access to my blog on my own local network, i.e., on 127.0.0.1 or localhost
Various bots from Google, Ahrefs, Yahoo, etc.
HTTP codes 404 (not found), etc.
IP address ranges from webhosting platforms or known AI platforms

Below table shows what filters are most effective. Some noteworthy highlights:

There are more than 9-times more irrelevant accesses than good ones, as given by column %good in first row with "Sum total". Or phrased differently: only roughly 10% of all accesses are relevant.
The major factor of all HTTP codes, which are irrelevant, is 404 (not found)
The most active bots, which can only be identified by IP address range, are in the class B subnet 162.55, which is Hetzner's your-server.de; these bots are more intense than all HTTP error codes combined
Ahrefs is the most frequent visitor of all bots (ca. 21%), way before Google bot (4.7%); in line with observations from last year: Website Checking with Ahrefs
It is interesting that Yandex is searching the blog more frequently than Bing