It was also disappointing that --ignore-crawlers of GoAccess was still letting through a lot of bot requests, so from the knowledge that I gathered from looking through access.log, I decided I will let it do its thing — the pretty charts and stats, but filter the log records myself.

To do the filtering I needed a list of known bot user-agent names, and when I couldn’t google it, I decided I will extract my own bot list from the some 11 months of logs I had from FeedSubscription.com. I’m posting the list below, just for reference.

Here is the UNIX pipeline that I used to extract it:

base_re='\w*(bot|crawler|spider)\w*'

zcat -f /var/log/access.log* | # look into compressed and uncompressed logs
grep -Eoi '" [0-9]+ [0-9]+ ".*'"$base_re" | # only lines with something-BOT-something in the UA string (or referrer)
grep -Eoi "$base_re" |

Essentially, I look for words that contain “bot” or “crawler” in the user-agent string. Besides that, I found some UA strings that didn’t match, but I recognized the names, and added included them in the list:

Chrome-Lighthouse
Google-InspectionTool
HeadlessChrome
scaninfo@paloaltonetworks.com
Feedly
Go-http-client
Nmap Scripting Engine
facebookexternalhit
facebookcatalog

Combining these two lists, I get stats that are much closer to what I see in the built-in tracking.

Here is how I use this list to filter out bot requests:

bot_list_re="($(cat bot-list.txt | paste -sd '|'))"

zcat -f /var/log/access.log* |
grep -vPi ".*$bot_list_re.*"

…which is essentially combining them all in a large regex like (name1|name2|name3|name4).

bot-list.txt

AdsBot
AhrefsBot
aiHitBot
Applebot
bingbot
BitSightBot
bot
bots
BSbot
CCBot
Chrome-Lighthouse
crawler
DataForSeoBot
domainsbot
DotBot
Exabot
facebookcatalog
facebookexternalhit
Facebot
Feedly
Go-http-client
Google-InspectionTool
Googlebot
GulperBot
HeadlessChrome
LinkedInBot
MJ12bot
MojeekBot
msnbot
Nicecrawler
Nmap Scripting Engine
org_bot
Pinterestbot
QBOT
redditbot
RepoLookoutBot
robot
robots
RU_Bot
scaninfo@paloaltonetworks.com
SemrushBot
serpstatbot
SeznamBot
Slackbot
SuperBot
t3versionsBot
TelegramBot
top1mbot
Twitterbot
VelenPublicWebCrawler
Vercelbot
webprosbot
WhatStuffWhereBot
YandexBot
YandexRenderResourcesBot
ZaldamoSearchBot
ZoominfoBot

I have started many times in the last ten years or so, and then it faded away only to start again as heard about its benefits in a podcast, article, or book.

My reasons

My reasons changed over time, and now I’ve settled on its more practical aspects: to be able to observe the state of affairs in my mind, and intervene before it gets too bad. Another reason, which has some overlap with the first one is to be able to keep my emotional balance in difficult conversations. I think I can check off the first one, but I’ve only got partial success on the second — working on it!

My how

Here are some of the more technical aspects of my practice:

• Frequency: I try to do it every day, but I skip a day every now and then, with the most success in the last year or so. I’ve learned to not make an issue about skipping a day or three.
• Time of day: I try to do it in the morning, before starting to work, but when the morning is busy, I do it in the afternoon.
• Duration: In the very beginning started real small, with 2–3 minutes, then 5, then 10, then 15, then gradually grew it to 25 in the last 3 weeks.
• How: I sit comfortably on a chair, spine vertical as if trying to reach the ceiling, breathe in deeply first in the belly and then into the lugs, breathe out in reverse order. Focus on the sensation of breathing.
• Succes criteria:
  • Mind feels refreshed, feet and palms warm up.
  • Breathing get automatic and rhythmic after a few minutes.
  • Forehead is relaxed.
  • Time flies.
• Side effects:
  • I can get rid of most headaches.
  • They say that breathing into the belly gives a nice massage to the internal organs.
• Commitment device: The Seinfeld Strategy.

My inspiration

My initial inspiration probably came from the Zen Habits blog, and the most recent is Nick Wignal’s blog.

I will only recommend one article: How to Start a Mindfulness Practice.

On my side project, I’ve made a monitoring system based largely on tail, grep, jq, and ssmtp, all weaved together into a UNIX pipeline and packaged as a make task. It’s started from the crontab as system boot, and sends me an email every time a notable event is recorded to the log.

Here is the pipeline:

tail -n0 --follow=name --retry logs/feedsubscription/{app,api}.log |
grep --line-buffered -E \
        -e '"severity":"(error|warning)"' \
        -e '"message":"Sending report"' \
    |
while read -r _skip_timestamp _skip_namespace _skip_app json; do
    (
        echo "Subject: RES App $(jq -r .severity <<<"$json")"
        echo "From: watch-app@feedsubscription.com"
        echo
        jq . <<<"$json"
    ) |
    if [ -t 1 ]; then cat; else ssmtp gurdiga@gmail.com; fi;
done

Here are the interesting bits:

1. Starts with tailing the appropriate logs. I use -n0 to only look at the new lines as they are added, skip the existing ones. The --follow=name --retry is to allow tail to work smoothly with log rotation.

2. I filter the lines with grep to only get the ones that interest me. I use --line-buffered to prevent block buffering: when grep delays the output until it collects a bunch of lines. The -E is to use some basic regexp magic. A few -e args to define the filter conditions, for example -e '"severity":"(error|warning)"' catches any error or warning (my logs are in JSON format). I add more as I need them.

3. I collect the logs from all of the system’s containers to a syslog syslog-ng container, and so I end up with some additional meta-data fields for every line. This is why I use read to split the line into fields, and only look at the actual JSON log message. The -r flag is to prevent read from interpreting backslashes in the JSON string as shell escapes. I throw away the meta-data fields because I don’t need them here: _skip_timestamp _skip_namespace _skip_app.

4. I then use jq to extract some bits from the JSON log line, and build an email. For example, I extract the severity field and put it in the email subject, and jus dump the entire JSON line, prettified with jq, into the email body. Here is an example:

    {
      "severity": "info",
      "message": "Sending report",
      "data": {
        "module": "email-sending",
        "feedId": "dexonline",
        "report": {
          "sentExpected": 3,
          "sent": 3,
          "failed": 0
        }
      }
    }
   

5. The lil if at the end is to help me with debugging: when I run the pipeline manually from the command line, just print the email to console with cat instead of sending it out with ssmtp.

Happy making and shell pipelining! 🙂

My go-to advice in these situations is to start building a copy of something that is conceivably useful. This is how I started and grew as a developer, and it seems to have worked quite well.

I like this approach because it keeps me focused on a finite user value. I came to appreciate specifically the “finite” aspect of it because it keeps the work contained and prevents scope creep.

One other reason why I find this approach practical is that it allowed me to touch multiple layers of the software stack: in my case it was an e-commerce website, so I got to dip my toes in:

• web frontend: HTML, CSS, and JS, HTTP parts relevant to the frontend like caching and asset optimisation, performance on the frontend;
• backend: Apache, PHP, MySQL, email, image processing, performance on the backend.
• devops: although at that time it meant SSH, Makefiles, rsync and coreutils, it still exposed enough of the deployment activities to get some understanding of what’s involved there.

Besides giving a wide perspective of what an app is made of, this is also an opportunity to pick one layer to specialise in, both of which I think are important for a beginner which tries to find their way in software development.

TLDR: It was checking if there are any new messages with “DMARC Reports” label in my GMail, and if yes, then would get the attached file and put it in the “DMARC Reports” folder in my GDrive. Then send me an email with a textual report about what happened. Pretty neat! 😎

// GmailApp: https://developers.google.com/apps-script/reference/gmail/gmail-app
// DriveApp: https://developers.google.com/apps-script/reference/drive/drive-app

const labelName = "DMARC reports";
const folderName = "DMARC reports";

function importReports() {
  const label = GmailApp.getUserLabelByName(labelName);

  if (label.getUnreadCount() === 0) {
    return;
  }

  let report = "";

  const say = (string) => {
    report += `\n${string}`;
    Logger.log(string);
  };

  say(`Found ${label.getUnreadCount()} new messages`);

  const unreadThreads = label.getThreads().filter((t) => t.isUnread());
  const folder = DriveApp.getFoldersByName(folderName).next();

  unreadThreads.forEach((thread) => {
    const attachment = thread.getMessages()[0].getAttachments()[0];
    const file = DriveApp.createFile(attachment.getAllBlobs()[0]);

    folder.addFile(file);
    thread.markRead();

    say(attachment.getName());
  });

  GmailApp.sendEmail("gurdiga@gmail.com", "New DMARC reports", report);
}

I guess one could do pretty advanced stuff with this if they use multiple Google services and need to juggle things between them. At the very least it can be another useful tool.

grep "^`date +%F`" logs/feedsubscription/api.log \
| grep '"message":"subscribe"' \
| ifne mail -s "Subscription requests" root

This will grep the API log, and email me today’s subscription requests, but only if there are any. It won’t do anything if there are no records found — and that’s exactly its whole purpose: only run the command if anything comes into STDIN.

Its limitation is that I can only run one command on that STDIN, and I wanted to do some more processing on it before passing it to the next command in the pipeline. Specifically this: grep some log file and email the matching lines, but first prepend some headers, which would look like this:

grep 'something' file \
| (
    echo "Subject: Subscribe report"
    echo "From: subscribe-report@feedsubscription.com"
    echo ""
    cat
) \
| ssmtp gurdiga@gmail.com

This doesn’t work as I wanted because it would send the email even if no lines matched. So I thought: if I pack the pre-processing and the sending in a send_report function, I’d be able to call that function with ifne only when there are matching lines, something like this:

grep 'something' file | ifne send_report
send_report: No such file or directory

…it seems like it doesn’t work with functions. After some googling, I found this approach:

export -f send_report
grep 'something' file | ifne bash -c send_report

It’s a bit awkward, but still readable and does what it needs to do.

The Docker composition on one of my side projects has recently got to 6 containers: SMTP-in, SMTP-out, app, subscription, website, and certbot. Since I have deployed a 0.1.0 version to DigitalOcean a couple of weeks ago, I caught myself feeling increasingly anxious about losing logs every time I deployed a new version of the container.

Every container is streaming its logs to STDOUT. I wanted to have a persistent log file for each container, which Docker almost does out-of-the-box, with the “little” caveat that those files are removed every time the container is rebuilt. 🙃

One of the things that I’ve challenged myself to do on this side project is to have it as self-contained as possible. — This is why I’ve set up my own SMTP server instead of buying a transactional email subscription. 😎

This is the kind of simplicity I was referring to when I said “simple log aggregation” in the title: no third-party services. I wanted it to be a Docker composition that I can start on my laptop as easily and as cleanly as on a VM in some cloud.

At the end of the day, all I needed is to have those logs — plain text or JSON — stored somewhere so that I can grep or jq them whenever I want to do see what happened when.

After googling around for a few mornings, I ended up deploying an additional logger container that runs Syslog and then instructed all the other containers to stream their logs into it. I’ve seen this called “the sidecar pattern” in other places. The logger container’s files live in a mounded directory, so they can have a life outside Docker, and across the container life cycle.

Some dirty details

The logger

I’ve started with the mumblepins/syslog-ng-alpine for the proof of concept, but then I saw a warning about old log format, or something. On a closer inspection, I realized that it hasn’t been updated in 4 years. Hm… Looking at its Dockerfile on GitHub I found that it was manually compiling Syslog from the source. Umm… OK. On the next day, inspired by that I’ve created my own Dockerfile, which used Alpine’s built-in package of Syslog.

Extracted a copy of its config file from the running container, added the bits from mumblepins that made it listen on the network, and with that I had it both up to date and working. Open Source is awesome. 😎

The docker-compose config

To tell a container to stream its logs to a Syslog is well documented, and quite straightforward:

app:
  # ...more settings...
  depends_on: [logger]
  logging:
    driver: syslog
    options:
      syslog-address: tcp://127.0.0.1:514
      syslog-format: rfc3164
      tag: subscription

This worked, but having this for 6 services looked too non-DRY for my taste, so I googled around for a morning or two and found about YAML merge type which is described under the Extension fields section of Docker Compose references, and which gave me this:

x-logging: &logging
  depends_on: [logger]
  logging:
    driver: syslog
    options:
      syslog-address: tcp://127.0.0.1:514
      syslog-format: rfc3164
      tag: "{​{.Name}}"

I added the tag: option to have the symbolic container name in logs instead of its hex ID. And so this snippet now allows me to include it in every service’s config like this:

smtp-in:
  container_name: smtp-in
  # ...
  <<: *logging

This is neat. 🤓

So, in no particular order, here they come:

Programming by intention

It’s a concept I’ve seen recently applied by James Shore and the TLDR version of it is this: I write some pseudo-code that reflects how would I like my code to look like given my current understanding. The good thing about it is that it guides me towards putting together the API of the code, and get some understanding of the pieces involved, with the smallest effort investment possible — it’s just a comment.

Here is a quick pseudo-example:

function main() {
  /*
  const dataDir = getDataDirFromFirstCliArg();
  const feedUrl = getFeedUrl(dataDir);
  const lastItemTimestamp = getLastItemTimestamp(dataDir);
  const rssItems = getRssItems(feedUrl);
  const newItems = getNewItems(rssItems, lastItemTimestamp);
  storeNewItems(newItems);
  */
}

I can mess with it until I like how it reads, and then, when I have some clarity about what’s supposed to do, I can take each of those pieces and TDD it.

I think this approach has something in common with Readme Driven Development.

“Logic sandwich”

This is a concept I’ve learned — again — from James Shore. Since this RSS-checking program would involve a significant portion of IO, I found that using the “logic sandwich” approach helped me keep the IO and business logic unentangled, which in turn, made more of the code TDD-able.

Conceptually, it means that the code is structured in code chunks that do either IO or business logic, but not both. Something like this:

function main() {
  const input = infrastructure.readData();
  const output = logic.processInput(input);

  infrastructure.writeData(output);
}

Microtest TDD

It’s a concept I’ve heard a few months ago from GeePaw Hill, and it’s a particular style of TDD where the emphasis is on a couple of things:

• unit test size leans on the smaller side, the smaller the better: as in “micro.” Yeah, I know: that’s just Unit Testing 101, right? 😉
• unit tests are “gray-box” tests (as opposed to “black-box” tests) — they know some of the things that happen in the code under test, which helps in keeping them more practical.

Functional Core, Imperative Shell

It’s a concept I’ve heard years ago from Gary Bernhardt and means that the program “tree” is structured in a way that the “leaf” parts are written in a functional style, and the parts closer to the “trunk” are written in an imperative style.

In this particular program, the imperative part is the main function that acts like a controller that wires all the more functional pieces together. I found that the code written in a more functional style is easier to TDD.

Result type

This is a concept I saw used in typed functional languages like Haskell, Elm, and F#. FWIW, it even has its own Wikipedia page. 🙂

I find it brings clarity to the program flow by clearly delineating the failure paths, and, in a well-typed language like TypeScript, this makes for pretty readable and resilient code.

Try easy!

Last but not least, it’s a meta-idea that I got from the “Accidental Genius” book: it’s called “Try easy!”. Its essence — seemingly counterintuitive at first — is that you can actually achieve more when you’re not trying to give 105% of yourself to an activity, but on the contrary, to relax a bit, maybe to 90%.

That way you’re more flexible and keep the ability to work around the obstacles that inevitable come up with every project. Yes, this is all metaphorical and meta, but I still found it valuable. On this project specifically, I actually started twice: once with a straight face, saying to myself “I’m going to TDD this to death!” and then gave up on this idea after a week of struggling because this attitude of perfectionism was making me feel crippled.

Then, after few days of sadness and disappointment, I’ve changed my posture to “try easy” and said something to the effect of “Let’s see if I can do just this one little thingy over here.” Then, another “little thingy,” and then another.

Happy coding!

1. Use some background sound

I found it’s easier to stay focused when I put on some background sound. 2 that worked well for me lately were RainyMood and Abraham Hicks’ guided meditation music, with the words stripped out, but any soothing sounds should work. I find it’s especially useful when it guides towards a breathing rhythm of 5–6 breaths per minute.

2. Push on the exhale

Usually, the inhale is the intentional part when breathing, and the exhale is somewhat automatic, and, for that reason, somewhat shallow. Going through a couple of books on breathing (1, 2) lately, one recurring theme I found was the focus on the exhale part of the breathing. For me, because it’s somewhat an unusual way to breathe, it helps me keep my attention focused.

3. Breath “horizontally”

“Horizontal” breathing is a term I found in the second book, and it means to breathe in and out of your belly while keeping your shoulders still. Besides some health benefits for the internal organs, it helps me focus.

The measure of success

I’m guessing everyone has their own measure of success when it comes to breathing and meditation. To me, it’s when I can feel the forehead part of my scalp relaxing. I found that during focused work I have it tensed for the most part, and so releasing that tension is what I’m looking for.

Enjoy!

The boky/postfix Docker image turned out to be quite useful, especially because it’s set up with a mechanism for generating the DKIM keys.

So here are the two steps to sending emails from a custom domain:

1. Postfix setup

$ docker run --rm --name postfix \
    -v `pwd`/opendkim-keys:/etc/opendkim/keys \
    -e "ALLOWED_SENDER_DOMAINS=feedsubscription.com" \
    -e "DKIM_AUTOGENERATE=yes" \
    -e "INBOUND_DEBUGGING=yes" \
    --no-healthcheck \
    -p 1587:587 \
    boky/postfix

One more thing about the Postfix container: if you want to persist the queue between container restarts/rebuilds, pass in a volume for /var/spool/postfix, for example:

-v `pwd`/postfix:/var/spool/postfix

2. DNS setup: SPF, DKIM, and DMARC

We need to add 3 TXT records in the feedsubscription.com domain:

¹ This particular SPF value assumes we already have a proper MX record.

² The complete definition for the DKIM record is generated by the container because of DKIM_AUTOGENERATE=yes, and in this particular setup can be found in ./opendkim-keys/feedsubscription.com.txt.

3. Testing

I used ssmtp to test the setup. I needed to have this in /etc/ssmtp/ssmtp.conf:

mailhub=localhost:1587

…then I can send an email:

$ ssmtp -vvv gurdiga@gmail.com < email.txt

Check logs:

$ docker logs -f <container_name>

I specifically enabled logging in above Postfix setup with INBOUND_DEBUGGING=yes so that I can see what’s happening.

A lil closing note

Although I am a little confused about the value of the DMARC record, without it the emails land in the spam folder.

Happy email sending!

Update Aug 29

Because last week we had a phishing incident at work, I googled some more about the mechanics of SPF, DKIM, and DMARC.

When an SMTP server receives an email from a domain, it can use SFP to verify that the sending server is authorized to send emails for that domain. For example, if I bring up a Postfix container on my laptop, and send an email from info@feedsubscription.com, the receiving SMTP server can check with the DNS for feedsubscription.com, get the SPF record for and verify that my laptop is in the list. If it’s not in the list, it can mark the message as spam or reject it altogether.

DKIM is just an additional level of authentication based on public-key cryptography. It adds a header to the email message — DKIM-Signature — which can be verified by the receiving SMTP server by checking the corresponding DKIM record in the feedsubscription.com DNS.

DMARC builds on both SPF and DKIM, and allows for more sophisticated policies. The receiving SMTP server can look at the DMARC record and decide what to do with every message. And this is why having DMARC — and also SFP and DKIM — gives the sender a higher level of confidence a message will reach its recipient …because the receiver can be more confident.

Here are a couple of links that I found informative:

• About SPF
• SPF record structure
• About DKIM
• About DMARC
• DMARC record structure
• More DMARC record structure