hpc.social


High Performance Computing
Practitioners
and friends /#hpc
05.04.2024 · 155 mins read
Share: 
This is a crosspost from   Blogs on Technical Computing Goulash Recent content in Blogs on Technical Computing Goulash. See the original post here.

Centralized system and LSF logging on a Turing Pi system

Logs are one of those indispensable things in IT when things go wrong. Having worked in technical support for software products in a past life, I’ve likely looked at hundreds (or more) logs over the years, helping to identify issues. So, I really appreciate the importance of logs, but I can honestly say that I never really thought about a logging strategy for the systems on my home network - primarily those running Linux.

One of my longtime friends, Peter Czanik, who also works in IT, happens to be a logging guru as well as an IBM Champion for Power Systems (yeah!). So it’s only natural that we get to talking about logging. He is often complaining that even at IT security conferences people are unaware of the importance of central logging. So, why is it so important? For security it’s obvious: logs are stored independently from the compromised system, so they cannot be modified or deleted by the attacker. But central logging is beneficial for the HPC operator as well. First of all, it’s availability. You can read the logs even if one of your nodes becomes unreachable. Instead of trying to breath life into the failed node, you can just take a look at the logs and see a broken hard drive, or a similar deadly problem. And it is also convenience, as all logs are available at a single location. Logging into each node on the 3 node cluster to check locally saved logs is inconvenient but doable. On a 10 node cluster it takes a long time. On a 100 node cluster a couple of working days. While, if your logs are collected to a central location, maybe a single grep command, or search in a Kibana or similar web interface.

Those who follow my blog will know that I’ve been tinkering with a Turing Pi V1 system lately. You can read my latest post here. For me, the Turing Pi has always been a cluster in a box. My Turing Pi is fully populated with 7 compute modules. I’ve designed Node 1 to be the NFS server and LSF manager for the cluster. LSF is a workload scheduler for high-performance computing (HPC) from IBM. Naturally I turned to Peter for his guidance on this, and the result is this blog. Peter recommended that I use syslog-ng for log aggregation and also helped me through some of my first steps with syslog-ng. And the goal was to aggregate both the system (syslog) as well as LSF logs on Node 1. TL;DR it was easy to get it all working. But I encourage you to read on to better understand the nuances and necessary configuration both syslog-ng and LSF that was needed.

The environment

The following software has been deployed on the Turing Pi:

The Turing Pi system is configured as follows:

Node 1 (turingpi) is the manager node of this cluster in a box and has by far the most storage. Naturally we want to use that as the centralized logging server.

Node Hostname Hardware Notes
1 turingpi CM3+ LSF manager, NFS server, 128GB SDcard
2 kemeny CM3 4GB eMMC flash
3 neumann CM3+ 8GB SDcard
4 szilard CM3+ 8GB SDcard
5 teller CM3+ 8GB SDcard
6 vonkarman CM3+ 8GB SDcard
7 wigner CM3+ 8GB SDcard

Syslog-ng & LSF setup

  1. Raspberry Pi OS configures rsyslog out of the box. The first step is to install syslog-ng on Node 1 in the environment. Note that installing syslog-ng automatically disables rsyslog on the nodes.

Output of apt update; apt-get install syslog-ng -y. Click to expand 
root@turingpi:~# apt update; apt-get install syslog-ng -y 
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Hit:2 http://deb.debian.org/debian bullseye InRelease                                                        
Hit:3 http://deb.debian.org/debian bullseye-updates InRelease                                                
Hit:4 https://repos.influxdata.com/debian stable InRelease                                                   
Hit:5 https://repos.influxdata.com/debian bullseye InRelease                                                 
Hit:6 http://archive.raspberrypi.org/debian bullseye InRelease                                  
Hit:7 https://packagecloud.io/ookla/speedtest-cli/debian bullseye InRelease                     
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbson-1.0-0 libdbi1 libesmtp6 libhiredis0.14 libivykis0 libmaxminddb0 libmongoc-1.0-0 libmongocrypt0
  libnet1 libprotobuf-c1 librabbitmq4 librdkafka1 libriemann-client0 libsnappy1v5 libsnmp-base libsnmp40
  syslog-ng-core syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-examples
  syslog-ng-mod-extra syslog-ng-mod-geoip2 syslog-ng-mod-getent syslog-ng-mod-graphite syslog-ng-mod-http
  syslog-ng-mod-map-value-pairs syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-rdkafka
  syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-slog syslog-ng-mod-smtp syslog-ng-mod-snmp
  syslog-ng-mod-sql syslog-ng-mod-stardate syslog-ng-mod-stomp syslog-ng-mod-xml-parser
Suggested packages:
  mmdb-bin snmp-mibs-downloader rabbitmq-server graphite-web mongodb-server libdbd-mysql libdbd-pgsql
  libdbd-sqlite3 activemq
The following packages will be REMOVED:
  rsyslog
The following NEW packages will be installed:
  libbson-1.0-0 libdbi1 libesmtp6 libhiredis0.14 libivykis0 libmaxminddb0 libmongoc-1.0-0 libmongocrypt0
  libnet1 libprotobuf-c1 librabbitmq4 librdkafka1 libriemann-client0 libsnappy1v5 libsnmp-base libsnmp40
  syslog-ng syslog-ng-core syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-examples
  syslog-ng-mod-extra syslog-ng-mod-geoip2 syslog-ng-mod-getent syslog-ng-mod-graphite syslog-ng-mod-http
  syslog-ng-mod-map-value-pairs syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-rdkafka
  syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-slog syslog-ng-mod-smtp syslog-ng-mod-snmp
  syslog-ng-mod-sql syslog-ng-mod-stardate syslog-ng-mod-stomp syslog-ng-mod-xml-parser
0 upgraded, 39 newly installed, 1 to remove and 0 not upgraded.
Need to get 7,015 kB of archives.
After this operation, 15.1 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main arm64 libbson-1.0-0 arm64 1.17.6-1 [69.7 kB]
Get:2 http://deb.debian.org/debian bullseye/main arm64 libmongocrypt0 arm64 1.1.0-1 [114 kB]
Get:3 http://deb.debian.org/debian bullseye/main arm64 libsnappy1v5 arm64 1.1.8-1 [17.2 kB]
Get:4 http://deb.debian.org/debian bullseye/main arm64 libmongoc-1.0-0 arm64 1.17.6-1 [257 kB]
Get:5 http://deb.debian.org/debian bullseye/main arm64 libivykis0 arm64 0.42.4-1 [25.3 kB]
Get:6 http://deb.debian.org/debian bullseye/main arm64 libnet1 arm64 1.1.6+dfsg-3.1 [56.8 kB]
Get:7 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-core arm64 3.28.1-2+deb11u1 [591 kB]
Get:8 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-mongodb arm64 3.28.1-2+deb11u1 [37.9 kB]
Get:9 http://deb.debian.org/debian bullseye/main arm64 libdbi1 arm64 0.9.0-6 [27.8 kB]
Get:10 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-sql arm64 3.28.1-2+deb11u1 [41.5 kB]
Get:11 http://deb.debian.org/debian bullseye/main arm64 libesmtp6 arm64 1.0.6-4.3 [52.0 kB]
Get:12 http://deb.debian.org/debian bullseye/main arm64 libhiredis0.14 arm64 0.14.1-1 [33.7 kB]
Get:13 http://deb.debian.org/debian bullseye/main arm64 libmaxminddb0 arm64 1.5.2-1 [29.6 kB]
Get:14 http://deb.debian.org/debian bullseye/main arm64 libprotobuf-c1 arm64 1.3.3-1+b2 [26.8 kB]
Get:15 http://deb.debian.org/debian bullseye/main arm64 librabbitmq4 arm64 0.10.0-1 [39.7 kB]
Get:16 http://deb.debian.org/debian bullseye/main arm64 librdkafka1 arm64 1.6.0-1 [515 kB]
Get:17 http://deb.debian.org/debian bullseye/main arm64 libriemann-client0 arm64 1.10.4-2+b2 [21.9 kB]
Get:18 http://deb.debian.org/debian bullseye/main arm64 libsnmp-base all 5.9+dfsg-4+deb11u1 [1,736 kB]
Get:19 http://deb.debian.org/debian bullseye/main arm64 libsnmp40 arm64 5.9+dfsg-4+deb11u1 [2,497 kB]
Get:20 http://deb.debian.org/debian bullseye/main arm64 syslog-ng all 3.28.1-2+deb11u1 [25.9 kB]
Get:21 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-add-contextual-data arm64 3.28.1-2+deb11u1 [40.5 kB]
Get:22 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-amqp arm64 3.28.1-2+deb11u1 [48.8 kB]
Get:23 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-examples arm64 3.28.1-2+deb11u1 [57.3 kB]
Get:24 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-extra all 3.28.1-2+deb11u1 [35.7 kB]
Get:25 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-geoip2 arm64 3.28.1-2+deb11u1 [36.9 kB]
Get:26 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-graphite arm64 3.28.1-2+deb11u1 [29.4 kB]
Get:27 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-http arm64 3.28.1-2+deb11u1 [50.5 kB]
Get:28 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-python arm64 3.28.1-2+deb11u1 [69.9 kB]
Get:29 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-rdkafka arm64 3.28.1-2+deb11u1 [41.5 kB]
Get:30 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-redis arm64 3.28.1-2+deb11u1 [37.6 kB]
Get:31 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-riemann arm64 3.28.1-2+deb11u1 [40.1 kB]
Get:32 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-slog arm64 3.28.1-2+deb11u1 [63.3 kB]
Get:33 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-smtp arm64 3.28.1-2+deb11u1 [38.0 kB]
Get:34 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-snmp arm64 3.28.1-2+deb11u1 [42.5 kB]
Get:35 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-stomp arm64 3.28.1-2+deb11u1 [39.1 kB]
Get:36 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-xml-parser arm64 3.28.1-2+deb11u1 [34.7 kB]
Get:37 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-getent arm64 3.28.1-2+deb11u1 [29.5 kB]
Get:38 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-map-value-pairs arm64 3.28.1-2+deb11u1 [34.0 kB]
Get:39 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-stardate arm64 3.28.1-2+deb11u1 [28.6 kB]
Fetched 7,015 kB in 5s (1,311 kB/s)           
Extracting templates from packages: 100%
(Reading database ... 90182 files and directories currently installed.)
Removing rsyslog (8.2102.0-2+deb11u1) ...
Selecting previously unselected package libbson-1.0-0.
(Reading database ... 90124 files and directories currently installed.)
Preparing to unpack .../00-libbson-1.0-0_1.17.6-1_arm64.deb ...
Unpacking libbson-1.0-0 (1.17.6-1) ...
Selecting previously unselected package libmongocrypt0:arm64.
Preparing to unpack .../01-libmongocrypt0_1.1.0-1_arm64.deb ...
Unpacking libmongocrypt0:arm64 (1.1.0-1) ...
Selecting previously unselected package libsnappy1v5:arm64.
Preparing to unpack .../02-libsnappy1v5_1.1.8-1_arm64.deb ...
Unpacking libsnappy1v5:arm64 (1.1.8-1) ...
Selecting previously unselected package libmongoc-1.0-0.
Preparing to unpack .../03-libmongoc-1.0-0_1.17.6-1_arm64.deb ...
Unpacking libmongoc-1.0-0 (1.17.6-1) ...
Selecting previously unselected package libivykis0:arm64.
Preparing to unpack .../04-libivykis0_0.42.4-1_arm64.deb ...
Unpacking libivykis0:arm64 (0.42.4-1) ...
Selecting previously unselected package libnet1:arm64.
Preparing to unpack .../05-libnet1_1.1.6+dfsg-3.1_arm64.deb ...
Unpacking libnet1:arm64 (1.1.6+dfsg-3.1) ...
Selecting previously unselected package syslog-ng-core.
Preparing to unpack .../06-syslog-ng-core_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-core (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-mongodb.
Preparing to unpack .../07-syslog-ng-mod-mongodb_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-mongodb (3.28.1-2+deb11u1) ...
Selecting previously unselected package libdbi1:arm64.
Preparing to unpack .../08-libdbi1_0.9.0-6_arm64.deb ...
Unpacking libdbi1:arm64 (0.9.0-6) ...
Selecting previously unselected package syslog-ng-mod-sql.
Preparing to unpack .../09-syslog-ng-mod-sql_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-sql (3.28.1-2+deb11u1) ...
Selecting previously unselected package libesmtp6.
Preparing to unpack .../10-libesmtp6_1.0.6-4.3_arm64.deb ...
Unpacking libesmtp6 (1.0.6-4.3) ...
Selecting previously unselected package libhiredis0.14:arm64.
Preparing to unpack .../11-libhiredis0.14_0.14.1-1_arm64.deb ...
Unpacking libhiredis0.14:arm64 (0.14.1-1) ...
Selecting previously unselected package libmaxminddb0:arm64.
Preparing to unpack .../12-libmaxminddb0_1.5.2-1_arm64.deb ...
Unpacking libmaxminddb0:arm64 (1.5.2-1) ...
Selecting previously unselected package libprotobuf-c1:arm64.
Preparing to unpack .../13-libprotobuf-c1_1.3.3-1+b2_arm64.deb ...
Unpacking libprotobuf-c1:arm64 (1.3.3-1+b2) ...
Selecting previously unselected package librabbitmq4:arm64.
Preparing to unpack .../14-librabbitmq4_0.10.0-1_arm64.deb ...
Unpacking librabbitmq4:arm64 (0.10.0-1) ...
Selecting previously unselected package librdkafka1:arm64.
Preparing to unpack .../15-librdkafka1_1.6.0-1_arm64.deb ...
Unpacking librdkafka1:arm64 (1.6.0-1) ...
Selecting previously unselected package libriemann-client0:arm64.
Preparing to unpack .../16-libriemann-client0_1.10.4-2+b2_arm64.deb ...
Unpacking libriemann-client0:arm64 (1.10.4-2+b2) ...
Selecting previously unselected package libsnmp-base.
Preparing to unpack .../17-libsnmp-base_5.9+dfsg-4+deb11u1_all.deb ...
Unpacking libsnmp-base (5.9+dfsg-4+deb11u1) ...
Selecting previously unselected package libsnmp40:arm64.
Preparing to unpack .../18-libsnmp40_5.9+dfsg-4+deb11u1_arm64.deb ...
Unpacking libsnmp40:arm64 (5.9+dfsg-4+deb11u1) ...
Selecting previously unselected package syslog-ng.
Preparing to unpack .../19-syslog-ng_3.28.1-2+deb11u1_all.deb ...
Unpacking syslog-ng (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-add-contextual-data.
Preparing to unpack .../20-syslog-ng-mod-add-contextual-data_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-add-contextual-data (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-amqp.
Preparing to unpack .../21-syslog-ng-mod-amqp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-amqp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-examples.
Preparing to unpack .../22-syslog-ng-mod-examples_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-examples (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-extra.
Preparing to unpack .../23-syslog-ng-mod-extra_3.28.1-2+deb11u1_all.deb ...
Unpacking syslog-ng-mod-extra (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-geoip2.
Preparing to unpack .../24-syslog-ng-mod-geoip2_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-geoip2 (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-graphite.
Preparing to unpack .../25-syslog-ng-mod-graphite_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-graphite (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-http.
Preparing to unpack .../26-syslog-ng-mod-http_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-http (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-python.
Preparing to unpack .../27-syslog-ng-mod-python_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-python (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-rdkafka.
Preparing to unpack .../28-syslog-ng-mod-rdkafka_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-rdkafka (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-redis.
Preparing to unpack .../29-syslog-ng-mod-redis_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-redis (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-riemann.
Preparing to unpack .../30-syslog-ng-mod-riemann_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-riemann (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-slog.
Preparing to unpack .../31-syslog-ng-mod-slog_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-slog (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-smtp.
Preparing to unpack .../32-syslog-ng-mod-smtp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-smtp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-snmp.
Preparing to unpack .../33-syslog-ng-mod-snmp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-snmp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-stomp.
Preparing to unpack .../34-syslog-ng-mod-stomp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-stomp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-xml-parser.
Preparing to unpack .../35-syslog-ng-mod-xml-parser_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-xml-parser (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-getent.
Preparing to unpack .../36-syslog-ng-mod-getent_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-getent (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-map-value-pairs.
Preparing to unpack .../37-syslog-ng-mod-map-value-pairs_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-map-value-pairs (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-stardate.
Preparing to unpack .../38-syslog-ng-mod-stardate_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-stardate (3.28.1-2+deb11u1) ...
Setting up librabbitmq4:arm64 (0.10.0-1) ...
Setting up libdbi1:arm64 (0.9.0-6) ...
Setting up libsnmp-base (5.9+dfsg-4+deb11u1) ...
Setting up libmaxminddb0:arm64 (1.5.2-1) ...
Setting up libesmtp6 (1.0.6-4.3) ...
Setting up libnet1:arm64 (1.1.6+dfsg-3.1) ...
Setting up libprotobuf-c1:arm64 (1.3.3-1+b2) ...
Setting up libsnappy1v5:arm64 (1.1.8-1) ...
Setting up libsnmp40:arm64 (5.9+dfsg-4+deb11u1) ...
Setting up libbson-1.0-0 (1.17.6-1) ...
Setting up libivykis0:arm64 (0.42.4-1) ...
Setting up libriemann-client0:arm64 (1.10.4-2+b2) ...
Setting up librdkafka1:arm64 (1.6.0-1) ...
Setting up libhiredis0.14:arm64 (0.14.1-1) ...
Setting up libmongocrypt0:arm64 (1.1.0-1) ...
Setting up libmongoc-1.0-0 (1.17.6-1) ...
Setting up syslog-ng-core (3.28.1-2+deb11u1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/syslog-ng.service → /lib/systemd/system/syslog-ng.service.
Setting up syslog-ng-mod-examples (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-xml-parser (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-stomp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-riemann (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-stardate (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-geoip2 (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-getent (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-amqp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-python (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-smtp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-snmp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-extra (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-rdkafka (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-graphite (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-add-contextual-data (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-mongodb (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-http (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-slog (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-map-value-pairs (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-sql (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-redis (3.28.1-2+deb11u1) ...
Setting up syslog-ng (3.28.1-2+deb11u1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u8) ...
Scanning processes...                                                                                         
Scanning processor microcode...                                                                               
Scanning linux images...                                                                                      

Running kernel seems to be up-to-date.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

2. With syslog-ng installed, it’s now time to build the configuration for it. A new configuration file fromnet.conf is shown below, in which a syslog-ng destination is created which will aggregate logs from the Turing Pi nodes in /var/log/fromnet in plain text format. Additionally, the logs will be written in JSON format to the file /var/log/fromnet.json.

root@turingpi:~# cat /etc/syslog-ng/fromnet.conf 
# source
source s_fromnet {
  syslog(port(601));
};
# destination 
destination d_fromnet {
  file("/var/log/fromnet");
  file("/var/log/fromnet.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n") );
};
# log path
log {
  source(s_fromnet);
  destination(d_fromnet);
};
  1. Unless we only want to see source IP addresses in the collected logs, it’s necessary to update the syslog-ng configuration file /etc/syslog-ng/syslog-ng.conf to record the hostnames from which the log messages have originated. This is done by adding the keep_hostname(yes) parameter to the options section as follows:
....
....
# First, set some global options. 
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);          
        keep_hostname(yes);dns_cache(no); owner("root"); group("adm"); perm(0640); 
        stats_freq(0); bad_hostname("^gconfd$"); 
};
....
....
  1. Next, the IBM LSF configuration is updated to prevent the creation of local logfiles for the LSF daemons. This is done by commenting the LSF_LOGDIR option in the configuration file $LSF_ENVDIR/lsf.conf. At the same time, we also set LSF_LOG_MASK=LOG_DEBUG for testing purposes to enable verbose logging for the LSF daemons.
....
....
# Daemon log messages
# LSF_LOGDIR=/opt/ibm/lsf/log
LSF_LOG_MASK=LOG_DEBUG
....
....
  1. Finally, to make the changes take effect, both syslog-ng and LSF are restarted.
root@turingpi:~# systemctl restart syslog-ng 
root@turingpi:~# . /opt/ibm/lsf/conf/profile.lsf  
root@turingpi:~# lsf_daemons restart 
Stopping the LSF subsystem 
Starting the LSF subsystem
  1. With the configuration ready on the centralized logging server, host turingpi, we now turn our attention to Nodes 2-7 in the cluster. Here we’ll use the parallel-ssh tool to streamline some operations. We start with the installation of syslog-ng across Nodes 2-7. Note that the output of the installation of syslog-ng across the compute nodes has been truncated.

Truncated output of parallel-ssh -h /opt/workers -i “apt-get install syslog-ng -y”. Click to expand 
root@turingpi:~# parallel-ssh -h /opt/workers -i "apt-get install syslog-ng -y" 
[1] 13:57:07 [SUCCESS] kemeny
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libbson-1.0-0 libdbi1 libesmtp6 libhiredis0.14 libivykis0 libmaxminddb0
  libmongoc-1.0-0 libmongocrypt0 libnet1 libprotobuf-c1 librabbitmq4
  librdkafka1 libriemann-client0 libsensors-config libsensors5 libsnappy1v5
  libsnmp-base libsnmp40 syslog-ng-core syslog-ng-mod-add-contextual-data
  syslog-ng-mod-amqp syslog-ng-mod-examples syslog-ng-mod-extra
  syslog-ng-mod-geoip2 syslog-ng-mod-getent syslog-ng-mod-graphite
  syslog-ng-mod-http syslog-ng-mod-map-value-pairs syslog-ng-mod-mongodb
  syslog-ng-mod-python syslog-ng-mod-rdkafka syslog-ng-mod-redis
  syslog-ng-mod-riemann syslog-ng-mod-slog syslog-ng-mod-smtp
  syslog-ng-mod-snmp syslog-ng-mod-sql syslog-ng-mod-stardate
  syslog-ng-mod-stomp syslog-ng-mod-xml-parser
Suggested packages:
  mmdb-bin lm-sensors snmp-mibs-downloader rabbitmq-server graphite-web
  mongodb-server libdbd-mysql libdbd-pgsql libdbd-sqlite3 activemq
The following packages will be REMOVED:
  rsyslog
The following NEW packages will be installed:
  libbson-1.0-0 libdbi1 libesmtp6 libhiredis0.14 libivykis0 libmaxminddb0
  libmongoc-1.0-0 libmongocrypt0 libnet1 libprotobuf-c1 librabbitmq4
  librdkafka1 libriemann-client0 libsensors-config libsensors5 libsnappy1v5
  libsnmp-base libsnmp40 syslog-ng syslog-ng-core
  syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-examples
  syslog-ng-mod-extra syslog-ng-mod-geoip2 syslog-ng-mod-getent
  syslog-ng-mod-graphite syslog-ng-mod-http syslog-ng-mod-map-value-pairs
  syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-rdkafka
  syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-slog
  syslog-ng-mod-smtp syslog-ng-mod-snmp syslog-ng-mod-sql
  syslog-ng-mod-stardate syslog-ng-mod-stomp syslog-ng-mod-xml-parser
0 upgraded, 41 newly installed, 1 to remove and 0 not upgraded.
Need to get 7,098 kB of archives.
After this operation, 15.3 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main arm64 libbson-1.0-0 arm64 1.17.6-1 [69.7 kB]
Get:2 http://deb.debian.org/debian bullseye/main arm64 libmongocrypt0 arm64 1.1.0-1 [114 kB]
Get:3 http://deb.debian.org/debian bullseye/main arm64 libsnappy1v5 arm64 1.1.8-1 [17.2 kB]
Get:4 http://deb.debian.org/debian bullseye/main arm64 libmongoc-1.0-0 arm64 1.17.6-1 [257 kB]
Get:5 http://deb.debian.org/debian bullseye/main arm64 libivykis0 arm64 0.42.4-1 [25.3 kB]
Get:6 http://deb.debian.org/debian bullseye/main arm64 libnet1 arm64 1.1.6+dfsg-3.1 [56.8 kB]
Get:7 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-core arm64 3.28.1-2+deb11u1 [591 kB]
Get:8 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-mongodb arm64 3.28.1-2+deb11u1 [37.9 kB]
Get:9 http://deb.debian.org/debian bullseye/main arm64 libdbi1 arm64 0.9.0-6 [27.8 kB]
Get:10 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-sql arm64 3.28.1-2+deb11u1 [41.5 kB]
Get:11 http://deb.debian.org/debian bullseye/main arm64 libesmtp6 arm64 1.0.6-4.3 [52.0 kB]
Get:12 http://deb.debian.org/debian bullseye/main arm64 libhiredis0.14 arm64 0.14.1-1 [33.7 kB]
Get:13 http://deb.debian.org/debian bullseye/main arm64 libmaxminddb0 arm64 1.5.2-1 [29.6 kB]
Get:14 http://deb.debian.org/debian bullseye/main arm64 libprotobuf-c1 arm64 1.3.3-1+b2 [26.8 kB]
Get:15 http://deb.debian.org/debian bullseye/main arm64 librabbitmq4 arm64 0.10.0-1 [39.7 kB]
Get:16 http://deb.debian.org/debian bullseye/main arm64 librdkafka1 arm64 1.6.0-1 [515 kB]
Get:17 http://deb.debian.org/debian bullseye/main arm64 libriemann-client0 arm64 1.10.4-2+b2 [21.9 kB]
Get:18 http://deb.debian.org/debian bullseye/main arm64 libsensors-config all 1:3.6.0-7 [32.3 kB]
Get:19 http://deb.debian.org/debian bullseye/main arm64 libsensors5 arm64 1:3.6.0-7 [51.2 kB]
Get:20 http://deb.debian.org/debian bullseye/main arm64 libsnmp-base all 5.9+dfsg-4+deb11u1 [1,736 kB]
Get:21 http://deb.debian.org/debian bullseye/main arm64 libsnmp40 arm64 5.9+dfsg-4+deb11u1 [2,497 kB]
Get:22 http://deb.debian.org/debian bullseye/main arm64 syslog-ng all 3.28.1-2+deb11u1 [25.9 kB]
Get:23 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-add-contextual-data arm64 3.28.1-2+deb11u1 [40.5 kB]
Get:24 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-amqp arm64 3.28.1-2+deb11u1 [48.8 kB]
Get:25 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-examples arm64 3.28.1-2+deb11u1 [57.3 kB]
Get:26 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-extra all 3.28.1-2+deb11u1 [35.7 kB]
Get:27 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-geoip2 arm64 3.28.1-2+deb11u1 [36.9 kB]
Get:28 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-graphite arm64 3.28.1-2+deb11u1 [29.4 kB]
Get:29 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-http arm64 3.28.1-2+deb11u1 [50.5 kB]
Get:30 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-python arm64 3.28.1-2+deb11u1 [69.9 kB]
Get:31 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-rdkafka arm64 3.28.1-2+deb11u1 [41.5 kB]
Get:32 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-redis arm64 3.28.1-2+deb11u1 [37.6 kB]
Get:33 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-riemann arm64 3.28.1-2+deb11u1 [40.1 kB]
Get:34 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-slog arm64 3.28.1-2+deb11u1 [63.3 kB]
Get:35 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-smtp arm64 3.28.1-2+deb11u1 [38.0 kB]
Get:36 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-snmp arm64 3.28.1-2+deb11u1 [42.5 kB]
Get:37 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-stomp arm64 3.28.1-2+deb11u1 [39.1 kB]
Get:38 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-xml-parser arm64 3.28.1-2+deb11u1 [34.7 kB]
Get:39 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-getent arm64 3.28.1-2+deb11u1 [29.5 kB]
Get:40 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-map-value-pairs arm64 3.28.1-2+deb11u1 [34.0 kB]
Get:41 http://deb.debian.org/debian bullseye/main arm64 syslog-ng-mod-stardate arm64 3.28.1-2+deb11u1 [28.6 kB]
Fetched 7,098 kB in 2s (3,566 kB/s)
(Reading database ... 37650 files and directories currently installed.)
Removing rsyslog (8.2102.0-2+deb11u1) ...
Selecting previously unselected package libbson-1.0-0.
(Reading database ... 37592 files and directories currently installed.)
Preparing to unpack .../00-libbson-1.0-0_1.17.6-1_arm64.deb ...
Unpacking libbson-1.0-0 (1.17.6-1) ...
Selecting previously unselected package libmongocrypt0:arm64.
Preparing to unpack .../01-libmongocrypt0_1.1.0-1_arm64.deb ...
Unpacking libmongocrypt0:arm64 (1.1.0-1) ...
Selecting previously unselected package libsnappy1v5:arm64.
Preparing to unpack .../02-libsnappy1v5_1.1.8-1_arm64.deb ...
Unpacking libsnappy1v5:arm64 (1.1.8-1) ...
Selecting previously unselected package libmongoc-1.0-0.
Preparing to unpack .../03-libmongoc-1.0-0_1.17.6-1_arm64.deb ...
Unpacking libmongoc-1.0-0 (1.17.6-1) ...
Selecting previously unselected package libivykis0:arm64.
Preparing to unpack .../04-libivykis0_0.42.4-1_arm64.deb ...
Unpacking libivykis0:arm64 (0.42.4-1) ...
Selecting previously unselected package libnet1:arm64.
Preparing to unpack .../05-libnet1_1.1.6+dfsg-3.1_arm64.deb ...
Unpacking libnet1:arm64 (1.1.6+dfsg-3.1) ...
Selecting previously unselected package syslog-ng-core.
Preparing to unpack .../06-syslog-ng-core_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-core (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-mongodb.
Preparing to unpack .../07-syslog-ng-mod-mongodb_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-mongodb (3.28.1-2+deb11u1) ...
Selecting previously unselected package libdbi1:arm64.
Preparing to unpack .../08-libdbi1_0.9.0-6_arm64.deb ...
Unpacking libdbi1:arm64 (0.9.0-6) ...
Selecting previously unselected package syslog-ng-mod-sql.
Preparing to unpack .../09-syslog-ng-mod-sql_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-sql (3.28.1-2+deb11u1) ...
Selecting previously unselected package libesmtp6.
Preparing to unpack .../10-libesmtp6_1.0.6-4.3_arm64.deb ...
Unpacking libesmtp6 (1.0.6-4.3) ...
Selecting previously unselected package libhiredis0.14:arm64.
Preparing to unpack .../11-libhiredis0.14_0.14.1-1_arm64.deb ...
Unpacking libhiredis0.14:arm64 (0.14.1-1) ...
Selecting previously unselected package libmaxminddb0:arm64.
Preparing to unpack .../12-libmaxminddb0_1.5.2-1_arm64.deb ...
Unpacking libmaxminddb0:arm64 (1.5.2-1) ...
Selecting previously unselected package libprotobuf-c1:arm64.
Preparing to unpack .../13-libprotobuf-c1_1.3.3-1+b2_arm64.deb ...
Unpacking libprotobuf-c1:arm64 (1.3.3-1+b2) ...
Selecting previously unselected package librabbitmq4:arm64.
Preparing to unpack .../14-librabbitmq4_0.10.0-1_arm64.deb ...
Unpacking librabbitmq4:arm64 (0.10.0-1) ...
Selecting previously unselected package librdkafka1:arm64.
Preparing to unpack .../15-librdkafka1_1.6.0-1_arm64.deb ...
Unpacking librdkafka1:arm64 (1.6.0-1) ...
Selecting previously unselected package libriemann-client0:arm64.
Preparing to unpack .../16-libriemann-client0_1.10.4-2+b2_arm64.deb ...
Unpacking libriemann-client0:arm64 (1.10.4-2+b2) ...
Selecting previously unselected package libsensors-config.
Preparing to unpack .../17-libsensors-config_1%3a3.6.0-7_all.deb ...
Unpacking libsensors-config (1:3.6.0-7) ...
Selecting previously unselected package libsensors5:arm64.
Preparing to unpack .../18-libsensors5_1%3a3.6.0-7_arm64.deb ...
Unpacking libsensors5:arm64 (1:3.6.0-7) ...
Selecting previously unselected package libsnmp-base.
Preparing to unpack .../19-libsnmp-base_5.9+dfsg-4+deb11u1_all.deb ...
Unpacking libsnmp-base (5.9+dfsg-4+deb11u1) ...
Selecting previously unselected package libsnmp40:arm64.
Preparing to unpack .../20-libsnmp40_5.9+dfsg-4+deb11u1_arm64.deb ...
Unpacking libsnmp40:arm64 (5.9+dfsg-4+deb11u1) ...
Selecting previously unselected package syslog-ng.
Preparing to unpack .../21-syslog-ng_3.28.1-2+deb11u1_all.deb ...
Unpacking syslog-ng (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-add-contextual-data.
Preparing to unpack .../22-syslog-ng-mod-add-contextual-data_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-add-contextual-data (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-amqp.
Preparing to unpack .../23-syslog-ng-mod-amqp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-amqp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-examples.
Preparing to unpack .../24-syslog-ng-mod-examples_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-examples (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-extra.
Preparing to unpack .../25-syslog-ng-mod-extra_3.28.1-2+deb11u1_all.deb ...
Unpacking syslog-ng-mod-extra (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-geoip2.
Preparing to unpack .../26-syslog-ng-mod-geoip2_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-geoip2 (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-graphite.
Preparing to unpack .../27-syslog-ng-mod-graphite_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-graphite (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-http.
Preparing to unpack .../28-syslog-ng-mod-http_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-http (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-python.
Preparing to unpack .../29-syslog-ng-mod-python_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-python (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-rdkafka.
Preparing to unpack .../30-syslog-ng-mod-rdkafka_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-rdkafka (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-redis.
Preparing to unpack .../31-syslog-ng-mod-redis_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-redis (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-riemann.
Preparing to unpack .../32-syslog-ng-mod-riemann_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-riemann (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-slog.
Preparing to unpack .../33-syslog-ng-mod-slog_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-slog (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-smtp.
Preparing to unpack .../34-syslog-ng-mod-smtp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-smtp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-snmp.
Preparing to unpack .../35-syslog-ng-mod-snmp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-snmp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-stomp.
Preparing to unpack .../36-syslog-ng-mod-stomp_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-stomp (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-xml-parser.
Preparing to unpack .../37-syslog-ng-mod-xml-parser_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-xml-parser (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-getent.
Preparing to unpack .../38-syslog-ng-mod-getent_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-getent (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-map-value-pairs.
Preparing to unpack .../39-syslog-ng-mod-map-value-pairs_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-map-value-pairs (3.28.1-2+deb11u1) ...
Selecting previously unselected package syslog-ng-mod-stardate.
Preparing to unpack .../40-syslog-ng-mod-stardate_3.28.1-2+deb11u1_arm64.deb ...
Unpacking syslog-ng-mod-stardate (3.28.1-2+deb11u1) ...
Setting up librabbitmq4:arm64 (0.10.0-1) ...
Setting up libdbi1:arm64 (0.9.0-6) ...
Setting up libsnmp-base (5.9+dfsg-4+deb11u1) ...
Setting up libmaxminddb0:arm64 (1.5.2-1) ...
Setting up libsensors-config (1:3.6.0-7) ...
Setting up libesmtp6 (1.0.6-4.3) ...
Setting up libnet1:arm64 (1.1.6+dfsg-3.1) ...
Setting up libprotobuf-c1:arm64 (1.3.3-1+b2) ...
Setting up libsnappy1v5:arm64 (1.1.8-1) ...
Setting up libbson-1.0-0 (1.17.6-1) ...
Setting up libivykis0:arm64 (0.42.4-1) ...
Setting up libriemann-client0:arm64 (1.10.4-2+b2) ...
Setting up libsensors5:arm64 (1:3.6.0-7) ...
Setting up librdkafka1:arm64 (1.6.0-1) ...
Setting up libhiredis0.14:arm64 (0.14.1-1) ...
Setting up libmongocrypt0:arm64 (1.1.0-1) ...
Setting up libsnmp40:arm64 (5.9+dfsg-4+deb11u1) ...
Setting up libmongoc-1.0-0 (1.17.6-1) ...
Setting up syslog-ng-core (3.28.1-2+deb11u1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/syslog-ng.service → /lib/systemd/system/syslog-ng.service.
Setting up syslog-ng-mod-examples (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-xml-parser (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-stomp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-riemann (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-stardate (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-geoip2 (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-getent (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-amqp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-python (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-smtp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-snmp (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-extra (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-rdkafka (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-graphite (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-add-contextual-data (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-mongodb (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-http (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-slog (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-map-value-pairs (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-sql (3.28.1-2+deb11u1) ...
Setting up syslog-ng-mod-redis (3.28.1-2+deb11u1) ...
Setting up syslog-ng (3.28.1-2+deb11u1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u8) ...
Stderr: debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
....
....

7. Following the installation of syslog-ng across Nodes 2-7. We verify that the installation was successful by checking the syslog-ng service status.

Output of parallel-ssh -h /opt/workers -i “systemctl status syslog-ng”. Click to expand 
root@turingpi:~# parallel-ssh -h /opt/workers -i "systemctl status syslog-ng" 
[1] 14:03:46 [SUCCESS] kemeny
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:01 EDT; 6min ago
       Docs: man:syslog-ng(8)
   Main PID: 28694 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 40.228s
     CGroup: /system.slice/syslog-ng.service
             └─28694 /usr/sbin/syslog-ng -F

Mar 28 13:57:00 kemeny systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:01 kemeny syslog-ng[28694]: DIGEST-MD5 common mech free
Mar 28 13:57:01 kemeny systemd[1]: Started System Logger Daemon.
[2] 14:03:50 [SUCCESS] vonkarman
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:49 EDT; 5min ago
       Docs: man:syslog-ng(8)
   Main PID: 27486 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 2min 5.540s
     CGroup: /system.slice/syslog-ng.service
             └─27486 /usr/sbin/syslog-ng -F

Mar 28 13:57:44 vonkarman systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:46 vonkarman syslog-ng[27486]: DIGEST-MD5 common mech free
Mar 28 13:57:49 vonkarman systemd[1]: Started System Logger Daemon.
[3] 14:03:51 [SUCCESS] teller
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:39 EDT; 6min ago
       Docs: man:syslog-ng(8)
   Main PID: 24821 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 2min 262ms
     CGroup: /system.slice/syslog-ng.service
             └─24821 /usr/sbin/syslog-ng -F

Mar 28 13:57:38 teller systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:38 teller syslog-ng[24821]: DIGEST-MD5 common mech free
Mar 28 13:57:39 teller systemd[1]: Started System Logger Daemon.
[4] 14:03:53 [SUCCESS] neumann
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:39 EDT; 6min ago
       Docs: man:syslog-ng(8)
   Main PID: 27734 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 1min 43.504s
     CGroup: /system.slice/syslog-ng.service
             └─27734 /usr/sbin/syslog-ng -F

Mar 28 13:57:38 neumann systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:38 neumann syslog-ng[27734]: DIGEST-MD5 common mech free
Mar 28 13:57:39 neumann systemd[1]: Started System Logger Daemon.
[5] 14:03:53 [SUCCESS] wigner
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:37 EDT; 6min ago
       Docs: man:syslog-ng(8)
   Main PID: 27512 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 1min 49.643s
     CGroup: /system.slice/syslog-ng.service
             └─27512 /usr/sbin/syslog-ng -F

Mar 28 13:57:36 wigner systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:36 wigner syslog-ng[27512]: DIGEST-MD5 common mech free
Mar 28 13:57:37 wigner systemd[1]: Started System Logger Daemon.
[6] 14:03:57 [SUCCESS] szilard
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 13:57:35 EDT; 6min ago
       Docs: man:syslog-ng(8)
   Main PID: 24136 (syslog-ng)
      Tasks: 5 (limit: 779)
        CPU: 2min 10.257s
     CGroup: /system.slice/syslog-ng.service
             └─24136 /usr/sbin/syslog-ng -F

Mar 28 13:57:34 szilard systemd[1]: Starting System Logger Daemon...
Mar 28 13:57:34 szilard syslog-ng[24136]: DIGEST-MD5 common mech free
Mar 28 13:57:35 szilard systemd[1]: Started System Logger Daemon.

8. Create the configuration file send.conf in /opt on host turingpi. Note that /opt is an NFS export on turingpi and is NFS mounted by all of the compute nodes. This file will set the HOST field to the local hostname for log messages that are sent. This in done in the subsequent steps where “placeholder” will be replaced using a sed operation with the local hostname. Additionally, a data source s_hpc is defined which will scan /opt/ibm/lsf/log for the presence of LSF daemon logfiles.

 
root@turingpi:/# cat /opt/send.conf
rewrite r_host { set("placeholder", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_src);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};
  1. On Nodes 2-7, copy the file /opt/send.conf to /etc/syslog-ng/conf.d/send.conf.
 
root@turingpi:/# parallel-ssh -h /opt/workers -i "cp /opt/send.conf /etc/syslog-ng/conf.d" 
[1] 14:19:29 [SUCCESS] kemeny
[2] 14:19:30 [SUCCESS] vonkarman
[3] 14:19:30 [SUCCESS] wigner
[4] 14:19:30 [SUCCESS] szilard
[5] 14:19:30 [SUCCESS] teller
[6] 14:19:31 [SUCCESS] neumann
  1. Using sed, replace the “placeholder” string in /etc/syslog-ng/conf.d/send.conf with the local hostname. And we also double check that the change was correctly made.
 
root@turingpi:/# parallel-ssh -h /opt/workers -i 'HOST=`hostname`; sed -i "s/placeholder/$HOST/g" /etc/syslog-ng/conf.d/send.conf' 
[1] 14:38:09 [SUCCESS] kemeny
[2] 14:38:09 [SUCCESS] teller
[3] 14:38:09 [SUCCESS] vonkarman
[4] 14:38:09 [SUCCESS] wigner
[5] 14:38:09 [SUCCESS] neumann
[6] 14:38:09 [SUCCESS] szilard

Output of parallel-ssh -h /opt/workers -i “cat /etc/syslog-ng/conf.d/send.conf”. Click to expand 
root@turingpi:/# parallel-ssh -h /opt/workers -i "cat /etc/syslog-ng/conf.d/send.conf" [1] 14:38:33 [SUCCESS] kemeny
rewrite r_host { set("kemeny", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

[2] 14:38:33 [SUCCESS] teller
rewrite r_host { set("teller", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

[3] 14:38:33 [SUCCESS] neumann
rewrite r_host { set("neumann", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

[4] 14:38:33 [SUCCESS] szilard
rewrite r_host { set("szilard", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

[5] 14:38:33 [SUCCESS] wigner
rewrite r_host { set("wigner", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

[6] 14:38:33 [SUCCESS] vonkarman
rewrite r_host { set("vonkarman", value("HOST")); };

destination d_net {
  syslog("turingpi" port(601));
};
source s_hpc {
  wildcard-file(
      base-dir("/opt/ibm/lsf/log")
      filename-pattern("*.log.*")
      recursive(no)
      follow-freq(1)
  );
};
log {
  source(s_sys);
  source(s_hpc);
  rewrite(r_host); 
  destination(d_net);
};

11. Finally, syslog-ng is restarted on Nodes 2-7 and the status of the service is checked to ensure that there are no errors.

 
root@turingpi:/opt# parallel-ssh -h /opt/workers -i "systemctl restart syslog-ng" 
[1] 14:49:03 [SUCCESS] kemeny
[2] 14:49:05 [SUCCESS] szilard
[3] 14:49:06 [SUCCESS] vonkarman
[4] 14:49:06 [SUCCESS] neumann
[5] 14:49:06 [SUCCESS] teller
[6] 14:49:07 [SUCCESS] wigner

Output of parallel-ssh -h /opt/workers -i “systemctl status syslog-ng”. Click to expand 
root@turingpi:/opt# parallel-ssh -h /opt/workers -i "systemctl status syslog-ng" 
[1] 14:49:31 [SUCCESS] kemeny
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:03 EDT; 28s ago
       Docs: man:syslog-ng(8)
   Main PID: 34982 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 398ms
     CGroup: /system.slice/syslog-ng.service
             └─34982 /usr/sbin/syslog-ng -F

Mar 28 14:49:02 kemeny systemd[1]: Starting System Logger Daemon...
Mar 28 14:49:02 kemeny syslog-ng[34982]: DIGEST-MD5 common mech free
Mar 28 14:49:03 kemeny systemd[1]: Started System Logger Daemon.
[2] 14:49:33 [SUCCESS] vonkarman
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:06 EDT; 25s ago
       Docs: man:syslog-ng(8)
   Main PID: 33710 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 934ms
     CGroup: /system.slice/syslog-ng.service
             └─33710 /usr/sbin/syslog-ng -F

Mar 28 14:49:03 vonkarman systemd[1]: Starting System Logger Daemon...
Mar 28 14:49:03 vonkarman syslog-ng[33710]: DIGEST-MD5 common mech free
Mar 28 14:49:06 vonkarman systemd[1]: Started System Logger Daemon.
[3] 14:49:33 [SUCCESS] neumann
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:06 EDT; 25s ago
       Docs: man:syslog-ng(8)
   Main PID: 34000 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 959ms
     CGroup: /system.slice/syslog-ng.service
             └─34000 /usr/sbin/syslog-ng -F

Mar 28 14:49:03 neumann systemd[1]: Starting System Logger Daemon...
Mar 28 14:49:03 neumann syslog-ng[34000]: DIGEST-MD5 common mech free
Mar 28 14:49:06 neumann systemd[1]: Started System Logger Daemon.
[4] 14:49:33 [SUCCESS] wigner
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:07 EDT; 25s ago
       Docs: man:syslog-ng(8)
   Main PID: 33941 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 1.115s
     CGroup: /system.slice/syslog-ng.service
             └─33941 /usr/sbin/syslog-ng -F

Mar 28 14:49:03 wigner systemd[1]: Starting System Logger Daemon...
Mar 28 14:49:04 wigner syslog-ng[33941]: DIGEST-MD5 common mech free
Mar 28 14:49:07 wigner systemd[1]: Started System Logger Daemon.
[5] 14:49:34 [SUCCESS] szilard
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:05 EDT; 26s ago
       Docs: man:syslog-ng(8)
   Main PID: 30348 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 816ms
     CGroup: /system.slice/syslog-ng.service
             └─30348 /usr/sbin/syslog-ng -F

Mar 28 14:49:03 szilard systemd[1]: Starting System Logger Daemon...
Mar 28 14:49:03 szilard syslog-ng[30348]: DIGEST-MD5 common mech free
Mar 28 14:49:05 szilard systemd[1]: Started System Logger Daemon.
[6] 14:49:34 [SUCCESS] teller
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-28 14:49:06 EDT; 25s ago
       Docs: man:syslog-ng(8)
   Main PID: 31034 (syslog-ng)
      Tasks: 2 (limit: 779)
        CPU: 965ms
     CGroup: /system.slice/syslog-ng.service
             └─31034 /usr/sbin/syslog-ng -F

Does it work?

The answer to this question is an emphatic YES!

Let’s begin with a simple test running the logger command on all of the compute nodes, while monitoring /var/log/fromnet on host turingpi.

 
root@turingpi:/home/lsfadmin# date; parallel-ssh -h /opt/workers -i 'HOST=`hostname`; logger This is a test from node $HOST. Do not panic!' 
Wed  3 Apr 21:41:45 EDT 2024 
[1] 21:41:46 [SUCCESS] teller 
[2] 21:41:46 [SUCCESS] neumann 
[3] 21:41:46 [SUCCESS] wigner 
[4] 21:41:46 [SUCCESS] kemeny 
[5] 21:41:46 [SUCCESS] szilard 
[6] 21:41:46 [SUCCESS] vonkarman

root@turingpi:/var/log# tail -f fromnet |grep panic 
Apr  3 21:41:46 szilard root[10918]: This is a test from node szilard. Do not panic! 
Apr  3 21:41:46 wigner root[11011]: This is a test from node wigner. Do not panic! 
Apr  3 21:41:46 neumann root[11121]: This is a test from node neumann. Do not panic! 
Apr  3 21:41:46 kemeny root[11029]: This is a test from node kemeny. Do not panic! 
Apr  3 21:41:46 teller root[10875]: This is a test from node teller. Do not panic! 
Apr  3 21:41:46 vonkarman root[10805]: This is a test from node vonkarman. Do not panic!

Next, let’s look at whether the LSF logging is also captured. Here we simply restart the LSF daemons on Nodes 2-7 and monitor the /var/log/fromnet file. The full output can be viewed below.

Output of tail -f /var/log/fromnet. Click to expand 
root@turingpi:/var/log# tail -f fromnet 
Apr  3 21:41:57 vonkarman systemd[10786]: systemd-exit.service: Succeeded. 
Apr  3 21:41:57 vonkarman systemd[10786]: Finished Exit the Session. 
Apr  3 21:41:57 vonkarman systemd[10786]: Reached target Exit the Session. 
Apr  3 21:41:57 vonkarman systemd[1]: user@0.service: Succeeded. 
Apr  3 21:41:57 vonkarman systemd[1]: Stopped User Manager for UID 0. 
Apr  3 21:41:57 vonkarman systemd[1]: Stopping User Runtime Directory /run/user/0... 
Apr  3 21:41:57 vonkarman systemd[1]: run-user-0.mount: Succeeded. 
Apr  3 21:41:57 vonkarman systemd[1]: user-runtime-dir@0.service: Succeeded. 
Apr  3 21:41:57 vonkarman systemd[1]: Stopped User Runtime Directory /run/user/0. 
Apr  3 21:41:57 vonkarman systemd[1]: Removed slice User Slice of UID 0. 
Apr  3 21:44:30 wigner dhcpcd[493]: eth0: Router Advertisement from fe80::da58:d7ff:fe00:6d83 
Apr  3 21:44:57 szilard sshd[11234]: Accepted publickey for root from 192.168.1.172 port 52600 ssh2: ED25519 S
HA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:57 szilard sshd[11234]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:58 szilard systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:58 szilard systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 szilard systemd-logind[382]: New session 30 of user root. 
Apr  3 21:44:58 szilard systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 szilard systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 szilard systemd[11237]: pam_unix(systemd-user:session): session opened for user root(uid=0) by
(uid=0) 
Apr  3 21:44:57 wigner sshd[11342]: Accepted publickey for root from 192.168.1.172 port 60388 ssh2: ED25519 SH
A256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:57 wigner sshd[11342]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:58 wigner systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:58 wigner systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 wigner systemd-logind[383]: New session 30 of user root. 
Apr  3 21:44:58 wigner systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 wigner systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 wigner systemd[11345]: pam_unix(systemd-user:session): session opened for user root(uid=0) by 
(uid=0) 
Apr  3 21:44:57 neumann sshd[11436]: Accepted publickey for root from 192.168.1.172 port 55144 ssh2: ED25519 S
HA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:57 neumann sshd[11436]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:57 neumann systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:57 neumann systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 neumann systemd-logind[398]: New session 30 of user root. 
Apr  3 21:44:58 neumann systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 neumann systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 neumann systemd[11439]: pam_unix(systemd-user:session): session opened for user root(uid=0) by
(uid=0) 
Apr  3 21:44:57 kemeny sshd[11345]: Accepted publickey for root from 192.168.1.172 port 59830 ssh2: ED25519 SH
A256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:57 kemeny sshd[11345]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:58 kemeny systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:58 kemeny systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 kemeny systemd-logind[386]: New session 30 of user root. 
Apr  3 21:44:58 kemeny systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 kemeny systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 kemeny systemd[11348]: pam_unix(systemd-user:session): session opened for user root(uid=0) by 
(uid=0) 
Apr  3 21:44:57 teller sshd[11189]: Accepted publickey for root from 192.168.1.172 port 35310 ssh2: ED25519 SH
A256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:57 teller sshd[11189]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:58 teller systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:58 teller systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 teller systemd-logind[382]: New session 30 of user root. 
Apr  3 21:44:58 teller systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 teller systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 teller systemd[11192]: pam_unix(systemd-user:session): session opened for user root(uid=0) by 
(uid=0) 
Apr  3 21:44:57 vonkarman sshd[11118]: Accepted publickey for root from 192.168.1.172 port 48654 ssh2: ED25519
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
Apr  3 21:44:58 vonkarman sshd[11118]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) 
Apr  3 21:44:58 vonkarman systemd[1]: Created slice User Slice of UID 0. 
Apr  3 21:44:58 vonkarman systemd[1]: Starting User Runtime Directory /run/user/0... 
Apr  3 21:44:58 vonkarman systemd-logind[382]: New session 29 of user root. 
Apr  3 21:44:58 vonkarman systemd[1]: Finished User Runtime Directory /run/user/0. 
Apr  3 21:44:58 vonkarman systemd[1]: Starting User Manager for UID 0... 
Apr  3 21:44:58 vonkarman systemd[11121]: pam_unix(systemd-user:session): session opened for user root(uid=0) 
by (uid=0) 
Apr  3 21:44:58 neumann systemd[11439]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 neumann systemd[11439]: Created slice User Application Slice. 
Apr  3 21:44:58 neumann systemd[11439]: Reached target Paths. 
Apr  3 21:44:58 neumann systemd[11439]: Reached target Timers. 
Apr  3 21:44:58 neumann systemd[11439]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 neumann systemd[11439]: Listening on GnuPG cryptographic agent and passphrase cache (access fo
r web browsers). 
Apr  3 21:44:58 neumann systemd[11439]: Listening on GnuPG cryptographic agent and passphrase cache (restricte
d). 
Apr  3 21:44:58 neumann systemd[11439]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 neumann systemd[11439]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 neumann systemd[11439]: Reached target Sockets. 
Apr  3 21:44:58 neumann systemd[11439]: Reached target Basic System. 
Apr  3 21:44:58 neumann systemd[11439]: Reached target Main User Target. 
Apr  3 21:44:58 neumann systemd[11439]: Startup finished in 379ms. 
Apr  3 21:44:58 neumann systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 neumann systemd[1]: Started Session 30 of user root. 
Apr  3 21:44:58 teller systemd[11192]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 teller systemd[11192]: Created slice User Application Slice. 
Apr  3 21:44:58 teller systemd[11192]: Reached target Paths. 
Apr  3 21:44:58 teller systemd[11192]: Reached target Timers. 
Apr  3 21:44:58 teller systemd[11192]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 teller systemd[11192]: Listening on GnuPG cryptographic agent and passphrase cache (access for
web browsers). 
Apr  3 21:44:58 teller systemd[11192]: Listening on GnuPG cryptographic agent and passphrase cache (restricted
). 
Apr  3 21:44:58 teller systemd[11192]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 teller systemd[11192]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 teller systemd[11192]: Reached target Sockets. 
Apr  3 21:44:58 teller systemd[11192]: Reached target Basic System. 
Apr  3 21:44:58 teller systemd[11192]: Reached target Main User Target. 
Apr  3 21:44:58 teller systemd[11192]: Startup finished in 373ms. 
Apr  3 21:44:58 teller systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 teller systemd[1]: Started Session 30 of user root. 
Apr  3 21:44:58 vonkarman systemd[11121]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 vonkarman systemd[11121]: Created slice User Application Slice. 
Apr  3 21:44:58 vonkarman systemd[11121]: Reached target Paths. 
Apr  3 21:44:58 vonkarman systemd[11121]: Reached target Timers. 
Apr  3 21:44:58 vonkarman systemd[11121]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 vonkarman systemd[11121]: Listening on GnuPG cryptographic agent and passphrase cache (access 
for web browsers). 
Apr  3 21:44:58 vonkarman systemd[11121]: Listening on GnuPG cryptographic agent and passphrase cache (restric
ted). 
Apr  3 21:44:58 vonkarman systemd[11121]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 vonkarman systemd[11121]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 vonkarman systemd[11121]: Reached target Sockets. 
Apr  3 21:44:58 vonkarman systemd[11121]: Reached target Basic System. 
Apr  3 21:44:58 vonkarman systemd[11121]: Reached target Main User Target. 
Apr  3 21:44:58 vonkarman systemd[11121]: Startup finished in 392ms. 
Apr  3 21:44:58 vonkarman systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 vonkarman systemd[1]: Started Session 29 of user root. 
Apr  3 21:44:58 szilard systemd[11237]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 szilard systemd[11237]: Created slice User Application Slice. 
Apr  3 21:44:58 szilard systemd[11237]: Reached target Paths. 
Apr  3 21:44:58 szilard systemd[11237]: Reached target Timers. 
Apr  3 21:44:58 szilard systemd[11237]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 szilard systemd[11237]: Listening on GnuPG cryptographic agent and passphrase cache (access fo
r web browsers). 
Apr  3 21:44:58 szilard systemd[11237]: Listening on GnuPG cryptographic agent and passphrase cache (restricte
d). 
Apr  3 21:44:58 szilard systemd[11237]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 szilard systemd[11237]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 szilard systemd[11237]: Reached target Sockets. 
Apr  3 21:44:58 szilard systemd[11237]: Reached target Basic System. 
Apr  3 21:44:58 szilard systemd[11237]: Reached target Main User Target. 
Apr  3 21:44:58 szilard systemd[11237]: Startup finished in 385ms. 
Apr  3 21:44:58 szilard systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 szilard systemd[1]: Started Session 30 of user root. 
Apr  3 21:44:58 wigner systemd[11345]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 wigner systemd[11345]: Created slice User Application Slice. 
Apr  3 21:44:58 wigner systemd[11345]: Reached target Paths. 
Apr  3 21:44:58 wigner systemd[11345]: Reached target Timers. 
Apr  3 21:44:58 wigner systemd[11345]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 wigner systemd[11345]: Listening on GnuPG cryptographic agent and passphrase cache (access for
web browsers). 
Apr  3 21:44:58 wigner systemd[11345]: Listening on GnuPG cryptographic agent and passphrase cache (restricted
). 
Apr  3 21:44:58 wigner systemd[11345]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 wigner systemd[11345]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 wigner systemd[11345]: Reached target Sockets. 
Apr  3 21:44:58 wigner systemd[11345]: Reached target Basic System. 
Apr  3 21:44:58 wigner systemd[11345]: Reached target Main User Target. 
Apr  3 21:44:58 wigner systemd[11345]: Startup finished in 375ms. 
Apr  3 21:44:58 wigner systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 wigner systemd[1]: Started Session 30 of user root. 
Apr  3 21:44:58 kemeny systemd[11348]: Queued start job for default target Main User Target. 
Apr  3 21:44:58 kemeny systemd[11348]: Created slice User Application Slice. 
Apr  3 21:44:58 kemeny systemd[11348]: Reached target Paths. 
Apr  3 21:44:58 kemeny systemd[11348]: Reached target Timers. 
Apr  3 21:44:58 kemeny systemd[11348]: Listening on GnuPG network certificate management daemon. 
Apr  3 21:44:58 kemeny systemd[11348]: Listening on GnuPG cryptographic agent and passphrase cache (access for
web browsers). 
Apr  3 21:44:58 kemeny systemd[11348]: Listening on GnuPG cryptographic agent and passphrase cache (restricted
). 
Apr  3 21:44:58 kemeny systemd[11348]: Listening on GnuPG cryptographic agent (ssh-agent emulation). 
Apr  3 21:44:58 kemeny systemd[11348]: Listening on GnuPG cryptographic agent and passphrase cache. 
Apr  3 21:44:58 kemeny systemd[11348]: Reached target Sockets. 
Apr  3 21:44:58 kemeny systemd[11348]: Reached target Basic System. 
Apr  3 21:44:58 kemeny systemd[11348]: Reached target Main User Target. 
Apr  3 21:44:58 kemeny systemd[11348]: Startup finished in 400ms. 
Apr  3 21:44:58 kemeny systemd[1]: Started User Manager for UID 0. 
Apr  3 21:44:58 kemeny systemd[1]: Started Session 30 of user root. 
Apr  3 21:44:59 kemeny res[691]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 kemeny lim[688]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 kemeny sbatchd[693]: Daemon on host <kemeny> received signal <15>; exiting 
Apr  3 21:44:59 kemeny lsf_daemons[11434]: Stopping the LSF subsystem 
Apr  3 21:44:59 kemeny systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 kemeny systemd[1]: lsfd.service: Consumed 11min 56.744s CPU time. 
Apr  3 21:44:59 szilard lim[685]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 szilard res[687]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 szilard sbatchd[689]: Daemon on host <szilard> received signal <15>; exiting 
Apr  3 21:44:59 vonkarman lim[686]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 vonkarman sbatchd[690]: Daemon on host <vonkarman> received signal <15>; exiting 
Apr  3 21:44:59 vonkarman res[688]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 teller lim[683]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 teller res[689]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 teller sbatchd[691]: Daemon on host <teller> received signal <15>; exiting 
Apr  3 21:44:59 teller lsf_daemons[11294]: Stopping the LSF subsystem 
Apr  3 21:44:59 wigner lim[719]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 wigner res[722]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 wigner sbatchd[724]: Daemon on host <wigner> received signal <15>; exiting 
Apr  3 21:44:59 wigner lsf_daemons[11438]: Stopping the LSF subsystem 
Apr  3 21:44:59 neumann res[713]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 neumann sbatchd[715]: Daemon on host <neumann> received signal <15>; exiting 
Apr  3 21:44:59 neumann lim[711]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 neumann lsf_daemons[11540]: Stopping the LSF subsystem 
Apr  3 21:44:59 neumann sshd[11436]: Received disconnect from 192.168.1.172 port 55144:11: disconnected by use
r 
Apr  3 21:44:59 neumann sshd[11436]: Disconnected from user root 192.168.1.172 port 55144 
Apr  3 21:44:59 szilard lsf_daemons[11331]: Stopping the LSF subsystem 
Apr  3 21:44:59 szilard sshd[11234]: Received disconnect from 192.168.1.172 port 52600:11: disconnected by use
r 
Apr  3 21:44:59 szilard sshd[11234]: Disconnected from user root 192.168.1.172 port 52600 
Apr  3 21:44:59 szilard sshd[11234]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 szilard res[11357]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration i
s not ready yet. 
Apr  3 21:44:59 szilard systemd-logind[382]: Session 30 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 szilard res[11357]: cg_load_hierarchies: Please use the LSF package with higher glibc version 
to enable LSF cgroup v2 support. 
Apr  3 21:44:59 szilard systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 szilard systemd[1]: lsfd.service: Consumed 1h 17min 44.040s CPU time. 
Apr  3 21:44:59 neumann sshd[11436]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 neumann systemd-logind[398]: Session 30 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 neumann res[11559]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration i
s not ready yet. 
Apr  3 21:44:59 neumann res[11559]: cg_load_hierarchies: Please use the LSF package with higher glibc version 
to enable LSF cgroup v2 support. 
Apr  3 21:44:59 neumann systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 neumann systemd[1]: lsfd.service: Consumed 1h 17min 21.135s CPU time. 
Apr  3 21:44:59 teller sshd[11189]: Received disconnect from 192.168.1.172 port 35310:11: disconnected by user 
Apr  3 21:44:59 teller sshd[11189]: Disconnected from user root 192.168.1.172 port 35310 
Apr  3 21:44:59 teller sshd[11189]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 teller systemd-logind[382]: Session 30 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 teller res[11307]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration is
not ready yet. 
Apr  3 21:44:59 teller res[11307]: cg_load_hierarchies: Please use the LSF package with higher glibc version t
o enable LSF cgroup v2 support. 
Apr  3 21:44:59 teller res[11307]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 teller lim[11305]: term_handler: Received signal 15, exiting 
Apr  3 21:44:59 teller systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 teller systemd[1]: lsfd.service: Consumed 1h 17min 47.675s CPU time. 
Apr  3 21:44:59 teller sbatchd[11309]: cg_load_hierarchies: Please use the LSF package with higher glibc versi
on to enable LSF cgroup v2 support. 
Apr  3 21:44:59 kemeny sshd[11345]: Received disconnect from 192.168.1.172 port 59830:11: disconnected by user 
Apr  3 21:44:59 kemeny sshd[11345]: Disconnected from user root 192.168.1.172 port 59830 
Apr  3 21:44:59 kemeny sshd[11345]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 kemeny systemd-logind[386]: Session 30 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 kemeny res[11467]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration is
not ready yet. 
Apr  3 21:44:59 kemeny res[11467]: cg_load_hierarchies: Please use the LSF package with higher glibc version t
o enable LSF cgroup v2 support. 
Apr  3 21:44:59 vonkarman lsf_daemons[11215]: Stopping the LSF subsystem 
Apr  3 21:44:59 vonkarman sshd[11118]: Received disconnect from 192.168.1.172 port 48654:11: disconnected by u
ser 
Apr  3 21:44:59 vonkarman sshd[11118]: Disconnected from user root 192.168.1.172 port 48654 
Apr  3 21:44:59 vonkarman sshd[11118]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 vonkarman systemd-logind[382]: Session 29 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 vonkarman res[11241]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration
is not ready yet. 
Apr  3 21:44:59 vonkarman res[11241]: cg_load_hierarchies: Please use the LSF package with higher glibc versio
n to enable LSF cgroup v2 support. 
Apr  3 21:44:59 vonkarman systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 vonkarman systemd[1]: lsfd.service: Consumed 1h 17min 34.650s CPU time. 
Apr  3 21:44:59 wigner sshd[11342]: Received disconnect from 192.168.1.172 port 60388:11: disconnected by user 
Apr  3 21:44:59 wigner sshd[11342]: Disconnected from user root 192.168.1.172 port 60388 
Apr  3 21:44:59 wigner sshd[11342]: pam_unix(sshd:session): session closed for user root 
Apr  3 21:44:59 wigner res[11464]: res/get_hostInfo: ls_gethostinfo() failed. Server host LIM configuration is
not ready yet. 
Apr  3 21:44:59 wigner systemd-logind[383]: Session 30 logged out. Waiting for processes to exit. 
Apr  3 21:44:59 wigner res[11464]: cg_load_hierarchies: Please use the LSF package with higher glibc version t
o enable LSF cgroup v2 support. 
Apr  3 21:44:59 wigner systemd[1]: lsfd.service: Succeeded. 
Apr  3 21:44:59 wigner systemd[1]: lsfd.service: Consumed 1h 17min 44.610s CPU time.

As expected, we observed that LSF log messages are written to the fromnet file. And importantly each entry contains the hostname, so that we can identify the origin of the message.

Conclusion

What started out as a chat about logging, grew into an idea of a blog, for which I am thankful for the collaboration of Peter. We’ve illustrated an example here of how to setup centralized logging on a Turing Pi system with syslog-ng to collect system and LSF logs.

Of course collecting log messages centrally is just the start of a journey. It is an important step as it allows for significantly easier debugging and troubleshooting. You can store logs to databases for easier search. And once you better understand which log messages are important, you can even potentially parse those and generate alersts from them or dashboards. All of these help you to make sure that your HPC system runs smoothly and with minimal downtime. For me this was a learning experience and I’ll be looking how I can implement more broadly centralized logging in my home network.

©️ hpc.social community | Archive | Posts | About | hpc.social blogs | Add Your Blog