LibreNMS Distributed Poller - how I did it

I recently found an application for distributed polling in LibreNMS, but the documentation on the aforementioned link seemed a little light, so at first I was hesitant to try it out. Upon some further searching I found this Reddit thread which provided a little more detail, but I still just had to try it out to actually see if I could get it working. Now I'm going to attempt to share my knowledge so someone else can do it in the future! It's not that hard, what's most important is to make sure you install all the extra needed packages and configure both the additional packages AND LibreNMS properly.

Note that in this particular scenario, I have two hosts: the first is the LibreNMS web server, database server, RRDCached server, Redis server...basically all of the server functions. The second host only exists as a poller. My hosts are both running Debian as well, Ubuntu should be more or less the same, other systems should be similar but obviously the package manager will be different.

Also note that in my case, I did the server install, the poller install, then kind of bounced back to the server to configure the additional packages. The order of the steps below is what I would do if I knew what I know now, i.e. complete the server install FIRST, then move on to the poller.

Step 1: Server install

The first step is really the easiest. Simply follow the instructions on how to install LibreNMS from their website. This includes running through the web installer. No special steps here!

Step 2: Install additional packages on the server

apt install memcached rrdcached redis-server python3-memcache php-memcached

Step 3: Configure memcached on the server

Edit /etc/memcached.conf and set the following values. Most of these are already in the file, so you simply need to edit the existing values and make sure they're uncommented.

-d
-m 64
-p 11211
-u memcache
-l 0.0.0.0

Note that in my case, I chose to listen on 0.0.0.0; you should only do this if you implement a firewall and make sure that memcached isn't available to the open internet!

Step 4: Configure rrdcached on the server

Edit /etc/default/rrdcached and add the following to the bottom of the file:

BASE_OPTIONS="-l 0:42217"
BASE_OPTIONS="$BASE_OPTIONS -R -j /var/lib/rrdcached/journal/ -F"
BASE_OPTIONS="$BASE_OPTIONS -b /opt/librenms/rrd -B"
BASE_OPTIONS="$BASE_OPTIONS -w 1800 -z 900"

Step 5: Configure LibreNMS on the server

Edit the following values in /opt/librenms/config.php or add them if they don't already exist:

$config['rrdcached']    = "<ip of your server>:42217";

# Enable distributed polling
$config['distributed_poller'] = true;
$config['distributed_poller_group']                      = 0;
$config['distributed_poller_name']           = php_uname('n');

 Then, edit add following values in /opt/librenms/.env:

CACHE_DRIVER=redis
REDIS_HOST=localhost
REDIS_PORT=6379

Step 6: Enable php-memcached on the server:

phpenmod memcached

Step 7: Allow poller access to the MariaDB instance on your server:

sudo mysql -u root -p <database password you configured during install>
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'<ip of your poller>' IDENTIFIED BY '<the same password you used to access mysql>' WITH GRANT OPTION;
exit

Step 8: Reboot!

At this point, it may be wise to reboot your server to make sure all the services initialized properly, or if you're feeling savvy you can restart each of them individually.

Once the server is back up, check to make sure all your services are listening on all interfaces using the following command. (NOTE: on some older systems, you may need to replace 'ss' with 'netstat'.)

ss -ant4

You should see something like the following:

State      Recv-Q   Send-Q      Local Address:Port        Peer Address:Port         
LISTEN     0        128         0.0.0.0:42217             0.0.0.0:*            
LISTEN     0        80          0.0.0.0:3306              0.0.0.0:*            
LISTEN     0        128         0.0.0.0:11211             0.0.0.0:*            
LISTEN     0        128         0.0.0.0:6379              0.0.0.0:*            

This means that memcached, MariaDB, rrdcached, and redis are all ready and listening on your server.

Step 9: Install LibreNMS on the poller(s)

Follow the same steps from install LibreNMS, but this time DO NOT RUN THE WEB INSTALLER. Additionally in the initial steps, you *may not* need to install MariaDB on the poller since you're not using it. However I didn't try this myself; I simply installed all the packages, then went back later and removed MariaDB from my poller.

Step 10: Configure LibreNMS on the poller(s)

Edit the following values in /opt/librenms/config.php or add them if they don't already exist:

$config['rrdcached']    = "<ip of your rrdcached server>:42217";

# Enable distributed polling $config['distributed_poller'] = true;
$config['distributed_poller_group']                      = 1;
$config['distributed_poller_name']           = php_uname('n');

Note that in this case, you should use the IP address of the server that's running the "full install" of LibreNMS, on which you installed rrdcached. Also, we're putting this poller in a new group, 1; otherwise the main server's poller and this poller will "work together" to poll the same devices at the same time.

Then edit or add the following values to /opt/librenms/.env:

DB_HOST=<ip of your main librenms server>
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=<database password you configured during install>
DB_PORT=3306
REDIS_HOST=<ip of your main librenms server>
REDIS_DB=0

Step 11: Reboot the poller(s)!

Again, it's probably easiest to just reboot. I'll admit I'm not 100% sure how you "restart" LibreNMS so I find that sometimes rebooting is the best option. I think it picks up these changes dynamically, but a fresh instance is always best.

 Step 12: Add the new poller group on the server

Navigate to http://<ip of your main librenms server>/poller/groups, then click the "Create new poller group" button. Name it whatever you want, this is just making the server "aware" of the new poller group.

At this point, your poller should show up in the main LibreNMS system, under the "settings" icon in the top right, then go to "Pollers", or http://<ip of your main librenms server>/poller. You should also be able to run "ss -ant4" on the server again and see a connection on the "peer address" of your poller to the "local address" of your server on port 3306; that proves that the poller connected to the database successfully.

 

Hope that helps someone in the future. Please leave a comment if I missed something, I got mine working a couple days before I actually got to sit down and do this write-up so I'm hoping I didn't forget something!