Category Archives: Command line tools

Installing Nagios with Nginx, PHP-FPM and Nagiosgraph on Ubuntu 16.04

Update Your System

Always ensure your system is up-to-date before installing new packages by running:

Please note, this guide assumes you already have Nginx and PHP-FPM installed.

Versions

For the sake of clarity, the server I’m installing this on has the following versions of Ubuntu, Nginx and PHP-FPM:

Nagios User and Group

We’ll start by creating a new user and group specific to Nagios:

Required Dependencies

Next, we’ll install the dependencies we need:

Install Nagios Core

Then, we want to install Nagios Core. To get the latest version, you’ll have to check the Nagios website to see what the latest stable release is, and change the version number accordingly:

At the time of writing, this is v4.3.2. Once downloaded, we need to extract the archive:

Now this has been extracted, we can delete the archive and change into the extracted directory:

Before we actually go ahead and install Nagios, we need to configure it. We’re going to install Nagios with postfix, which we’ve already installed. To do this, we run the following command:

If you read the output, which should be rather verbose, it should let you know where everything is and how to continue:

As it says, review the options above, and if they look okay, run the following command to compile the main program and CGIs:

Installing from source always scares me at this point, as there’s quite a lot going on in the console. Don’t worry, this is normal. If this finishes and ends with ‘Enjoy’, then you’re doing well.

Now, we can get down to actually installing Nagios along with some initial scripts and sample configuration files:

In order for us to make external commands via the web interface to Nagios, we’ll need to sort out permissions. To do this, we’ll add the web server user to our nagcmd group:

Nagios Plugins

Just like Nagios Core, you’ll need to consult the Nagios website to find the latest version of the Nagios Plugins download:

At the time of writing, this is v2.2.1. Once downloaded, we need to extract the archive:

Now this has been extracted, we can delete the archive and change into the extracted directory:

Before installing Nagios Plugins, we’ll need to configure it:

Now, compile Nagios Plugins:

Then, install it:

To finish off our installation, we’ll need to copy over the check_* files to our Nagios configuration, so we can use them later for monitoring:

Install NRPE

Find the latest stable version of NRPE from the NRPE downloads page. If you’re starting to see a trend, you’ll be right. Please note, you may need to manually download the file from their website, as Sourceforce don’t seem to provide direct links. Go ahead and change the version number in the following command if necessary:

At the time of writing, this is v3.1.1. Once downloaded, we need to extract the archive:

If the above command fails with:

Then you may need to manually upload the tar.gz file, after downloading it from the Sourceforce website. Once downloaded to your machine, you can use scp or rsync to upload the file to your server:

After transferring the correct file to the server, go ahead and run the tar command again, as described above.

Now this has been successfully extracted, we can delete the archive and change into the extracted directory:

Before installing NRPE, we’ll need to configure it:

Now build and install NRPE:

Enable the NRPE service:

If you have xinetd installed, you’ll need to configure your /etc/xinetd.d/nrpe file to only include the IP address of the Nagios Server. For more information, Google it.

Now that Nagios, Nagios Plugins and NRPE are installed, we need to configure Nagios.

Configure Nagios

Open the main Nagios configuration file in your favourite text editor:

And remove the following comment:

Pro tip: to search in vim, use the / command following by the string you want to search for, and hit enter.

Then save and exit.

Next, we want to create the directory where all our server configuration files will live in:

Configure Nagios Contacts

Open the Nagios contacts configuration in your favourite text editor:

And change the default email address to your email address:

Then save and exit.

Configure check_nrpe Command

Let’s go ahead and add a new command to our Nagios configuration using our favourite text editor:

And add the following to the end of the file:

Then, save and exit. This allows you to use the check_nrpe command in your Nagios service definitions.

Configure Nginx

Up until now, you may have found most installation guides show you how to install Nagios, Nagios Plugins and NRPE. However, very few show you how to set this up on an Nginx and PHP-FPM environment.

We’ll start by installing the fcgiwrap dependancy, so we can run the Nagios CGI scripts:

Next, you’ll need to create a new virtual host for Nagios. Depending on your current setup, these are usually found in /etc/nginx/sites-available. If that’s the case, create a new configuration file for Nagios using your favourite text editor:

And use the following Nginx virtual host configuration:

Then, save and quit. Once that’s done, go ahead and add the symlink needed for the sites-enabled directory:

Please note, the above configuration assumes a few things. First, you have a self-signed certificate installed and a dhparam.pem file. If you don’t have these, you can generate them with the following commands:

You’ll notice this also includes location blocks for nagiosgraph, which we will install later within this guide, so don’t worry about those for now.

Finally, you’ll need to change the nagioshost.local server name to the actual virtual host URL you’ll be hosting this from.

Next, start the Nagios service and restart Nginx to take the configuration files into effect:

If there are no errors, then we can go ahead and add Nagios to our startup commands:

Now, we can go ahead and visit the public URL we’ve setup for Nagios. At this point, if you see a 404, 502 or other error, check your Nginx and virtual host error logs to see what isn’t happy.

If it’s all looking good, and your public URL looks something like this:

Nagios Core

Configuring Nagios Hosts

The final part to our configuration is setting up our hosts. Let’s make a new file for our server configuration:

And enter the following:

Then, save and quit. With the above configuration file, Nagios can work out if your host is up or down. If this is enough for you, then great. Otherwise, we’ll go ahead and add some more configuration files to check additional services such as ping and SSH. Simply add the following at the end of the same file:

Install Nagiosgraph

If you want to go one step further, you can install Nagiosgraph. This enables you to view pretty graphs which show the history of your statistics. To install Nagiosgraph, download the latest version from the Sourceforce website:

At the time of writing, this is v1.5.2. Once downloaded, we need to extract the archive:

Now this has been successfully extracted, we can delete the archive and change into the extracted directory:

Before installing Nagiosgraph, we can make some pre-flight checks:

Please be aware: because we’re using Nginx, expect to get some errors. If everything else is okay, you should end up with an output such as:

If all looks good, we can go ahead run the installation:

Tidying up

Now that everything’s up and running, so we can go ahead and do some tidying up. Simply remove the folders from your home directory which we downloaded:

And we’re done. Enjoy!

Generating autoload_classmap.php files in Zend Framework 2

Application speed is a critical subject which gets worse as your site experiences more users. Even if you’re running your application on a super-server, it’s going to be much quicker and more cost effective to optimise your website first.

One way to help optimise your website (if you’re running a Zend Framework 2 application) is to use the class map autoloader. This helps to increase your website’s speed by not having to load your class by changing strings about using the namespace given. For something as simple as loading a class, there’s a much better way of going about it: the class map autoloader.

To automatically generate your class map files, we’re going to use the classmap_generator.php file which comes shipped with Zend Framework 2. I’ve created a great little one-liner for you:

for module in $(ls module); do; php vendor/bin/classmap_generator.php -l module/$module; done;

which can be run in the main directory of your Zend Framework 2 project.

Within your Module.php file within each module, find the getAutoloaderConfig() method and ensure you have entered the class map autoloader configuration. You’ll find something like this:

And should replace it with this:

And that’s it! You should now be using class map autoloading.

El Capitan upgrade – Virtual hosts not working?

After upgrading to El Capitan, Apple’s new Operating system (version 10.11), I found all my virtual hosts weren’t working anymore!

This is due to the upgrade creating a new /etc/apache2/httpd.conf file and overwriting your old one. Luckily, they don’t quite remove your previous configuration file, they back it up for you under /etc/apache2/httpd.conf~previous.

If you’ve installed El Capitan and are not sure how to fix this, I’ve written a quick one-liner that will get you back on your feet. Simply open Terminal (From a Finder window, go to Applications > Utilites > Terminal) and paste in the following line:

And there you have it. Back up and running in no time at all.

Let’s go into that command a little deeper so you can understand what we’re doing:

rm -rf /etc/apache2/httpd.conf

This line is simply removing the new configuration file Apple has created for us. We have a backup of it already and we’re going to change the name of our backed up file to this one, so we don’t need it anymore. Please be sure you have already upgraded to El Capitan otherwise you may not have a backup file. To check if you have a backup file available, simply run ls -la /etc/apache2/httpd.conf~previous and you should get something back like this: -rw-r--r--  1 root  wheel  21002  4 Jul 17:41 /etc/apache2/httpd.conf~previous – if not, do not continue or you may remove your only configuration file.

The second line:

mv /etc/apache2/httpd.conf~previous /etc/apache2/httpd.conf

simply renames your backed up configuration file to the one we just deleted.

Lastly we want to restart Apache by running:

sudo apachectl restart

which will re-cache your configuration files which we have just updated.

And there you have it, an easy switch over but something which I found rather annoying during the upgrade. Hopefully this tutorial helps you out if you’ve run into the same problem.

Setting up a LAMP server on AWS EC2

Amazon Web Services have grown quite rapidly over the last few years. This is why they’re generally the most popular scalable hosting that large companies use to ensure their applications are up and running with no pitfalls.

Amazon provide something called ‘free tier’ so you can get started straight away and get stuck in. Go ahead, set up an account and have a play about. It’s free for the first year, either cancel your account after before that point or, if you’ve gotten far enough, it’s pretty cheap anyway.

Once you’re logged in via SSH, use the following commands to set up your LAMP server. Please read the code line by line to ensure you have copied and pasted every command in correctly, and that you understand what each one does (using the helpful comments).

This code will install a simple LAMP server on AWS installing PHP 5.5, MySQL, Intl, Git, APC and ElasticSearch.

If you’re looking for specific packages to install, take a look at Amazon’s package list: http://aws.amazon.com/amazon-linux-ami/2013.09-packages/

 

Counting the most common PHP error logs

Keeping an eye on your error logs is something that most developers neglect. Server administrators, on the other hand, tend to look out for these minor details, as they’ll keep your website secure and ensure there are no warnings, notices or even fatal errors that you may have missed during development.

You can check to where your error log files are piping their information by checking a phpinfo() file (just a php file with the following code inside (be sure to start the file with a php open tag):

Or you can check via the command line using this command:

which simply shows the php information and configuration, piping with grep and finding the string ‘error_log’. It will then return where your error log files are. This will either be at an absolute path, starting with a forward slash (/var/log/php5-fpm.log), or, it will be a relative file such as ‘error_log’. This means the error log will be at the relative path of the executed script that produced the error.

Once you have found the path for your error log, you can run a simple command (it’s simple once we break it down) that will count your errors so you can go ahead and remove the most common errors that are clogging up your php error log:

Let’s break that down. The cat command simply outputs the content of the file you have defined. You’ll notice everything after that uses a pipe (|) to layer on other commands to that output. The sort command – yes, you guessed it – sorts the data numerically and alphabetically. The uniq command removes any duplicates, line by line (makes each line unique) and the -c flag puts a count at the beginning of each line showing how many matches it found. The sort command sorts the data again (after identifying the duplicates) with the -r (which reverses the order, you’ll see why in the last command) and -n flag. Finally, the head command returns the last 10 lines (10 lines by default – you can change this number using the -n, just check the manual).

If your errors (like mine) are relative, you can create a global file using this command:

After starting a new job, I had to take over a few servers which kept going down. After finding out the disk space was full almost every week which caused the server to go ka-boom, I found the PHP FPM error log was over 15GB is size! Amazing, huh? About 40 sites had been developed with no debugging turned on locally so all of these errors had been adding up. Since these sites had quite a lot of traffic, we found ourselves having to truncate the error log almost ever week to free up disk space until I could get round to fixing the most common errors.

Delete a file from the command line that’s still in use

I found myself taking over a few servers which had around 30 sites on them, all with hundreds of their own errors. I had a look in the /var/log directory and found the php5-fpm.log file was over 10GB in size! There are ways of easily truncating this but I found a particular issue which could help some of you. Deleting a large file that’s currently open on the server. A large file which is being used by the system can still remain in your memory. So when you delete your large file and expect your disk space to be free from that file, you’ll be disappointed when you run:

df -h

This command shows the server disk usage. Turns out, there is a process still running in the background which needs to be stopped until this can free up your memory.

To show the list of open files currently running on your server, you can use a simple command called lsof:

lsof -nP | grep '(deleted)'

We’re using the n and P flags and then just piping grep to show the deleted files. The flags used are as follows (taken from the man file):

-P inhibits the conversion of port numbers to port names for network files. Inhibiting the conversion may make lsof run a little faster. It is also useful when port name lookup is not working properly.

-n inhibits the conversion of network numbers to host names for network files. Inhibiting conversion may make lsof run faster.  It is also useful when host name lookup is not working properly.

This will return the files currently deleted that are still in use. Take note of the key (you can check this if you run lsof -nP at the top). The second column shows the process ID. To remove this, simply kill it using:

kill 1234

Where 1234 will be your process ID.

After a quick check using:

df -h

You’ll see that your disk space is now free!

Installing WordPress via the command line

I found myself having to create more and more WordPress websites after clients would specifically ask for ‘a WordPress website’. Although WordPress is built as a blogging tool, it can be used as a very simple content management system (without too much customisation in the back end).

After installing WordPress manually for the seventh time, I thought it would be an idea to make a quick shell script that I would be able to use over SSH, as servers are able to download files via wget much quicker than I can downloading to my local machine and then uploading via FTP or copying it over via scp or rsync. Some hosting providers have automated scripts already installed within Plesk, cPanel and other web host management systems that allow you to install WordPress with a click of a button. It will install the latest version, set up the database and you’re set. The only issue here is when you’re working on dedicated or cloud servers where these tools are not readily available.

I’ve always been the one to start something from scratch so I know the exact the process, how long it should take and what to expect, as well as the ability to easily customise so it can meet my needs.

This simple shell script will:

  • download and unzip the latest version of WordPress
  • remove the sample configuration file
  • generate random salts (taken directly from the WordPress website): https://api.wordpress.org/secret-key/1.1/salt/
  • create a custom configuration file
  • define the SITE_URL and HOME_URL variables as per your handle name
  • copy over any predefined themes and plugins
  • change permissions and ownership of the installed files (otherwise, if you’re running this as root, you’ll end up with internal server errors)

I’ve added a load of comments against each part of the code so it should be easy enough to follow. Anyway, here it is:

I think the only bits I am missing, that I will eventually get round to doing is:

  • Install a new database for the WordPress website
  • Set up new database credentials and insert them into the configuration file
  • Add an environment variable (so it will use scp instead of cp for copying over files if you’re not installing locally, change the predefined URL names and other useful bits)

I hope this script helps someone installing WordPress on their server or local machine (or at least makes the process a little quicker). If there’s anything that I have missed that might be useful to you, please mention them in the comments and I will update the installation script.

This code is also available on GitHub so feel free to download or even contribute: https://github.com/olivertappin/wordpress/blob/master/wordpress.sh

Watermarking images using PHP

My girlfriend recently started a photography business. I found her uploading her photos to Facebook or her website with her logo in the bottom right of the image. She would use Photoshop to do this manually but the work really started when there were quite a few photos to do this to.

I know there are batch processes in Photoshop but this can be quite slow depending on what you’re doing. Photoshop seems to show you the running batch process which uses up a lot of CPU rending the changes on screen. As she’s lucky enough to have me for a boyfriend, I thought I would help out by writing a little script she could use to do this for her.

The class takes a predefined logo and sticks it on the image, saves it and moves on to the next one. There are two versions of the logo, one to be used on a light background, the other to be used on a dark background:

View the class on GitHub: https://github.com/olivertappin/watermark/blob/master/watermark.php

To use the code:

How to use Ack – Beyond grep

I was first introduced to ack when working for a small web agency in the south by a guy called Stuart. I remember him explaining it as “it’s much faster than grep, but that’s because it cheats”. This article will show you how to use ack as well as the useful flags available to speed up your back-end web development.

If you’re on Mac OS X using Homebrew, installation is as easy as

brew install ack

Otherwise, installation is just as easy across other packages, just visit http://beyondgrep.com/install/

Using ack is pretty simple, and super fast. It’s rather useful as it shows the line numbers within the code and you can change how many lines you want to show before and after using the -c flag like so:

ack '\.container' -C 9

The backslash stops the dot from being used as a regular expression, so we’re essentially searching for just ‘.container’. The results are as follows:

➜ resource git:(master) ✗ ack '\.wrapper' -C 9
sass/style.scss
27- height: 26px;
28- padding-left: 162px;
29- box-sizing: border-box;
30- background-repeat: no-repeat;
31- background-image: url('../img/logo.svg');
32-}
33-.content {
34- padding: 55px 0 20px 0;
35-}
36:.wrapper {
37- margin: 0 auto;
38- padding: 0 20px;
39- max-width: 1000px;
40- position: relative;
41-}
42-
43-
44-/* --- Page --------------------- */
45-

If you’re a new guy taking over someone else’s work and have no idea where anything is, without having anyone to ask, ack will end up being your best friend. Looking for that horrible problematic inline styling on that div tag? Just use ack:

ack '\<div style\=\"margin-top: -27px;\">'

And there it is. No more nasty !important hacks. I know this has helped me on a number of projects on so many occasions, especially during handovers and the person of interest is off.

If you’re ever looking for something, don’t forget you always have man files. I always forget we have these to look at, it’s just a case of using them and searching for the right keyword to find what you’re looking for. Just use:

man ack