Live Performance Monitoring on an OpenStack Cloud

14 min readOct 27, 2023

What is OpenStack ?

OpenStack is an open source platform, which offers powerful virtual servers and required services for cloud computing. It is mostly deployed as Infrastructure-as-a-service (IaaS), which aims to provide hardware tools and components for processing, storage, and networking resources throughout a data center.

OpenStack can be understood as a software platform that uses pooled virtual resources to build and manage clouds, both public and private ones.

By default, OpenStack offers a couple of cloud-related services like networking, storage, image services, identity, etc., and can be clubbed with a few more to get a customized cloud optimization to support the cloud-native apps.

Devstack is a series of extensible scripts, which is used to set up an OpenStack environment with ease. It is used alot, as it gives an interactive environment for development with OpenStack.

What is Prometheus ?

Prometheus is a open source Linux Server Monitoring tool mainly used for metrics monitoring, event monitoring, alert management, etc.
Prometheus has changed the way of monitoring systems and that is why it has become the Top-Level project of Cloud Native Computing Foundation (CNCF).
Prometheus uses a powerful query language i.e. “PromQL”.
In Prometheus tabs are on and handles hundreds of services and microservices.
Prometheus use multiple modes used for graphing and dashboarding support.

What is Node Exporter ?

Node exporter is one of the Prometheus exporters which is used to expose servers or system OS metrics.
With the help of Node exporter we can expose various resources of the system like RAM, CPU utilization, Memory Utilization, disk space.
Node exporter runs as a system service which gathers the metrics of your system and that gathered metrics is displayed with the help of Grafana visualization tool.

What is Grafana ?

Grafana is a free and open source visualization tool mostly used with Prometheus to which monitor metrics.
Grafana provides various dashboards, charts, graphs, alerts for the particular data source.
Grafana allows us to query, visualize, explore metrics and set alerts for the data source which can be a system, server, nodes, cluster, etc.
We can also create our own dynamic dashboard for visualization and monitoring.
We can save the dashboard and can even share with our team members which is one of the main advantage of Grafana.

In this article, we’ll learn how to install OpenStack cloud and Monitoring a live Performance of that cloud. OpenStack is a cloud computing infrastructure (IaaS) that helps in controlling large pools of computation power, storage, and networking resources throughout a datacenter.

Monitoring is an important part of cloud computing platform, Prometheus to collect the monitoring data of OpenStack cloud and uses the visualization tool Grafana to display the monitoring data in real time which premise of network analysis, system management, job scheduling, load balancing, event prediction, fault detection and recovery operations in cloud computing platform. Monitoring can help OpenStack cloud to dynamically quantify resource usage, detect service defects, discover user usage patterns, and assist resource scheduling, which plays an important role in improving the service quality of cloud.

Pre-requisites for using OpenStack

There are a few basic pre-requisites you need to fulfill, before setting up OpenStack on your system.

Ubuntu OS
A minimum of 4 GB RAM
Multi-core enabled processor
At least 10GB of free hard disk space
A good internet connection

Steps to Install Openstack on Ubuntu

Installing OpenStack on Ubuntu is a rather complex process. The steps to install it, are quite easy even if you’re not much proficient with the command line, simply follow the steps and get it up and running.

Step 1: Preparing the system

Before we start off, we need to ensure that our system is updated, for that run following command:

$ sudo apt-get update && sudo apt-get upgrade -y

The command will ask for root privileges. Enter your user password and wait for your system to upgrade. After the upgrade is finished, make sure to reboot your system. It will initialize and setup your upgrades in the next reboot.

Step 2: Creating stack user with Sudo privileges

Now, it’s time to start with the important steps to install Openstack on Ubuntu. First we will create a new user named stack for our system to setup OpenStack, as it should be installed on a non-root user with sudo enabled.

Open a fresh terminal, and run the useradd command:

$ sudo useradd -s /bin/bash -d /opt/stack -m stack

You also need to enable stack user to have root privileges and run without a password, for that run:

$ echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

The Output will look like this –

Once you have created the stack user, it’s time to log in to it using the following command:

$ sudo su - stack

It will log in you to bash, as stack user.

Step 3: Downloading Devstack

For this step, we considered you already git installed on your system. Now, enter this command to download/clone devstack from its repository to your system:

$ git clone https://opendev.org/openstack/devstack

Devstack repo contains a script stack.sh , which we will use to setup OpenStack. It also contains templates for configuration files.

The Output will look like this –

Step 4: Creating configuration file

Now, we have downloaded and need to setup our configuration files for it.

You need to first navigate to the devstack folder, by running:

$ cd devstack

Afterwards, create a local.conf file, by running:

$ sudo vim local.conf

and paste the following content –

[[local|localr]]
ADMIN_PASSWORD=StrongAdminSecretDATABASE_PASSWORD=$ADMIN_PASSWOCinderRABBIT_PASSWORD=$ADMIN_PASSWORDSERVICE_PASSWORD=$ADMIN_PASSWORD

now, you can just paste using the mouse by right-clicking and clicking Paste and enter :x to save & exit.

Here, I used a minimal configuration to setup DevStack, you can explore

Note:
1. StrongAdminSecret is the password we used here, you can change it with your choice.
2. You can find a sample configuration file for local.conf in the Samples directory under the Devstack repository.

Step 5: Installing Openstack with Devstack

Now, as we have setup the configuration files properly.

Let’s run the script to setup OpenStack on our system, using the following command:

$ ./stack.sh

(This script we’re using is part of DevStack itself)

The script will install the listed features for your OpenStack environment –

Horizon — OpenStack Dashboard
Keystone — Identity Service
Nova — Compute Service
Glance — Image Service
Neutron — Network Service
Placement — Placement API
Cinder — Block Storage Service

The setup will take around 10 to 20 minutes, based on your system performance and internet speed, as many git trees and packages are installed during the process.

After your installation successfully finishes, your terminal will look like the image below.

Now, we can see that it is saying that Horizon (Openstack Dashboard) is available at the given URL, it will vary from system to system.

Step 6: Accessing OpenStack using a web browser

Now, as we have successfully setup OpenStack using Devstack, let’s access it via our browser.

Browse this URL on your browser –

https://server-ip/dashboard

It will open up the OpenStack login page, as shown below.

Now, enter the credentials. You can also log in as admin here, by having User Name as admin & for Password using the one we added to local.conf file.

After logging in, your dashboard will look similar to this.

Step 7: Launching a Instance on Openstack

First we will go to instances and click on new Instance to create once we fill all the parameters we are able to access the Instance.

Pre-requisites for Live Monitoring

Root user account with sudo privilege.
Prometheus system user and group.
Sufficient storage on your system and good internet connectivity.
Ports Required- 9090 (Prometheus), 3000 (Grafana), 9100 (Node Exporter)

We will update the system repository index by using the following command.

sudo apt update -y

switch to root user

sudo su -

Export the release of Prometheus

export RELEASE="2.2.1"

Steps to Install Monitoring System

Step 1: Creating Prometheus System Users and Directory

We will have to create a Prometheus user named Prometheus and a Prometheus directory named as Prometheus.

Using below commands we can create a user and directory.

sudo useradd --no-create-home --shell /bin/false prometheussudo useradd --no-create-home --shell /bin/false node_exportersudo mkdir /etc/prometheussudo mkdir /var/lib/prometheus

Step 2: Update Prometheus user

As user groups and directories are created successfully which store the Prometheus data and files.

Now we will have to update the group and user ownership on the newly created directories.

By using the below command we update the ownership.

sudo chown prometheus:prometheus /etc/prometheussudo chown prometheus:prometheus /var/lib/prometheus

Step 3: Download Prometheus Binary File

Now we will download the latest version of Prometheus. We can copy the download link as per our Operating System from Prometheus download page

Using below command we can download Prometheus, here we are downloading Prometheus 2.26 version, you use above link to download specific version.

Navigate to /opt directory

cd /opt/

Download the Prometheus setup using wget

wget https://github.com/prometheus/prometheus/releases/download/v2.26.0/prometheus-2.26.0.linux-amd64.tar.gz

Now we have successfully downloaded the Prometheus file and now we will extract that file.

Step 4: Extract Prometheus

We can use sha256sum command line to generate a checksum of the Prometheus downloaded file.

We will also extract the downloaded file using the tar command.

sha256sum prometheus-2.26.0.linux-amd64.tar.gz

Output:

f1f2eeabbf7822572dce67565dc96ffaa2dd1897dd1d844562552b11123f151a prometheus-2.26.0.linux-amd64.tar.gz

We have verify that the output from above command with checksum matches the sha256sum checksum which is on official Prometheus download page.

It will ensure that our downloaded file is not a corrupted file.

Now we will extract the Prometheus setup file using the following commands.

tar -xvf prometheus-2.26.0.linux-amd64.tar.gz

navigate to prometheus extracted folder

cd prometheus-2.26.0.linux-amd64

To check list of setup files

ls

Step 5: Copy Prometheus Binary files

Now we have two libraries in our directory i.e. Prometheus and promtool. We will have to copy that both libraries to our /usr/local/bin directory.

By following below commands we will perform the copy operation.

sudo cp /opt/prometheus-2.26.0.linux-amd64/prometheus /usr/local/bin/sudo cp /opt/prometheus-2.26.0.linux-amd64/promtool /usr/local/bin/

Step 6: Update Prometheus user ownership on Binaries

Now we will update the user and group ownership on the binaries of Prometheus.

Using following commands we will update the user and group ownership.

sudo chown prometheus:prometheus /usr/local/bin/prometheussudo chown prometheus:prometheus /usr/local/bin/promtool

Step 7: Copy Prometheus Console Libraries

We will need to copy the console and console_libraries directories to /etc/Prometheus/.

Use below commands to copy console and console_libraries.

sudo cp -r /opt/prometheus-2.26.0.linux-amd64/consoles /etc/prometheussudo cp -r /opt/prometheus-2.26.0.linux-amd64/console_libraries /etc/prometheussudo cp -r /opt/prometheus-2.26.0.linux-amd64/prometheus.yml /etc/prometheus

Step 8: Update Prometheus ownership on Directories

Now we will update the user and group ownership on the directories to Prometheus user using -R.

By executing this commands ownership is set on is ensured. Execute the following commands.

sudo chown -R prometheus:prometheus /etc/prometheus/consolessudo chown -R prometheus:prometheus /etc/prometheus/console_librariessudo chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

Step 9: Check Prometheus Version

Now the Prometheus is successfully installed on our system. We will check the version of Prometheus and promtool to configure it.

Follow the commands to verify prometheus version.

prometheus --versionpromtool --version

Step 10: Prometheus configuration file

We have already copied /opt/prometheus-2.26.0.linux-amd64/prometheus.yml file /etc/prometheus directory, verify if it present and should look like below and modify it as per your requirement.

cat /etc/prometheus/prometheus.yml# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.  static_configs:
  - targets: ['localhost:9090']

Step 11: Creating Prometheus Systemd file

To run Prometheus as service we have to setting up prometheus, We will provide a path for both configuration file and data directory. We will start it with the Prometheus user using the following command.

sudo -u prometheus /usr/local/bin/prometheus \
        --config.file /etc/prometheus/prometheus.yml \
        --storage.tsdb.path /var/lib/prometheus/ \
        --web.console.templates=/etc/prometheus/consoles \
        --web.console.libraries=/etc/prometheus/console_libraries

Now we will create a system service file in /etc/systemd/system location.

sudo nano /etc/systemd/system/prometheus.service

After creating file successfully, copy the below files and it to the newly created file. /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries[Install]
WantedBy=multi-user.target

After adding the program save the file with Ctrl+O and exit with Ctrl+X.

To use the newly created service we will have to reload the daemon services, Use the below command to reload daemon services.

sudo systemctl daemon-reload

start and enable prometheus service using below commands

sudo systemctl start prometheussudo systemctl enable prometheus

We will check the Prometheus status weather it is running or not

sudo systemctl status prometheus

Output :-

Step 12: Accessing Prometheus

Now as Prometheus installation and configuration is set up and it is ready to use we can access its services via web interface.Also check weather port 9090 is UP in firewall.

Use below command to enable prometheus service in firewall

sudo ufw allow 9090/tcp

Now Prometheus service is ready to run and we can access it from any web browser.

http://server-IP-or-Hostname:9090.

As we can see the Prometheus dashboards, we can also check the target.As we can observe Current state is UP and we can also see the last scrape.

Step 13: Install Grafana

Now lets Install Grafana for wonderful dashboards and data visualization for monitoring systems, servers, services, etc

We can install grafana on Ubuntu either by downloading .deb package from Grafana Download page or using APT which is more easier.

Add the Grafana GPG key in Ubuntu using wget

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.0.4_amd64.debsudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_5.0.4_amd64.deb

Then, Enable the automatic start of Grafana by systemd:

sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server.service

Grafana is running now, and we can connect to it at http://your.server.ip:3000. The default user and password is admin / admin.

Step 14: Configure Prometheus as Grafana DataSource

Once you logged into Grafana Now first Navigate to Settings Icon ->> Configuration ->> data sources

Now lets click on Add Data sources and select Prometheus

Now configure Prometheus data source by providing Prometheus URL

As per your requirement you can do other changes or you can also keep remaining configuration as default.

Now click on Save & test so it will prompt a message Data Source is working.

Step 15: Install Node Exporter

Node Exporter collects the metrics of your system such as Memory usage, CPU usage, RAM, disk space, etc.

To install Node Exporter first navigate to Prometheus official download page, Scroll down and you will get node_exporter section and then select Linux OS for amd64.

Now right click on node exporter and copy link address

Now lets run the copied URL with wget command

wget https://github.com/prometheus/node_exporter/releases/download/v1.2.0/node_exporter-1.2.0.linux-amd64.tar.gz

Unzip the downloaded the file using below command

sudo tar xvzf node_exporter-1.2.0.linux-amd64.tar.gz

Now do ls and your can see node_exporter binary file.

Go to that file and move this file to your /usr/local/bin directory using below command

cd node_exporter-1.2.0.linux-amd64sudo cp node_exporter /usr/local/bin

As early at the time of Prometheus installation we have created node_exporter user and also updated the ownership permission so we will not repeat it again

Step 16: Creating Node Exporter Systemd service

Now lets create a node_exporter service in /lib/systemd/system directory named node_exporter.service using below commands

cd /lib/systemd/systemsudo nano node_exporter.service

Paste the below content in your service file

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
— collector.mountstats \
— collector.logind \
— collector.processes \
— collector.ntp \
— collector.systemd \
— collector.tcpstat \
— collector.wifi
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target

Now lets start and enable the node_exporter service using below commands

sudo systemctl daemon-reloadsudo systemctl enable node_exportersudo systemctl start node_exportersudo systemctl status node_exporter

Step 17: Configure the Node Exporter as a Prometheus target

Now to scrape the node_exporter lets instruct the Prometheus by making a minor change in prometheus.yml file

So go to etc/prometheus and open prometheus.yml

cd /etc/prometheussudo nano prometheus.yml

Now in static_configs in your configuration file replace the target line with the below one

- targets: [‘localhost:9090’, ‘localhost:9100']

Now restart the Prometheus Service

sudo systemctl restart prometheus

Hit the URL in your web browser to check weather our target is successfully scraped by Prometheus or not

https://localhost:9100/targets

We have covered How to Install Prometheus and Grafana on Ubuntu 20.04 LTS with Node Exporter.

Step 18: Creating Grafana Dashboard to Monitor

Now lets build a dashboard in Grafana so then it will able to reflect the metrics of the Linux system.

So we will use 14513 to import Grafana.com, Lets come to Grafana Home page and you can see a “+” icon. Click on that and select “Import”

Now provide the Grafana.com Dashboard ID which is 14513 and click on Load

Now provide the name and select the Prometheus Datasource and click on Import.

There you are done with the setup. Now your Dashboard is running up!.

Conclusion

In this article, Combined with the characteristics of Open-stack cloud platform, on this design and implements a comprehensive, intelligent and efficient monitoring system by utilizing the Powerful Monitoring tool Prometheus and the visualization tool Grafana. The experiment proves that the system has good effectiveness and real-time performance, which makes the open stack cloud platform more complete and also helps to maintain the reliability and stability of Openstack cloud platform. Sometimes have instances that are running very slow without having any real clues of what the issues might be like we may have unresponsive instances or unused instances so we can allocate the resources to other instances. As I expected the Monitoring system has been Implemented on an Open- stack cloud. As a future work we can Implement a monitoring system on a individual instances or separate node’s.