Prometheus Architecture...

 Prometheus Architecture

Look at the below figure, and we can see that Prometheus has major 3 things: 

  • to pull Metrics, 
  • store it in the Time-Series database, and 
  • allow third parties to retrieve metrics using PromQL
Metrics are mainly pulled from 3 sources: PushGateway, Exporters, and Instrumentation. These pulled metrics are then stored in a time-series database. Later time-series stored metrics are used by third parties such as Prometheus Web UI, Alertmanager, and Grafana.

Components of Prometheus:

There are 3 main components of Prometheus:

  • Retrieval/Pull Metrics
    • This component of Prometheus is mainly responsible for pulling metrics from targets.
    • Targets can be exporters,  instrumentation, and pushgateway.
  • Time Series Database
    • Once the retrieval collects data from the target, then it transfers to a time-series database.
    • The data are stored in the form of a metric. Each metric is identified by its name and labels.
  • HTTP Server
    • It allows collected time-series data from the time-series database to be available for querying by third-party tools such as Prometheus web UI, Alertmanager, etc. 
    • These tools use a Query language known as PromQL to query metrics.
We get to know that Prometheus scrapes metrics from target endpoints. But the question arises, who collects metrics from target nodes(or machines) and exposes them to the target endpoint? The simple answer is agents.  From use case to use case, there can be different agents such as:
  • Exporters
  • Instrumentation
  • Pushgateway

Exporters ??

Your servers or system or node don't automatically expose the internal data of the system in the consumable format of Prometheus. So, we need some kind of tools that can take internal data, convert it into the format that Prometheus understands, and finally expose it to the "/metrics" endpoint. And that tool is none other than exporters. As the name itself signifies exporting  something.
By definition, an exporter is a piece of program or software that runs in the target machine(or node) and collects data of a service or internal process running in that machine and exposes it.  The exposed data looks as shown in the below figure. 

The popular exporter is "node_exporter". It retrieves data such as  I/O, RAM, CPU, Disk, filesystem usage, Network stats, etc, and exposes them. The exposed data contains 2 fields: HELP(description of what metrics is) and TYPE(type of metric) only. They are plain text and highly readable, which is one of Prometheus's strengths. And these exposed data or metrics are being pulled by Prometheus.


But the problem with the pull-based model is that there are some short-lived jobs that exist for a short period of time, as a result, Prometheus will miss out on pulling the data from it. Therefore, that’s why we have a component called Pushgateway.

Pushgateway ??

In our system, there are short-lived jobs or processes running in the system, as these jobs or process doesn't last long enough time for Prometheus to be able to scrape metrics from them. To avoid losing metrics from these short-lived jobs or processes, the Prometheus team reversed the design from pull-based to push-based. Therefore, short-lived jobs or process will push their metrics to the pushgateway and the pushgateway exposes it to the "/metrics" endpoint. And later on, Prometheus will pull the metric from the "/metrics" endpoint like any other target.

So, till now Prometheus is able to pull metrics from the system, and short-lived process. But what if we want to send metrics from our custom application or custom program written in some programming languages such as golang, python, etc. For that, we require application instrumentation.

Application Instrumentation??

To send metrics from our custom program we need to use Prometheus client libraries to do so. These client libraries allow you to expose any application metrics you need Prometheus to track. Additionally, it also supports exposing metrics and converting in the format that Prometheus understands.
Prometheus has client libraries for various different languages:
  • Golang
  • Java
  • Python
  • Ruby
  • Rust
  • And many more
So, after scrapping metrics from different sources, the retrieval component passes the metric to the database.

Time-series database ??

Data are collected from the target endpoint at a fixed interval of time. The time interval provides the timestamp after every scrapping. When the data and timestamp combine together, they are called metrics.

NOTE:
  • What is meant by time-series data?
    • Every combination of metrics and unique labels (and their value) is going to be one time series.
To store these time-series data or metrics it requires a special type of database known as time-series database. The time-series database has the following things:
  • timestamp
  • Value
  • metric name
We get to know about the time-series database. But where do these are stored and how?? So, these metrics are stored on a local disk or storage. It requires 3 steps to store data locally:
Memory:
  • Firstly, the fresh batch of data is stored in Memory for 2 hours.
  • This reduces disk I/O.
Write Ahead Log(WAL)
  • As we know memory is non-persistent in nature and could lead to loss of the data if the process terminates. To prevent this scenario, a WAL in disk keeps the state of in-memory data so that it can be replayed if Prometheus, for any reason crashes or restarts.
Disk
  • Lastly, after a 2-hour time window, the chunks get written to disk.

HTTP Server
To retrieve these metrics from the time-series database any request has to go through the HTTP server. The HTTP server is responsible for querying the metrics. In order to query, it requires a query language known as PromQL. In short and simple, it accepts PromQL queries from different tools such as Prometheus UI, Alertmanager, Grafana, etc.

Next, we will learn about PromQL language but before jumping into next let's get familiar now with one more component of Prometheus i.e. Service Discovery.

Service Discovery ??

In addition to that, Prometheus expects you to hard code all of the targets that it needs to scrape matrices. To do so, we have to configure a list of targets in the configuration file. However, in environments where target machines are very dynamic in nature. Things like Kubernetes where pods keep spinning up and down, or EC2 instances spinning up and down in case of auto-scaling, in such cases require dynamically updating the list of targets in the configuration file. And that’s where the use of Service discovery comes into play.
Service discovery is all about providing a list of targets for Prometheus to scrap so that you don’t have to hard code the targets every time as the targets dynamically change.

After receiving the metrics from the list of all targets. It is stored in a time series database because these metrics are collected at a fixed time. So, they are classified with time intervals. To store these time-series data, it requires a special kind of time-series database instead of a normal database such as NoSQL, etc. Later on, these metrics will be queried for different purposes such as:
  • Alerting for Alertmanager
  • Prometheus Web UI
  • Visualization tools such as Grafana.
To query these metrics data from the Prometheus HTTP server requires a query language known as PromQL. Everyone whether Prometheus Expression browser, Grafana, or Alertmanager behind the scene uses PromQL to query metrics.

Prometheus Expression Browser ??


And obviously, once Prometheus collects the data, we want to see those data. To do so, we need an interface from which we can query metrics, and that interface is the Prometheus Expression browser.

Alertmanager ??

The primary purpose of alertmanager is to alert before something happens. For say, Disk is filling up and only a few GiBs are left. The primary purpose of the alert is to awaken the developer to take a look into it. Let's see how it is used with Prometheus.

As we have discussed, Prometheus is used to generate alerts. Prometheus is simply responsible for triggering an alert, but not responsible for sending emails, or SMS. However, we need a separate entity called Alertmanager, which handles all of the SMS, the emails, and the SMTP integration. So that it can send the notification to the end user. When an alert is triggered or a defined condition or rule is met, Prometheus is going to push an alert to a process called Alertmanager. Then Alertmanager will be responsible for routing an alert notification to email or Slack or whatever integration you choose.

At a very high level, Alertmanager is a service that receives HTTP POST requests from the Prometheus server via its API.

Alertmanager also has a web interface to allow:

  • The visualization of alerts
  • Silencing of firing alerts
  • Applying inhibition rules.

Next, we will learn about the query language called PromQL, which is used by many third-party tools to query metrics.

Comments

Post a Comment

All Post

Logging ??

Image
 Today we will learn about the following things: What is logging ?? Why do we need logging ?? How can logging help us ?? Why suddenly from the last 5-10 years logging has been evolving ?? Logging tools ?? First thing first, let's start with logging. Logging is human-readable information that is primarily text in nature. Logging is triggered or generated when some kind of event occurs, such event is known as a log event.  Each logging message contains the timestamp, which signifies the time at which the event occurred. Now let, us understand why we need logging.  Importance of logging ?? In today's world software all around the world captures the whole market. By definition, Software is a program that is made up of millions of lines of code.  The programmer while writing the code adds the logs at all places to know the code behaviors as well as unexpected ones.  The logs can be of types such as ERROR, WARN, INFO, FATAL, DEBUG, TRACE, etc.     NOTE : Logs are also added to know w
Read more

Argo CD 101

Image
Why Argo CD ?? Because it fulfills all the points of the GitOps principle . As it extends the benefits of declarative specifications, acting as a GitOps Operator, follows a pull-based approach,  and considers Git as a source of truth for configuration management. It follows a pattern of reconciliation i.e. continuously performing a control loop every 3 minutes to achieve the desired state by fetching the desired configuration from Git. Let’s understand it in more detail, how can we achieve or implement GitOps using Argo CD ?? We all know that every company and every developer uses Git as a source code management software to store code. Developers can commit their infrastructure configurations, such as Kubernetes resource manifest, in Git to create the environment needed for application deployment. Once a developer implements a feature( with new applications and Kubernetes configurations) and merges the main branch. With the help of the GitOps agent or GitOps operator, Argo CD will imme
Read more

What is GitOps in easy way ??

Image
Here we will discuss: "What is GitOps", and "What are its 4 important principles". And lastly, "Why GitOps operator follow a pull-based model ?". What is GitOps ?? GitOps is made up of two words: Git and Ops . Git is where you keep your desired manifest(or configurations) that you want to deploy in your cluster(or environment). And Ops refers to Operation. Operation refers to the environment where the things are operated to ensure that the actual state or live state of that environment must be the same as the desired state in the git. To achieve that from the actual state to the desired state an operation is performed inside the environment.  Such an environment could be a Kubernetes cluster. So, anything that you want to deploy in the Kubernetes simply put it in the git. From git, the GitOps operator(i.e. Argo CD) will deploy to the Kubernetes cluster. To understand the GitOps better we need to understand 4 principles of it: 4 Principles of GitOps: 1.
Read more

Why need of Cloud Computing ??

Image
  Do you recall the result day of your class 10th and 12th? I believe you do. One thing that caught everyone's attention was the server being down when you accessed the website to check the results. Have you ever wondered why the server experienced downtime?   What was the underlying reason for it? Before we proceed, it's essential to grasp the concept of  servers  and  requests . A server can be a single laptop/computer/node or a cluster of laptops/computers that offer specific services. In this scenario, the server was responsible for running the CBSE   board website through which you could access your results. And second thing it was publicly accessible from the Internet. Another important aspect to consider is the concept of a request. A request is like a command that initiates the execution of a program. When you enter your roll number , name , etc., and press the submit button, it counts as one request. This request is sent to the server, prompting it to execute the
Read more

AWS and its Services ??

Image
So, in a previous blog, we learned about cloud computing and its importance . Here we will start with the popular cloud service provider, Amazon Web Services(AWS).  AWS or Amazon Web Services is a cloud service provider that offers various cloud computing services that are accessible via the public internet. The popular services provided by AWS are:  Virtual machine(EC2),  Virtual Network(VPC), Restrict access to cloud resources(IAM), etc.  Executing functions(Lambda),  Automating your infrastructure(CloudFormation),  Object Storage(S3),  Block Storage for Virtual Machines(EBS),  Running and Managing Containers(ECS and fargate),  Network Filesystem(EBS),  SQL database(RDS),    Load Balancer(ELB),   Monitoring and logging(CloudWatch),  We will learn these services one by one.  In this blog, we will cover Virtual Machines (EC2). These services can be accessed in four ways: Console :  access AWS services through a graphical user interface (GUI) CLI :    to manage and access AWS services
Read more

Build a Slack activity dashboard with Metabase

Image
Introduction In this article, we will be leveraging Airbyte - an open-source data integration platform and Metabase - an open-source way for everyone in your company to ask questions and learn from data - to build the Slack activity dashboard above. Airbyte provides us with a rich set of source connectors, and one of those is the Slack connector which allows us to get data off a Slack. We are going to use this connector to get the data of the Slack Workspace and copy them into a Postgres database destination. And then will connect Postgres database with Metabase in order to create the activity dashboard. Pre-requisites: Git Docker Docker Compose Airbyte Postgres Database Metabase Slack Step 1: Replicating data from Slack to Postgres with Airbyte Deploying Airbyte There are several easy ways to deploy Airbyte, as listed here. For this tutorial, I will just use the Docker Compose method from my workstation:  ``` $ git clone https://github.com/airbytehq/airbyte.git $ cd airbyte $ docke
Read more

Observability 101

Image
Hey Everyone !! Good to see you all here to learn Prometheus. This blog is the prerequisite to learning Prometheus.  Prometheus is a part of Observability. Before wasting our time, let's jump into the topic called Observability. Observability Fundamentals What is Observability ?? Observability == Observe + Ability. Observability is made up of 2 words: Observe and Ability .  Observe means to observe the data generated from the system, and the ability  means to measure and understand the system's state based on the generated data.  In the broader sense, Observability gives you the insight of the system's internal working of each component. And that helps while troubleshooting an issue. From where does the concept of Observability come in?? In the early days when applications were deployed in Monolithic structures, humans were able to monitor each component present in the system/server due to the limited number of components. As the microservice application started to boom,
Read more

Monitoring 101

What is Monitoring ?? Monitoring refers to a tool that collects metrics data, provides a dashboard to visualize the collected data, and generates alerts based on alerting rules.  Monitoring tool =~ Metrics + Visualization + Alerting Monitoring can be done in 2 ways: Blackbox Monitoring Whitebox Monitoring BlackBox Monitoring In this type of monitoring, Applications or Systems are monitored/observed from outside via APIs.  Whitebox Monitoring In this type of monitoring, Applications or Systems are monitored or observed from inside. Here, the system itself is under observation which exposes its internal state data and performance data of the system. This type of internal introspection is very powerful because it exposes the operating system telemetry and the health of each different internal component.  So after collecting internal state data, now we need to know,  what are different ways to expose or send these data ? There are 3 different ways to expose the data: Exported through loggi
Read more

Computer Networking 101

Image
Hello Everyone !! I hope you are doing well. We all know the importance of the Internet world. But how does that happen? In the series on Computer Networking we will learn the following topics: What is the Client-Server model? What is Internet Protocol? How does our device connect to Wi-Fi or Hotspot? What happens when you search for www.google.com , www.facebook.com , www.amazon.com , www.linkedin.com , www.github.com , www.youtube.com ,etc?
Read more