Prometheus and Grafana in IBM Cloud Openshift - System requirements

 Introduction

Prometheus is a popular open source monitoring system and Grafana, open source tool compliments it in visualization aspects. Combination of these two tools helps the users to understand the complex data with the help of data metrics of any containerized systems. This combination is also more popular and common monitoring stack used by Devops teams.

Prometheus 

Prometheus is a system to collect and process metrics, not an event logging system. The main Prometheus server runs standalone and has no external dependencies. It collects metrics, stores them, and makes them available for querying, sends alerts based on the metrics collected. The details provided here is tested with 2.20 and higher and may not be applicable to earlier versions. 

Prometheus Concepts

To plan and exercise sizing requirement for Prometheus, below concepts need to be understood first.

TimeSeries is streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Besides stored time series, Prometheus may generate temporary derived time series as the result of queries. Prometheus can handle millions of time series. Memory usage is directly proportional to time series count. A time series is thus represented as a series of chunks, which ultimately end up in a time series file (one file per time series) on disk.

prometheus_local_storage_memory_series: The current number of series held in memory

Scrape: Prometheus is a pull-based system. To fetch metrics, Prometheus sends an HTTP request called a scrape. It sends scrapes to targets based on its configuration.

Metrics & Labels: Every time series is uniquely identified by its metric name and optional key-value pairs called labels. The metric name specifies the general feature of a system that is measured (e.g. http_requests_total - the total number of HTTP requests received).  4 types of metrics are Counter, Gauge, Histogram, & Summary

Labels: enable Prometheus's dimensional data model: any given combination of labels for the same metric name identifies a particular dimensional instantiation of that metric (for example: all HTTP requests that used the method POST to the /api/tracks handler). 

Samples form the actual time series data. Each sample consists of a float64 value and a millisecond-precision timestamp

Instance/Target & Job: In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.

Capacity planning exercise for Prometheus

Planning for sizing predominantely includes Memory usage, Disk usage & CPU usage.

Memory usage There are 2 parts in Memory usage: Ingestion and Query. Both needs to be considered in capacity planning for Prometheus.

Data ingestion: Memory requirement depends on the number of time series, the number of labels you have, and your scrape frequency in addition to the raw ingest rate. Finally this capacity needs to be considereing 50% more for garbage collection overhead.

Query: It is important to consider the concurrency and the complex customized query requirement to query data from Prometheeus. 

Found this online capacity planning calculator helpful in validating your requirements. 

Disk Usage

The Prometheus server will store the metrics in a local folder, for a period of 15 days, by default.Any production-ready deployment requires you to configure a persistent storage interface that will be able to maintain historical metrics data and survive pod restarts.

Prometheus stores its on-disk time series data under the directory specified by the flag storage.local.path (The default path is ./data). The flag storage.local.retention allows you to configure the retention time for samples.

Thumb rule that Prometheus recommends to determine the Disk requirement is calculated as below:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

For example: For 15 days storage 1296000(seconds) * 10000 
(samples/second) * 1.3(bytes/sample) = 16,848,000,000 (bytes). Which 
would be approximately 16 Gigabytes. 

To lower the rate of ingested samples, you can either reduce the number of time series you scrape (fewer targets or fewer series per target), or you can increase the scrape interval. However, reducing the number of series is likely more effective, due to compression of samples within a series.

More details on Prometheus storage can be found here.

Scale out

There are in fact various ways to scale and federate Prometheus. The architecture is to have multiple sharded Prometheis, each scraping a subset of the targets and aggregating them up within the shard. A leader federates the aggregates produced by the shards, and then the leader aggregates them up to the job level.

An interesting read on Scale out is here for further information.

Grafana

Grafana requirement is simple and it just requires minimum 255mb RAM and a single core. You might need a little more RAM if the requirement includes: * Server side rendering of images * Alerting * Data source proxy

The bottleneck for Grafana performance is the time series database backend with complex queries. By default, Grafana comes with SQLite, an embedded database stored in the Grafana installation location.

Error Handling in Application Integration

Error Handling in Application Integration

Application integration not always comes with happy paths irrespective of domains and the mechanism adopted. More and more enterprises are adopting micro services that calls for integration of various applications in real time. Not always this integration result in successful data flow due to various reasons that includes business, application and network errors or limitations. Handling such failures in real time is critical to the functioning of systems and fulfill the assured delivery requirement. Understanding the types of errors/failures, processing, retrying and transforming provides higher rate of success.

The areas or activities that need to be looked into solve this problem are listed below:

• Identifying the errors
• Defining error categories
• Formulating recoverable and non recoverable errors
• Defining workflow steps - Automated or Manual 
• Retry mechanism for recoverable errors
• Persistence logic for long term recovery
• Message reconstruction process
• Defining manual intervention process

Usecase with REST API integration

Error Identification and Categorization

400 category : User Input Error(System related)

•401  - Authentication issue : Retry for getting fresh token. Even after fails, then send alert.

•404 – Retry logic required

•403 – May be one time retry and then Alert

500 category – Internal Server Error (Business validation related)

•For messages, it is important figure out all the codes from target systems.

•Important to find out if the error codes is common or specific to each type of data invalidation in each spoke. We have to start only from the codes. 

Recoverable and non recoverable errors

Errors like temporary network failures, application maintenance downtime can be recoverable. Data validation error requires transformation either automatically or manually depending on the complexity of business rules. Retry logic should be designed such a way to handle these cases individually. 

Retry mechanism for recoverable errors

Picture below illustrates Short term and long term retry logic. 

Circuit breaker design

The simple circuit breaker is used to with short term retry to avoid making the external call when the circuit is open, and the breaker itself should detect if the underlying calls are working again. We can implement this self-resetting behavior by trying the remote call again after a suitable interval, and resetting the breaker if it succeed. This also prevent the unexpected failures with remote calls. 

    

Persistence logic for long term recovery

The error along with the  message need to be temporarily stored in order to resend the message after the correction has been made to the integration flow. The data model should include: 

Configurable fields for Category Definition:

1. Retry attempt – Number

2. Frequency/duration – Number  - to show minutes

3. Alert required – varchar -  to store email

4. Manual Flag – int -  Manual, Auto, None

5. Priority  - int - High, Medium, Low

These are additional message fields:

1. Status – varchar or int to store Pending, Completed, On Going. – (Message Table)

2. Unique id (Trace ID) - required to identify the error apart from message  object so that we can treat each message differently. This has to be primary key.  (Message table)

3. ID column has to be oppty/LI id. 

Defining manual intervention process

There are errors that cannot be resolved automatically by program logic. This kind of error call for manual intervention. For example, if target system is expecting alpha numerical value for certain mandatory field and source system sends numerical value, the synchronization fails in this case. Unless the value is changed to meet the target system, it is not possible to make the flow successful. User intervention is required most of the time to correct the data or system. Every such cases needs be identified, processes to be well defined and messages handled accordingly before resending the message.   

Conclusion

Few other areas that need to be included are Scheduler for triggering the long term retry, message reconstruction process, and  alert/notifying mechanism. With these systems in place, it becomes easy and manageable to handle both expected and unexpected error scenarios. 

Application integration is never complete without robust error handling process and the benefit of that is enormous. Happy to assist further and take your feedback especially the improvements that you could think of. 

Enterprise Integration - Hub & Spoke design

Various business domains in large enterprises using many applications, tools and utilities (ERP, CRM, Sales, HR, Finance etc.) poses bigger challenge to IT to support with effective collaboration across business domains, data analysis and critical data synchronization. With many such heterogeneous applications being added up over time integrating each of them point to point would increase the complexity exponentially. To reduce integration complexity, Hub and Spoke design paradigm is regarded as one of the best solution. The hub part needs to be designed to be highly concurrent, distributed, scalable, micro services oriented, cloud hosted and container orchestrated to offer all the elements of data integration like real-time data streaming, transformation, synchronization, quality, and management to ensure that information is timely, accurate, and consistent across heterogeneous applications.

Event driven design:

Spoke applications triggers messages to hub on its create or update event. Upon receiving such messages and based on the routing rules, hub decides message forwarding to corresponding spokes. Routing rules determines the message flow across spokes systems and these rules relies on trusted data sources for fetching data. The data validation, synchronization, transformation, enrichment, filtering for every message is carried out by integrating hub with trusted data sources. 

Tools & technologies for building hub using IBM public cloud : 

·       The real-time data pipeline is built using Kafka cluster that helps hub handle large amount of data and provides message reliability.

·       To simplify our concurrent code development, and avail infrastructure that allows us to scale without modifying application, Akka toolkit is used. Akka seamlessly handles the distribution of messages and communication in big scale. The other biggest advantage is that it simplifies concurrency logic which in turn improves coding efficiency.

·       Play web server provides lightweight, stateless web server framework and we have chosen this to expose APIs with minimal resource consumption.

·       Redis is used for caching and error handling.

·       Grafana and DataDog are used for monitoring infrastructure and services to update the administrator about system health.

·       Zipkin is used to trace the message flow across systems and provides real time update on the message transition.

Core components that Hub offers are:

·       API service: Exposes integration over REST to provide integration of spokes with hub. Here, interface contract is established to standardize the communication from spoke to hub. Provides HTTPS endpoint and OAuth2.0 for securing the communication.

·       Processor Service: Validates, standardize, enriches, filters and transforms the incoming data. Applies message routing rules to determine destination spoke for incoming messages.

·       Persistence Service: Provides data persistence to store every transaction that flows through the hub. DashDB is being used to store consolidated data.

·       Adapter Service: Service to consume the spoke interfaces is provided by adapter service. This generalized service gets derived by individual spoke adapter to be in compliant with its own interface.

·       Dashboard service:  Provides consolidated report on the opportunity messages. Helps support and administration staff with the message transaction details. 

Real time integration exception handling and assured delivery:

It is critical to handle extraneous conditions arising from unexpected events, invalid data and process errors at runtime. The ecosystem must adapt multi-pronged approach to handle various types of errors with assured delivery, error routing, error mapping, logging, notification, and dashboard monitors for error details. Robust retry framework is used to automatically resolve recoverable errors. Hub forks workflow process between recoverable and non-recoverable exceptions to manage them appropriately.

Circuit breaker mechanism is adopted to handle remote calls efficiently. This provides us lot of savings in terms of optimal resource usage and mitigates failure cascading across systems.