An Application Pipeline, often referred to as a deployment pipeline, is a set of automated processes and stages that software code goes through to move from development to production. The primary goal of an application pipeline is to streamline and automate the software delivery process, enabling efficient and reliable deployment of applications. The pipeline consists of various stages, each serving a specific purpose in the software development and release lifecycle.

Application pipelines facilitate a continuous delivery and integration (CI/CD) approach, promoting automation, collaboration, and rapid iteration in the software development process. They are essential for delivering high-quality software with efficiency and reliability.

Posts

Content-aware auto-scaling of stream processing applications on container orchestration platforms

/in /by

Modern applications are designed as an interacting set of microservices, and these applications are typically deployed on container orchestration platforms like Kubernetes. Several attractive features in Kubernetes make it a popular choice for deploying applications, and automatic scaling is one such feature. The default horizontal scaling technique in Kubernetes is the Horizontal Pod Autoscaler (HPA). It scales each microservice independently while ignoring the interactions among the microservices in an application. In this paper, we show that ignoring such interactions by HPA leads to inefficient scaling, and the optimal scaling of different microservices in the application varies as the stream content changes. To automatically adapt to variations in stream content, we present a novel system called DataX AutoScaler that leverages knowledge of the entire stream processing application pipeline to efficiently auto-scale different microservices by taking into account their complex interactions. Through experiments on real-world video analytics applications, such as face recognition and pose classification, we show that DataX AutoScaler adapts to variations in stream content and achieves up to 43% improvement in overall application performance compared to a baseline system that uses HPA.

Read more