Implementation of New Security Features in CMSWEB Kubernetes Cluster at CERN

TL;DR

The CMSWEB Kubernetes cluster hosts critical CMS services and faces security risks from permissive pod-to-pod communication. The authors implement three security features: network policies to limit intra-cluster traffic, OPA Gatekeeper with Helm-deployed constraint templates for policy enforcement, and Vault with a Vault Agent Injector to securely manage secrets. Key contributions include detailing the Helm-based deployment of OPA policies, the integration of Kubernetes admission controllers via Gatekeeper, the Vault deployment and injector workflow, and the secret-management automation script. The results provide a more secure, auditable, and scalable CMSWEB Kubernetes environment with strengthened access control and centralized secret storage.

Abstract

The CMSWEB cluster is pivotal to the activities of the Compact Muon Solenoid (CMS) experiment, as it hosts critical services required for the operational needs of the CMS experiment. The security of these services and the corresponding data is crucial to CMS. Any malicious attack can compromise the availability of our services. Therefore, it is important to construct a robust security infrastructure. In this work, we discuss new security features introduced to the CMSWEB Kubernetes ("k8s") cluster. The new features include the implementation of network policies, deployment of Open Policy Agent (OPA), enforcement of OPA policies, and the integration of Vault. The network policies act as an inside-the-cluster firewall to limit the network communication between the pods to the minimum necessary, and its dynamic nature allows us to work with microservices. The OPA validates the objects against some custom-defined policies during create, update, and delete operations to further enhance security. Without recompiling or changing the configuration of the Kubernetes API server, it can apply customized policies on Kubernetes objects and their audit functionality enabling us to detect pre-existing conflicts and issues. Although Kubernetes incorporates the concepts of secrets, they are only base64 encoded and are not dynamically configured. This is where Vault comes into play: Vault dynamically secures, stores, and tightly controls access to sensitive data. This way, the secret information is encrypted, secured, and centralized, making it more scalable and easier to manage. Thus, the implementation of these three security features corroborate the enhanced security and reliability of the CMSWEB Kubernetes infrastructure.

Figures (5)

• Figure 1: Network policy of crab namespace in CMSWEB Test environment
• Figure 2: Workflow of Request when OPA is in place
• Figure 3: Deployment of OPA policies with Helm
• Figure 4: Working of Vault when Integrated with Kubernetes
• Figure 5: Workflow of Vault Script