Project

Objective

As explained in the internship topics, the goal is to design a file distribution mechanism for Kubernetes clusters that keeps the qualities of a configMap, but adds :

  • Efficient distribution of large files
  • Target is to distribute files up to 1GB max, contrary to the 1MB max on configMap
  • End-to-end encryption during the decryption of data
  • Target is to ensure that data cannot be modified (signing) and/or read (encryption) during its distribution from the source to the applications consuming it.

The qualities of a ConfigMap that we want to preserve are :

  • Applications should be able to link to a specific version of the data (or even host this version of data) within their gitOps repository.
  • The gitOps repository should indicate which version of data is currently deployed (or at least "eventually deployed") on the application pods.
  • It should be possible to trigger a fallback by reverting this a git commit.
  • Data should be presented to applications as a standard filesystem.
  • End-user applications should just see the files as a file system folder, check for file updates with filemtime, and read them. Any other process (retrieval, rebuilding, decryption of data) must be hidden behind the filesystem layer.
  • Data should be refreshable while pods are running.
  • No pod restart required to see updated data.
  • Caching should be at node level.
  • To avoid duplicated memory usage if several pods load the same files.

High level design

The following diagram presents the general GitOps workflow for deploying applications and distributing ConfigMaps to a Kubernetes cluster.

The yellow elements are the parts that need to be specified and implemented as part of this internship. Specifically :

  • Decide on a root data source to hold the versioned data. ConfigMaps use the main GitOps repository, but for performance reasons it's not recommended to store large files in a frequently manipulated repository. Here, it could be a separate git repository (if file is text content, it can provide very efficient diffs), or completely independent storage.

  • Define a custom resource, to place in the GitOps repository to reference a specific version of data from the data source.

  • Build a virtual file system that allows accessing the data specified in the GitOps repo, effortlessly from application containers.

    This requires also defining how this volume retrieves data from the data source, which is where the requirements of this internship become tangible : How to efficiently distribute changes to a big file without redistributing the full big file every time ? How to ensure that the data is encrypted throughout this distribution chain, and decrypted only when exposed on the container's local file system ?

Image

High level roadmap

Image