Publishing a package
Notice: Under Development
Reusable, customizable components can be built and shared as packages.
Overview
Packages are a pattern for developing reusable, customizable configuration. Packages are typically published and consumed by different teams.
Because packages can be updated to new versions, consumers can pull in changes to a package after fetching it.
Example use cases for packages
- Languages: Java / Node / Ruby / Python / Golang application
- Frameworks: Spring, Express, Rails, Django
- Platforms: Kubeflow, Spark
- Applications / Stacks:
- Rails Backend + Node Frontend + Prometheus
- Spring Cloud Microservices (discovery-server, config-server, api-gateway, admin-server, hystrix, various backends)
- Infrastructure Stacks: CloudSQL + Pubsub + GKE
# Optional: copy the mysql-kustomize package to follow along
kpt pkg get https://github.com/GoogleContainerTools/kpt.git/package-examples/mysql-kustomize mysql
Factor / denormalize the configuration data
Structuring packages into separate publisher and consumer focused pieces provides a clean UX for consumers to modifys the package.
Example: provide separate directories with pieces consumers are expected to edit (replicas) vs publisher implementation (health check command).
As a package publisher, it is important to think about where and how you want to promote customization.
We will use kustomize to structure the package:
- Factoring out a common field value
- Example:
namespace
,commonLabels
,commonAnnotations
- Example:
- Factoring a single resource into multiple files
- Example:
resources
+patches
- Example:
Remote kustomize bases may be used to reference the publisher focused pieces directly from a git repository rather than including them in the package.
One disadvantage of this approach is that it creates a dependency on the remote package being accessible in order to push – if you can’t fetch the remote package, then you can’t push changes.
Example package structure:
$ tree mysql/
mysql/
├── Kptfile
├── README.md
├── instance
│ ├── kustomization.yaml
│ ├── service.yaml
│ └── statefulset.yaml
└── upstream
├── kustomization.yaml
├── service.yaml
└── statefulset.yaml
The upstream
directory acts as a kustomize base to the instance
directory.
Upstream contains things most consumers are unlikely to modify –
e.g. the image (for off the shelf software), health check endpoints, etc.
The instance
directory contains patches with fields populated for things
most consumers are expected to modify – e.g. namespace, cpu, memory,
user, password, etc.
While the package is structured into publisher and consumer focused pieces, it is still possible for the package consumer to modify (via direct edits) or override (via patches) any part of the package.
Factoring is for UX, not for enforcement of specific configuration values.
Commands, Args and Environment Variables
How do you configure applications in a way that can be extended or overridden – how can consumers of a package specify new args, flags, environment variables or configuration files and merge those with those defined by the package publisher?
Notes
- Commands and Args are non-associative arrays so it is not possible to target specific elements – any changes replace the entire list of elements.
- Commands and Args are separate fields that are concatenated
- Commands and Args can use values from environment variables
- Environment variables are associative arrays, so it is possible to target specific elements within the list to be overridden or added.
- Environment variables can be pulled from ConfigMaps and Secrets
- Kustomize merges ConfigMaps and Secrets per-key (deep merges of the values is not supported).
- ConfigMaps and Secrets can be read from apps via environment variables or volumes.
Flags and arguments may be factored into publisher and consumer focused pieces
by specifying the command
in the upstream
base dir and the args
in the
instance
dir. This allows consumers to set and add flags using args
without erasing those defined by the publisher in the command
.
When specifying values for arguments or flag values, it is best to use an environment variable read from a generated ConfigMap. This enables overriding the value using kustomize’s generators.
Example: Enable setting --skip-grant-tables
as a flag on mysql.
# {"$ref": ...
comments are setter references, defined in the next section.
# mysql/instance/statefulset.yaml
# Wire ConfigMap value from kustomization.yaml to
# an environment variable used by an arg
apiVersion: apps/v1
kind: StatefulSet
...
spec:
template:
metadata:
labels:
app: release-name-mysql
spec:
containers:
- name: mysql
...
args:
- --skip-grant-tables=$(SKIP_GRANT_TABLES)
...
env:
- name: SKIP_GRANT_TABLES
valueFrom:
configMapKeyRef:
name: mysql
key: skip-grant-tables
# mysql/instance/kustomization.yaml
# Changing the literal changes the StatefulSet behavior
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: mysql
behavior: merge
literals:
# for bootstrapping the root table grants -- set to false after bootstrapped
- "skip-grant-tables=true" # {"$kpt-set":"skip-grant-tables"}
Generating ConfigMaps and Secrets
Kustomize supports generating ConfigMaps and Secrets from the kustomization.yaml.
- Generated objects have a suffix applied so that the name is unique for the data. This ensures a rollout of Deployments and StatefulSets occurs.
- Generated object may have their values overridden by downstream consumers.
Example Upstream:
# mysql/upstream
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: mysql
literals:
- skip-grant-tables=true
Example Instance:
# mysql/instance
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: mysql
behavior: merge
literals:
- "skip-grant-tables=true" # {"$kpt-set":"skip-grant-tables"}
- "mysql-user=" # {"$kpt-set":"mysql-user"}
- "mysql-database=" # {"$kpt-set":"mysql-database"}
Setters and Substitutions
It may be desirable to provide user friendly commands for customizing the package rather than exclusively through text editors and sed:
- Setting a value in several different patches at once
- Setting common or required values – e.g. the image name for a Java app package
- Setting the image tag to match the digest of an image that was just build.
- Setting a value from the environment when the package is fetched the first time – e.g. GCP project.
Setters and substitutions are a way to define user and automation friendly commands for performing structured edits of a configuration.
Combined with the preceding techniques, setters or substitutions can be used
to modify generated ConfigMaps and patches in the instance
dir.
See the setter and substitution guides for details.
# mysql/instance/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
#
# namespace is the namespace the mysql instance is run in
namespace: "" # {"$kpt-set":"namespace"}
configMapGenerator:
- name: mysql
behavior: merge
literals:
# for bootstrapping the root table grants -- set to false after bootstrapped
- "skip-grant-tables=true" # {"$kpt-set":"skip-grant-tables"}
- "mysql-user=" # {"$kpt-set":"mysql-user"}
- "mysql-database=" # {"$kpt-set":"mysql-database"}
...
# mysql/instance/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
template:
spec:
containers:
- name: mysql
...
ports:
- name: mysql
containerPort: 3306 # {"$kpt-set":"port"}
resources:
requests:
cpu: 100m # {"$kpt-set":"cpu"}
memory: 256Mi # {"$kpt-set":"memory"}
# mysql/instance/service.yaml
apiVersion: v1
kind: Service
...
spec:
ports:
- name: mysql
port: 3306 # {"$kpt-set":"port"}
targetPort: mysql
Updates
Individual directories may have their own package versions by prefixing the
version with the directory path – e.g.
package-examples/mysql-kustomize/v0.1.0
.
When publishing a new version of a package, publishers should think about how their changes will be merged into existing packages.
Changing values in the instance package is not recommended, but adding them may be ok – changes to fields will overwrite user changes to those same fields, whereas adds will only conflict if the user added the same field.