Recommendations

One port, the one that serves traffic. Everything operability lives under /-/, which is itself an index of the rest. Restrict access to /-/ by path at the ingress, except /-/up, which is public.

Wire the platform’s checks up like this:

What fails what:

/-/status reports the worst status of any check, so it is the one that tells the truth about the instance. A dependency that every request needs makes it fail; one that only some routes need makes it warn, and the check lists those routes in affectedEndpoints. /-/up is the same fold with the checks stripped, so a third party can watch it without being told anything but up or down.

Checks run on a timer, not per request. Every endpoint serves the latest snapshot, and each check’s time says when it was taken.

Dependency failures are handled in the application, by returning 503 with Retry-After on the routes that need the dependency, not by the infrastructure taking the instance out.

503 is the failure code because every caller agrees on it: the kubelet fails on anything outside 200-399, the ALB matcher defaults to 200, and curl -f exits non-zero on 400 and above. In practice a broken process more often times out than answers 503, so probe timeouts matter as much as the response.

/-/status on an instance whose database is unreachable looks like:

{
  "status": "fail",
  "checks": {
    "event-loop:lag": [
      { "status": "pass", "observedValue": 3, "observedUnit": "ms", "time": "2026-09-04T10:42:14Z" }
    ],
    "lifecycle:phase": [
      { "status": "pass", "observedValue": "serving", "time": "2026-09-04T10:42:14Z" }
    ],
    "jvm:heap": [
      { "status": "warn", "observedValue": 91, "observedUnit": "percent", "time": "2026-09-04T10:42:14Z" }
    ],
    "postgres:connectivity": [
      { "status": "fail", "componentType": "datastore", "output": "connection refused", "time": "2026-09-04T10:42:11Z" }
    ]
  }
}

Keys in checks are component:measurement. Each maps to an array, since a component may be observed more than once. The top-level status is the worst of the checks. /-/live here would return only event-loop:lag, and /-/ready only lifecycle:phase, both pass.

/-/info looks like:

{
  "service": {
    "name": "orders",
    "version": "2.14.0",
    "instance": { "id": "orders-7c9d8f-x2kq" }
  },
  "deployment": { "environment": { "name": "prod" } },
  "vcs": { "ref": { "head": { "revision": "a3f9c1e" } } },
  "process": { "runtime": { "name": "OpenJDK Runtime Environment", "version": "21.0.4" } }
}

Content negotiation

Nice to have. Every endpoint honours Accept:

The HTML and Markdown renderings list every application/* media type the endpoint serves, so a human who finds the page knows what a machine should ask for. The HTML page also includes the JSON of the same response in a folded <details> section, which needs no script. If Accept matches none of these, serve the default rather than 406.

/-/metrics differs in two rows. text/plain means the Prometheus text exposition, because that is what scrapers ask for: Prometheus sends application/openmetrics-text;version=1.0.0, text/plain;version=0.0.4;q=0.5, */*;q=0.1, and older scrapers send text/plain or nothing. For the same reason the default is the text exposition, not JSON. There is no standard pull-format JSON for metrics; the nearest is the OTLP/HTTP JSON encoding, which is a push payload served off-label, and is what application/json returns.

Why

The health checks that will call you

Two mechanisms are at work: restart checks (Docker, ECS, liveness) and routing checks (ALB, readiness). Only K8S keeps them cleanly separate.

Some wrinkles:

• Docker’s HEALTHCHECK on its own restarts nothing. It flips a flag that something else - Swarm, Compose depends_on: condition: service_healthy, an autoheal sidecar, ECS - has to act on.
• ALB checks are routing checks, but ECS treats an ALB-unhealthy target as a failed task and replaces it. So on ECS a routing check becomes a restart check, which is why the wiring table above sends the ALB to /-/live on ECS.
• Docker and ECS run the check inside the container, so the image needs curl or equivalent. ALB and the kubelet probe over the network, so they also prove the port is bound and reachable.
• K8S’s startup probe has rough equivalents in ECS’s startPeriod on the container health check and healthCheckGracePeriodSeconds on the service for ALB checks.

The consequence: a check that fails when a downstream dependency is down is appropriate as a routing check and harmful as a restart check, since restarting will not fix the dependency. On ECS, where routing checks trigger restarts, a dependency outage cascades into every task being cycled.

What can actually go wrong

Four kinds of unhealthy, and they want different responses.

Broken - deadlock, leaked memory, wedged event loop. Only a restart will fix it. Fail everything.

Lifecycle - starting up, draining on shutdown, not the leader. Instance-specific and will resolve itself. Fail routing, pass restart.

Under pressure looks like a routing failure - take the hot instance out until it cools. But if load is even, every instance is hot. Failing readiness across the fleet leaves the Service with no endpoints, and each instance that drops out sends more load to the survivors. Readiness is a tool for one hot instance; autoscaling is the tool for a hot fleet. On ECS the question is moot: the ALB check is the only routing check and ECS turns it into a restart check, so pressure must pass everything.

Dependency down is worse. It is fleet-wide by nature, so failing readiness everywhere means callers get connection refused instead of a meaningful 503, new pods never go ready so unrelated rollouts stall, and when the dependency returns the fleet is not already up and waiting. Instead: return 503 with Retry-After on the routes that need the dependency, keep serving the ones that do not, and report dependency status on /-/status for humans and alerting. There, a dependency needed by every request is fail and one needed by some routes is warn: a SaaS control plane that proxies to per-customer infrastructure is down when its database is down, not when one customer’s proxy target is. Nothing an orchestrator acts on should include dependency checks. Keeping /-/status separate is what stops someone wiring a DB check into the liveness probe later because it was already there.

The exception is a per-instance dependency - a sidecar, a local cache, a node that cannot reach the DB while its peers can. That is a genuine readiness case, but from inside the process it is indistinguishable from the dependency being down for everyone. Only monitoring, which sees the whole fleet, can tell the two apart. So leave it out of readiness and alert on it instead.

Which means that for a stateless service, readiness is mostly about lifecycle: the start and end of an instance’s life, plus a handful of stateful special cases (leader election, rebalancing, deliberate withdrawal for debugging).

One port

A separate management port was for putting operability traffic on a different network interface, which meant something on a rack server and means nothing in a pod or an awsvpc task, where every port shares one network namespace. Worse, a separate connector usually has its own acceptor and thread pool, so it can answer “alive” while the listener that actually takes traffic is deadlocked. Restrict access by path at the ingress instead.

The prefix

Group the endpoints under a prefix, so a single ingress rule covers them and they cannot collide with application routes. Spring uses /actuator, a control-theory word for the part that acts on a system, which nobody outside Spring recognises. Quarkus uses /q. Prometheus, Alertmanager, Thanos, Loki and GitLab use /-/, which is the one with prior art outside a single framework.

Body formats

There is no ratified standard for a health check body. The nearest is the IETF draft “Health Check Response Format for HTTP APIs”, which reached version 06 in 2021 and expired. It defines application/health+json with status of pass, warn or fail, and a checks map of named sub-checks each with their own status, observedValue, observedUnit, time and output. The media type was never registered with IANA, but libraries in most languages implement the draft and it is the format people reach for. The alternative is Eclipse MicroProfile Health, an actual spec, but it is application/json with UP/DOWN and no warn, and warn is exactly what /-/status needs for pressure and dependency states.

All four health endpoints return the same shape: a status and the checks that produced it. They differ only in which checks are included. /-/live folds the liveness checks, /-/ready the lifecycle checks, /-/status everything, and /-/up is /-/status with checks omitted. Including checks in the probe responses is not just for humans: the kubelet records the response body of a failed httpGet probe in the pod’s events, so kubectl describe pod shows which check failed rather than just “503”.

Public status

An SLA needs a third-party monitor, and a third-party monitor is far easier to run against an unauthenticated URL. But /-/status says which database is down and how full the heap is, which is not for strangers. Path-based authorisation at the ingress can allow or deny a path; it cannot strip a body. So the public view has to be a second path: /-/up, the same fold as /-/status with everything but the top-level status removed.

Anything public under /-/ must not do work per request. An unauthenticated endpoint that pings the database on every hit is an amplification vector. So checks run on a timer and the endpoints serve the latest snapshot, which is what the draft’s per-check time field is for. That is good practice for the protected endpoints too.

/-/ is an index so the rest are discoverable, by tooling and by a human with curl. HAL is another expired IETF draft, but it is what Spring’s /actuator index returns and every language has a client for it.

For metrics, OpenMetrics is the Prometheus exposition format formalised. Scrapers content-negotiate; an old one will ask for text/plain; version=0.0.4 and should get it.

Info format

There is no standard for what an info endpoint returns. Spring’s is freeform, populated by whatever contributors are on the classpath. But there is a standard vocabulary for describing a running process: the OpenTelemetry Resource semantic conventions. service.name, service.version, service.instance.id, deployment.environment.name, process.runtime.name, container.image.tags, k8s.pod.name, cloud.region, vcs.ref.head.revision. Every span and metric the process emits already carries these, so /-/info has a precise definition: the Resource this process attaches to its telemetry. What it returns must match what you see in your tracing UI, which is a useful thing to be able to check.

Serialise the SDK’s Resource with the dotted keys nested, which is how Datadog’s span overview displays them. Nesting is safe because the conventions forbid a namespace also being a key. Two attributes to filter out: process.command_line and process.executable.path are populated by the standard detectors and can leak secrets.

In order to expose a K8S cluster running on AWS EC2 instances (as an EKS cluster does) you will need an ELB in some form to load balance across the EC2 instances hosting the cluster. The ELB can either be a K8S ingress or it can be a route to the K8S ingress - that is, the actual ingress running inside the K8S cluster is exposed on ports on the EC2 instances, and the ELB balances across those nodes.

Types of AWS Load Balancer Controller

There are two types of AWS Load Balancer Controller.

1. The legacy in-tree controller

   This is baked into an EKS kubernetes cluster without any installation. You do not see any Pods or Service or Ingress Class for it as resources inside the cluster.

   It will create an NLB for any K8S Service with spec: type: LoadBalancer which will proxy to that service.

   It is legacy and deprecated.

2. The AWS Load Balancer Controller

   This is an external controller. It can be installed via helm. It will create resources (a Deployment, Pods, an Ingress Class) in the cluster.

   It will create an NLB for any K8S Service with spec: type: LoadBalancer which will proxy to that service.

   It will create an ALB for any K8S Ingress with spec: ingressClassName: alb.

   It is the recommended way to integrate with AWS load balancers.

   Prior to v2 it was called the AWS ALB Ingress Controller which is deprecated.

   Installation is documented here.

Differences between the Controller types

Importantly, many of the documented annotations are not supported by the legacy in-tree controller. This includes the service.beta.kubernetes.io/aws-load-balancer-proxy-protocol we will discuss below.

Some of the documented annotations have different default values. Notably service.beta.kubernetes.io/aws-load-balancer-scheme (whether the provisioned ELB should be internal or internet-facing) defaults to internet-facing on the AWS Load Balancer Controller but internal on the legacy in-tree controller.

The Ingress-Nginx Controller

The Ingress-Nginx Controller is probably the preferred K8S Ingress Class when running on AWS. It allows you to manage your ingress costs inside the cluster, by making your K8S Services have spec: type: ClusterIP and exposing them via an Ingress with spec: ingressClassName: nginx, whereas using a K8S Service with spec: type: LoadBalancer or an Ingress with spec: ingressClassName: alb will create further costly AWS ELBs. A single NLB can then act as the front door to the Ingress-Nginx Controller.

Installation of the Ingress-Nginx Controller creates, among other things, a K8S Service and Deployment. The K8S Service will have spec: type: LoadBalancer, which on AWS with the AWS Load Balancer Controller installed will provision an NLB to proxy traffic to the exposed port(s) on the EC2 instances, which in turn will forward that traffic to the nginx pods.

Configuring the provisioned NLB

The nature of the provisioned NLB can be controlled by adding AWS annotations to the nginx K8S Service. For our purposes important values are:

• service.beta.kubernetes.io/aws-load-balancer-scheme

  Not a URI scheme, instead whether the provisioned ELB should be internal or internet-facing.

• service.beta.kubernetes.io/aws-load-balancer-ssl-cert

  The ARN of an AWS Certificate Manager TLS certificate that will be served by the NLB to allow it to serve https requests.

• service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'

  If present, the generated Target Groups will be configured to send the Proxy Protocol v2 header.

Installing the Ingress-Nginx Controller

Via Manifest

The Ingress-Nginx team provide a manifest for installing the controller with TLS terminated at the NLB. It requires setting a couple of values, documented in the linked installation instructions. However, it has several quirks to be aware of:

1. It does not enable Proxy Protocol v2 on either the NLB or nginx

   See Enabling Proxy Protocol v2

2. It only creates one replica of the nginx pod

   As the pods are exposed on the EC2 instances forming the cluster’s nodes as the registered targets of the NLB’s target group(s), this means that all but one of the registered targets will be considered unhealthy.

   This can be fixed by adding spec: replicas: 2 (or however many EC2 nodes you have) to the Deployment manifest.

3. It does not specify a value for service.beta.kubernetes.io/aws-load-balancer-scheme

   So whether your NLB is internet facing will depend on the default, which differs per AWS controller type.

   This can be fixed by adding metadata: annotation: service.beta.kubernetes.io/aws-load-balancer-scheme to the Service manifest.

4. It takes control of redirecting http to https, preventing management of this at the Ingress resource level

   Full discussion below.

Enabling Proxy Protocol v2

Proxy Protocol v2 can be enabled as so:

1. On the NLB side by adding metadata: annotation: service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' to the Service manifest
2. On the nginx side by adding data: use-proxy-protocol: "true" to the ConfigMap manifest

http to https redirection

The nlb-with-tls-termination manifest implements http to https redirection by exposing an extra port 2443 named tohttps on the nginx Pods, adding a special listener on port 2443 to nginx via data: http-snippet: on the ConfigMap that always returns a 308 to https, and then specifying that the http port on the Service manifest has a target port of tohttps, which configures the NLB’s Target Group for port 80 to route traffic to port 2443 on the nginx pod (ultimately - there is a further port mapping on the EC2 instance to get to 2443).

Given that it does not enable Proxy Protocol v2, it makes sense for the manifest to do this, as most services that serve over TLS will want a redirect from http to https and without the Proxy Protocol nginx cannot know what the original protocol was. An NLB cannot do the redirect, so it has to happen at the nginx level.

This breaks if you enable Proxy Protocol v2, because the http-snippet on the ConfigMap will not be updated to expect the proxy protocol header despite adding data: use-proxy-protocol: "true" to the ConfigMap resource.

Redirecting via a custom port is unnecessary if you have a spec: tls: hosts array on the Ingress resource AND you have Proxy Protocol v2 enabled (so that nginx knows if the request were actually via https), as that will configure nginx to send redirects from http to https for all rules in that Ingress resource by default, and will also allow you to disable this behaviour per Ingress by adding metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: "false" to the Ingress resource.

The Ingress redirect only works if you specify a spec: tls: hosts array on the Ingress resource. I’m not sure if this is an odd thing to do if TLS is being terminated downstream of the ingress at the NLB.

If you need precise control over which services have and do not have http to https redirects, it can be achieved with the following steps:

1. Enable Proxy Protocol v2.
2. Change the Service manifest - find the spec: ports port with name http and change its targetPort from tohttps to http.
3. Change the Deployment manifest - find the spec: template: spec: containers container with name controller. Delete the port with name tohttps.
4. Change the ConfigMap manifest - delete the data: http-snippet.
5. Make any Ingress resources you have that should redirect from http to https have a spec: tls: hosts containing all the hosts they serve, and metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: "false" if they should serve over http.

Other options I have not yet explored in full:

1. Terminate TLS at nginx rather than the ELB. Then it makes sense to have spec: tls: hosts on the Ingress resources, and there is no need to worry about Proxy Protocol v2. Requires getting the certificate into K8S so nginx can serve it.
2. Use an ALB rather than an NLB by adding metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-type: alb to the Service manifest. This should remove the need to have Proxy Protocol v2 enabled.

Via Helm

There is an Ingress-Nginx Controller Helm chart. I have not experimented with it yet but it may be possible to use it to avoid tweaking a downloaded manifest.

Proxy Protocol v2

Typically, a network layer 4 device like an NLB cannot provide any further information to the upstream services to which it proxies. The bits are simply sent “as is”. This is a problem with TLS terminated at the NLB, because a typical HTTP request does not contain the requested scheme within the body of the request. Consequently, the fact that the original scheme was https is lost to upstream services. Upstream services may need to construct URLs with that knowledge.

The Proxy Protocol v2 specifies a way for Layer 4 devices like an NLB to provide proxy information to upstream services via a header sent before the rest of the request (including before the request line). The Target Groups for AWS NLBs can be configured to send it (in the console, edit the Target Group’s “Attributes”), and nginx can be configured to expect it.

Ideally applications do not need this information - they should return links as URI references rather than URLs, either omitting the scheme (e.g. //example.com/my/thing) or the entire authority (e.g. /my/thing). However, some frameworks insist on returning URLs for e.g. the Location header, reconstructing the scheme from the X-Forwarded-Proto or Forwarded header, falling back on the scheme the process is listening for, because previous (obsolete) versions of the HTTP specification required the Location header to contain a URL.

If http to https redirection is expected to happen upstream of an NLB then proxy information will be required.

Using Clikt as a pure parser

If all you want is to turn command line arguments into a data class, without using the Clikt Command abstraction, you can do it as follows:

import com.github.ajalt.clikt.core.NoOpCliktCommand
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.options.required

fun main(vararg args: String) {
  val config: Config = CliParser.parseConfig(*args)
}

data class Config(val username: String)

class CliParser private constructor() : NoOpCliktCommand(
  name = "",
  help = "help text",
) {

  private val username: String by option(help = "the username").required()

  private fun toConfig() = Config(username = username)

  companion object {

    fun parseConfig(vararg args: String): Config =
      CliParser()
        .apply { parse(args.toList()) }
        .toConfig()
  }
}

Injecting the environment as a map

If you want to decouple Clikt from System.getenv you can do so as so:

class CliParser(
  env: Map<String, String> = System.getenv()
) : CliktCommand {

  init {
    context {
      envarReader = env::get
      autoEnvvarPrefix = "MY_APP"
    }
  }
}

Putting them together

import com.github.ajalt.clikt.core.NoOpCliktCommand
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.options.required

fun main(vararg args: String) {
    val config: Config = CliParser.parseConfig(*args)
}

data class Config(val username: String)

class CliParser private constructor(
  env: Map<String, String>
) : NoOpCliktCommand(
  name = "",
  help = "help text",
) {

  init {
    context {
      envarReader = env::get
      autoEnvvarPrefix = "MY_APP"
    }
  }

  private val username: String by option(help = "the username").required()

  private fun toConfig() = Config(username = username)

  companion object {

      fun parseConfig(
          args: List<String>,
          env: Map<String, String> = System.getenv(),
      ): Config =
          CliParser(env)
              .apply { parse(args) }
              .toConfig()

    fun parseConfig(
        vararg args: String,
        env: Map<String, String> = System.getenv(), 
    ): Config = parseConfig(
        args.toList(), 
        env,
    )
  }
}

Edit 2023-02-06 - we’ve made a couple of improvements:

• It’s now invariant in V, the type of the values, so an AbstractPropertyMap<String> is a subtype of AbstractPropertyMap<CharSequence>
• The type of the properties is now captured by the type of the argument to property, so in an AbstractPropertyMap<CharSequence> val aString by property("a string") will have its type inferred as String. (This is obviously useful if extending AbstractPropertyMap<Any>.)

Create this abstract class:

abstract class AbstractPropertyMap<out V>(
  private val properties: MutableMap<String, V> = mutableMapOf()
) : Map<String, V> by properties {
  protected fun <V2 : @UnsafeVariance V> property(initialValue: V2) =
    PropertyDelegateProvider<Any, ReadOnlyProperty<Any, V2>> { _, prop ->
      properties[prop.name] = initialValue
      ReadOnlyProperty(properties::getValue)
    }
}

You can then create subclasses as so:

class Links(
  id: String,
  otherId: String,
) : AbstractPropertyMap<URI>() {
  val self by property(URI.create("/v1/$id"))
  val other by property(URI.create("/v1/other/$otherId"))
}

val links = Links("1", "2")

links.self == URI.create("/v1/1")
links["self"] == links.self

links.other == URI.create("/v1/other/2")
links["other"] == links.other

links.keys == setOf("self", "other")
links.values.toList() == listOf(links.self, links.other)
links.entries == setOf(
  SimpleImmutableEntry("self", links.self),
  SimpleImmutableEntry("other", links.other)
)

Which begs the question - why?

I think the resulting class has a few nice properties:

1. Links is a Map<String, URI>, so you can retrieve values with keys only known at runtime and iterate over its entry set / key set / values without needing reflective access.
2. You only specify the key names once - no duplication, no room for error.
3. The compiler forces you to instantiate Links with all the expected keys.
4. The compiler enforces the type of all the properties created by delegation
5. You have type safe access to the properties on the resulting instance

In contrast, a mapOf<String, URI>() lacks 3 & 5. A simple class lacks 1 & 4.

A combination of the two could be made without the delegation magic, but would be more noisy and require duplicating key names:

class Links(
    id: String,
    otherId: String,
) : AbstractMap<String, URI>() {
    val self = URI.create("/v1/$id")
    val other = URI.create("/v1/other/$otherId")

    override val entries: Set<Map.Entry<String, URI>> = setOf(
        AbstractMap.SimpleImmutableEntry("self", self),
        AbstractMap.SimpleImmutableEntry("other", other),
    )
}

Fuller details can be found in GitHub’s documentation About security hardening with OpenID Connect and specifically for AWS at Configuring OpenID Connect in Amazon Web Services

Steps:

In AWS IAM

1) Add an Identity Provider in AWS IAM

• Provider: token.actions.githubusercontent.com
• Audience: sts.amazonaws.com

2) Create an IAM Role

3) Under Permissions add a policy granting whatever it is the role needs to do (can be an inline policy, whatever - this is a standard IAM role, specific to what you need to do, nothing to do with it being GitHub Actions that will do it).

4) Under Trust relationships add something like this:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "<ARN of the token.actions.githubusercontent.com Identity Provider>"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
            },
            "StringLike": {
              "token.actions.githubusercontent.com:sub": "repo:<GitHub org or username>/*"
            }
          }
        }
      ]
    }

The StringLike match is important - without it anyone who knows the ARN of the role could configure GitHub to authenticate as that role, but as they cannot change the sub that GitHub sends to AWS to authenticate you have complete control here of which repositories and branches / tags can assume this role. See Configuring the OIDC trust with the cloud.

In GitHub Action

1) Allow id_token write permission

By default the GitHub Action cannot write an id_token, so you need (at a minimum) this at the top of your workflow file:

   permissions:
     id-token: write
     contents: read

You may need further permissions for other actions! See Adding permissions settings for more details / examples.

2) Add the following step to your job:

   - name: configure aws credentials
     uses: aws-actions/configure-aws-credentials@v1
     with:
       role-to-assume: <ARN of the Role you created above>
       aws-region: $

And that’s it, really.

I’ve shown it as a gradle test run but it should be more widely applicable.

Nice thing about is is that by mounting the local directory the whole way through when the files are generated they are generated where you would expect to see them locally.

Basics:

1. GitHub Action steps

   jobs:
     build:
       steps:
         - name: start minikube
           uses: medyagh/setup-minikube@latest
   
         - name: K8S Tests
           run: run-k8s-tests.sh
   

2. run-k8s-tests.sh

   #!/usr/bin/env bash
   
   set -exuo pipefail
      
   main() {
     trap 'cleanup' EXIT
   
     kubectl config use-context minikube
      
     ensure_a_clean_namespace
   
     prepare_minikube
   
     kubectl apply \
       -f other-resources.yml \
       -f k8s-tests.yml
   
     wait_for_tests_to_finish
   }
   
   ensure_a_clean_namespace() {
     kubectl delete all --all -n k8s-test
   }
      
   prepare_minikube() {
     minikube mount .:/home/gradle/project &
   }
   
   wait_for_tests_to_finish() {
     # Wait for the job to start - otherwise there will be no pods
     kubectl wait --for=jsonpath='{.status.active}'=1 job/k8s-tests --timeout=30s -n k8s-test
      
     # Wait for the pods to be ready - be aware if the jobs is really fast this
     # will fail because ready will go from true to false too fast for the check
     # I cannot find a truly reliable of checking whether the pods *have* started in the past
     kubectl wait --for=condition=ready pods --selector=job-name=k8s-tests --timeout=30s -n k8s-test
   
     # Follow the logs to get real time info on what is happening
     kubectl logs job/k8s-tests --follow -n k8s-test
   
     # The pods should be back with ready=false when they exit
     kubectl wait --for=condition=ready=false pods --selector=job-name=k8s-tests --timeout=30s -n k8s-test
     # Meaning this should work
     local exit_code; exit_code=$(kubectl get pods --selector=job-name=k8s-tests --output=jsonpath='{.items[0].status.containerStatuses[0].state.terminated.exitCode}' -n k8s-test)
     return "$exit_code"
   }
      
   cleanup() {
     local running_jobs; running_jobs=$(jobs -p)
     # shellcheck disable=SC2086
     kill $running_jobs || true
     # shellcheck disable=SC2086
     wait
   }
   
   main "$@"
   

3. k8s-tests.yml

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: k8s-tests
   spec:
     parallelism: 1
     completions: 1
     backoffLimit: 0
     # Set this to something appropriate!
     activeDeadlineSeconds: 600
     template:
       spec:
         containers:
           - name: k8s-tests
             image: gradle:7.4.0-jdk17
             command: ["gradle", "test"]
             volumeMounts:
               - mountPath: /home/gradle/project
                 name: gradle-project
         restartPolicy: Never
         volumes:
           - name: gradle-project
             hostPath:
               path: /home/gradle/project
   

Nice feature - run-k8s-tests.sh will run quite happily locally if you have minikube & bash installed.

The meat of the complication is the wait_for_tests_to_finish bash function.

If you are testing an image you are building, you can build and run it the tests fairly efficiently in minikube as so:

#!/bin/bash
set -euo pipefail

eval "$(minikube docker-env)"

docker buildx build . -t myimage:local

./run-k8s-tests.sh

(I’m using buildx build here, but if you don’t need buildx a docker build will do.)

I’ve experienced both - I was on a project with upwards of 15 devs, doing routine TDD on trunk with a zero bug policy - they were raised as physical pink index cards on a physical board, and always went to the top, so there were the next bit of work picked up.

My insight is that actually we were bug tracking with a bug database. The bug database was the physical cards. And we had bugs, so it wasn’t a zero bugs policy. What it was, was a very short-lived bug policy - we tolerated & tracked them for a very short period of time.

And the same was true of our trunk based development - we did branch. As soon as a pair diverged from trunk on their work station, we had a branch, just locally on that work station - and it needed to be merged by pushing to trunk (and sometimes, of course, that caused merge conflicts that had to be resolved). But crucially, those branches were very short-lived. Because they only existed locally we expected to merge them into trunk multiple times a day.

My theory is that the two practises are actually not in themselves the silver bullet, and indeed may represent local maxima. The real benefits are branches and tracked bugs that are short-lived. The pros of the TBD & zero bugs practises is that it makes it very difficult not to have those benefits. If you track your bugs as physical index cards on a physical board they can’t live for long - you’ll have to fix them or throw them away. If you only branch onto local machines no-one else can cherry-pick or merge in your changes, and you’re in danger of losing them to some hardware failure, so there’s a huge incentive to get your stuff on trunk ASAP.

However, if you can find a way to have those benefits whilst having remote branches & keeping bugs in a tracking system you may be able to get other benefits. I like a protected trunk/main branch - when doing TBD the build got broken from time to time, and it held everyone else up (the classic 5:30 cowboy check-in & run…). Set the CI up to auto-merge on successful build and you mitigate that a lot without adding much overhead - provided you keep the branch short-lived.

Everyone’s talking about Log4Shell. A lot of the talk is, of course, about the nature of code injection attacks, and the failure to separate code and data. It’s an area where I feel sympathy for the Log4J 2 devs - I think the mistake they made is a terrifyingly easy one to make, as every SQL injection or naked eval regularly demonstrates. Layers of abstraction can lead to forgetting when an input to a function is now untrusted data.

However, I think there’s another aspect of it that deserves consideration - the fact that the ability to do this was installed on so many systems.

How many systems need to do LDAP JNDI lookups at all? I’m guessing a pretty small percentage. How many need to do so from their logging system? A smaller percentage. And of those, how many need to load complex classes (with static initialisation) from a remote location via their logging system? I’m guessing an absolutely tiny fraction of the systems vulnerable to Log4shell were actually benefiting from the feature(s) that caused the vulnerability.

And yet there the code was, sitting on all those systems, waiting for the moment someone found the vulnerability.

I’ve been experimenting with JPMS and jlink, and if you create a JRE without the java.naming module then, unsurprisingly, you aren’t vulnerable to Log4Shell even if you’re running an old Log4J 2 version. But inertia and a reasonably high barrier to entry means that adoption of JPMS in general, and adoption of jlink cut down JREs in particular, has been pretty minimal, unfortunately. And some of the java namespaced modules are still disconcertingly enormous - for instance using Java Beans (as lots of things, including Log4J 2, require) means bringing in java.desktop, which brings in the swing, awt & applet packages amongst others. Perhaps rather more than you expected to call set on some data class…

However, this could have been managed at the Log4J 2 level. JndiManager is in the org.apache.logging.log4j.core.net package in the log4j-core jar, along with rather a lot of things capable of doing I/O over multiple protocols. So you can’t use Log4J 2 without having this logic installed. Had JNDI lookups been in its own little jar, brought in as a separate dependency, the bug would still have been in that code - but I’d hazard a guess that it would have been drastically mitigated because most systems would not have had it sitting there, a little otherwise pointless time bomb waiting to explode.

There’s a tension here with ease of use, and particularly keeping a low barrier to entry. It is nice when it turns out you can just use a feature without scrabbling around trying to work out which dependency you need to add to get it to work. But having so many features lying around unused can have a high price.

Keep it small. Keep it focussed. Leave it out by default. If you need a feature 99% of users do not need, it is entirely reasonable that you should need to add a new dependency for it to start working.

I want:

• a nightly build without cache, to prove that the build works from scratch and isn’t being propped up by a cache that cannot be recreated
• a regularly renewed cache, so that it doesn’t keep growing - often the cache retrieval, expansion & update is the longest part of my build.

It occurred to me that these two requirements play nicely together - if the nightly build could start with no cache, but prime the cache, then all builds during the day would benefit from a nice minimal cache.

I think the following GitHub action achieves this, by including the current date in the base cache key:

name: My Build

on:
  schedule:
    # Daily at 2AM
    # * is a special character in YAML so you have to quote this string
    - cron: '0 2 * * *'

env:
  cache-name: my-build-1

jobs:
  build:
    runs-on: ubuntu-18.04

    steps:
      - uses: actions/checkout@v2

      - name: Get current date
        id: date
        run: echo "::set-output name=date::$(date +'%Y-%m-%d')"

      - name: Cache whatever
        uses: actions/cache@v2
        with:
          path: ~/path_needing_caching
          # Always want a cache miss on the first build of the day, which should
          # be the scheduled 2AM one. Proves the build works from scratch, and
          # primes a nice clean cache to work with each day.
          key: ${{ env.cache-name }}_${{ steps.date.outputs.date }}-${{ github.ref }}-${{ github.run_number }}
          restore-keys: |
               ${{ env.cache-name }}_${{ steps.date.outputs.date }}-${{ github.ref }}-
               ${{ env.cache-name }}_${{ steps.date.outputs.date }}-

They are not natively supported yet (see KT-13108: Denotable union and intersection types). However, you can get something very like them using Kotlin’s by delegation:

// Third party code - I can't change
interface WebDriver
interface HasInputDevices
class RemoteWebDriver : WebDriver, HasInputDevices
class EventFiringWebDriver : WebDriver, HasInputDevices

// My code
interface CompositeWebDriver : WebDriver, HasInputDevices

class CompositeRemoteWebDriver(
  delegate: RemoteWebDriver = RemoteWebDriver()
) : CompositeWebDriver,
  WebDriver by delegate, 
  HasInputDevices by delegate

class CompositeEventFiringWebDriver(
  delegate: EventFiringWebDriver = EventFiringWebDriver()
) : CompositeWebDriver,
  WebDriver by delegate, 
  HasInputDevices by delegate

class WebDriverClient(driver: CompositeWebDriver) {
  // code depending on methods in WebDriver & HasInputDevices
}

// Client can now depend on either implementation
val client1 = WebDriverClient(CompositeRemoteWebDriver())
val client2 = WebDriverClient(CompositeEventFiringWebDriver())

It’s still a bit painful; obviously there’s a reasonable amount of repetition in the declaration of each implementation, and adding a new interface to CompositeWebDriver’s supertypes means also adding it as an extra supertype to every implementation with a by delegate declaration. Still, it decouples the client code from the actual third party implementation without having to implement every method in each interface.