Running Flux in OpenShift

[milroy]

I've gotten a simple test setup working that runs Flux in pods managed by OpenShift. I used a deployment for replicating the pods to simulate Flux instances running across multiple nodes. The config is essentially the same as a Kubernetes spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app: flux
  replicas: 2
  template:
    metadata:
      labels:
        app: flux
    spec:
      containers:
      - name: flux
        image: fluxrm/flux-sched:latest
        command: ["sleep"]
        args: ["infinity"]
        ports:
        - containerPort: 8050

Note the command and args parameters. These allow the containers to keep running, otherwise they will exit immediately upon container startup. Now the bootstrap config is as you would expect if the containers were physical hosts (i.e. use the pod IPs, NICs, and names):

[bootstrap]
hosts = [ { host="flux-deployment-1-q579j", bind="tcp://eth0:8050", connect="tcp://10.130.2.69:8050" }, { host="flux-deployment-1-tnnzb", bind="tcp://eth0:8050", connect="tcp://10.130.2.70:8050" }, ]

To start the Flux instances issue the following command in each container:

$ flux start -o,--config=/home/fluxuser/config

Now to test that they can communicate, run the following on rank0:

ƒ(s=2,d=0) fluxuser@flux-deployment-1-q579j:~$ flux exec -r all hostname
flux-deployment-1-q579j
flux-deployment-1-tnnzb
ƒ(s=2,d=0) fluxuser@flux-deployment-1-q579j:~$

[dongahn]

Note the command and args parameters. These allow the containers to keep running, otherwise they will exit immediately upon container startup. Now the bootstrap config is as you would expect if the containers were physical hosts (i.e. use the pod IPs, NICs, and names):

I assume you need to manually create the bootstrap toml. How do you collect the info to create a valid toml?

[milroy]

I've been creating it manually, but there may be ways to populate it automatically by querying the APIserver. I'll start looking into it.

I'm not sure how to handle autoscaling; perhaps just append new pod info to the TOML file as they're created?

[dongahn]

Yeah I think one of the keys would be the autopopulation...

[dongahn]

I'm not sure how to handle autoscaling; perhaps just append new pod info to the TOML file as they're created?

So we need to take a look at this a bit I think. If autoscaling is so that we kill the existing pods, this should be fine. If autoscaling just add pods, we will need more elasticity support. This will take some time for us to get there. Perhaps the first autoscaling support should just be limited to "kill" and "run larger". Do we know how to specify such semantics in the deployment spec?

[milroy]

If autoscaling just add pods, we will need more elasticity support. [...] Do we know how to specify such semantics in the deployment spec?

It adds or deletes pods within a range (min, max) to maintain specified performance metrics (e.g. --cpu-percent=50). The configuration can be set via kubectl or within the deployment YAML.

There's also a replica field (appears in the deployment YAML above) which maintains a specified number of replicas. This parameter can be changed manually (via kubectl) to add or delete pods.

[dongahn]

Is there an easy way to create this toml upon and store that into the file system of the container. This way, users can instantiate flux start without worrying about the manual creation of the toml.

[milroy]

Perhaps a template TOML can be filled in as the pods are initializing, or this may need to be done once the pods are running (see above).

[dongahn]

Yeah... we should look into that. It seems asking users to fill this in is a bit too much.

[milroy]

I definitely agree, especially since we will need to enable automated creation and deletion of pods.

[dongahn]

My other question is: will this recipe work on my Mac Laptop where I run Docker Desktop that comes with Kubernetes?

[milroy]

Yes! I said "essentially the same as a Kubernetes" YAML, but I think it's identical in this case. Let me know how it works.