This was a lot of fun to figure out, and served as a test case for the dhcpd container.  At the end of it I have a 7 node raspberry pi cluster that is accessed and controlled from one node over wifi, and the other nodes get their IPs from the dhcpd container.

Step 1: get the pi control node to run a static IP on wifi

My home wifi router is provided by the cable company, and it doesn't have the ability to reserve IP addresses for specific MAC addresses, so my pi node IPs keep changing. Also, I found you can't run a dhcpd service in a container while on a host that uses dhcpd as a client, since port 67 & 68 would already be in use (kind of a duh moment, but anyways).

Researching the concept of static IP over wifi on a Pi was surprisingly challenging, but I found a decent link that explained it best:
https://raspberrypi.stackexchange.com/questions/37920/how-do-i-set-up-networking-wifi-static-ip-address/74428#74428

In summary, I needed to:

• edit my /etc/network/interfaces file to set up the static IPs I would be using for each network segment, the wifi and my eth0 segment for the cluster nodes.

source-directory /etc/network/interfaces.d
auto lo
iface lo inet loopback
iface eth0 inet manual
allow-hotplug wlan0
iface wlan0 inet static
        address 192.168.0.38
        netmask 255.255.255.0
        gateway 192.168.0.1
    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

• disable dhcpd as a client, restart networking and reboot.

sudo systemctl disable dhcpcd
sudo systemctl enable networking
sudo reboot

Step 2: get the dhcpd container working

First, I needed to set a static IP on my eth0 interface to service the node cluster network. I have chosen to use 192.168.100.0/24 for my internal cluster network, and will set 192.168.100.1 as the static IP for eth0. The simplest method to do this is to add the ip to the end of the line in the /etc/cmdline.txt file. Here is mine as an example, which has a few extra options to support kubernetes on the pi. Just add the ip=<your choice> to the end.

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=f6795000-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory ip=192.168.100.1

Once I rebooted the pi, the node is available over wifi and has a static IP on the eth0 interface.

Unfortunately, I was not able to get my custom centos container to work effectively on the pi cluster, so I ended up using a container image that another person built: 
https://hub.docker.com/r/prehley/rpi-dhcpd

This image worked out swimmingly. I created a dhcp folder on my pi control node, and inside I write a simple dhcpd.conf with static IP reservations for my pi nodes:

#pi zone
subnet 192.168.100.0 netmask 255.255.255.0 {
    allow bootp;
    authoritative;
    option routers 192.168.100.1;
    option domain-name "pizpone";
    option domain-name-servers 1.1.1.1, 8.8.8.8;
    default-lease-time 3600;
    max-lease-time 21600;
    range dynamic-bootp 192.168.100.100 192.168.100.110;
    host p1 {
        hardware ethernet b8:27:eb:c8:cd:01;
        fixed-address 192.168.100.91;
        option domain-name-servers 1.1.1.1;
        }
    host p2 {
        hardware ethernet b8:27:eb:c8:cd:02;
        fixed-address 192.168.100.92;
        option domain-name-servers 1.1.1.1;
        }
... <truncated>
}

Of course, the above example is truncated to show the pattern. I've got six host nodes that will be served by this scope.

I'll run this container as a local docker container rather than a docker service.

docker run -d --name dhcpd --net=host --restart always -v "$(pwd)/dhcp":/data prehley/rpi-dhcpd

This creates a container that mounts a volume for the config data and restarts if the node is restarted. Once the subordinate nodes are restarted (I have them all powered by a wifi enabled surge protector, so it's 'Alexa, turn off cluster' and 'Alexa, turn on cluster', and it comes up automagically! I can ssh into each node vi the control node, and I disable the subordinate nodes' wifi config by renaming the /etc/wpa_supplicant/wpa_supplicant.conf which breaks the wifi settings. There is likely a better way to do that.

Finally, I added /etc/hosts entries for the nodes in the control node. The nodes where set up for docker before, by installing the docker package and adding the pi user to the docker groups.

Step 3: Set the pi master node to masquerade traffic to the cluster network

I found the relevant steps in this article, but I skipped the dnsmasq part:
https://pimylifeup.com/raspberry-pi-wifi-bridge/

Essentially, I skipped down to step 11 to enabled ip forwarding by editing the /etc/sysctl.conf file and uncommenting net.ipv4.ip_forward=1

Next, I ran the following commands:

sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE  
sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT  
sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT  
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Finally, I edited the /etc/rc.local file and added this line above the exit 0 line, to re-enable ip forwarding after reboot:

iptables-restore < /etc/iptables.ipv4.nat

Step 4: get the swarm started

Very easy to do. At this point in my exploration, I wonder why kubernetes is the dominant container cluster architecture rather than docker swarm. I imagine it has to do with Google's influence as the originator of the kubernetes project. At some point, I'll have to do a comparison, but getting this pi swarm running was far easier than the pi-kubernetes cluster.

First, I had undo my original swarm that was working over the wifi, so I ran 'docker swarm leave' on each node, and 'docker swarm leave --force' on the master node.

On the master node:

docker swarm init --advertise-addr 192.168.100.1

I copied the output, which is a join command for the slave nodes. I ran this on each node via ssh:

ssh -i myKey.pem pi@p1 'docker swarm join --token TOKEN-example-1234 192.168.100.1:2377'

Once all nodes had been joined to the swarm, it was easy to start a test service:

docker service create --name webtest -p8080:80 --replicas 6 httpd

By setting up the node network to route through the control node's static IP and then forwarding it out the wifi, each of the slave nodes can pull the images needed to run the service.

This experiment helped me to understand how to run a dhcpd container, which I will next test in my school's network, as well as how to create a docker swarm on the raspberry pi. Having built a successful swarm, I can attack the issue of running docker machines to create the swarm at work using the time machine servers as well as repurposing the current linux services as docker swarm nodes.

The next step would be to build a service that is more interesting than a basic web service , and access it through a load balancing nginx node. But, that is a problem for another time.