Single Node OKD (OpenShift)
Everytime I get to use Red Hat’s Open Shift I think to myself: “I should use OpenShift more”. It is a really great tool not just for high availability clusters but for general software development also. Its Web interface is super intuitive and nice to use, and you can provision all sorts of stuff in minutes. And everything is open source. Its really great.
So naturally I decided to run it locally, especially after the news that starting from version 4.8 you can use it on single node architecture also. I decided to run it virtualized in libvirt so I can destroy it if I no longer needed. Hooray!
Sadly my “Hooray” was short lived, since the installation proved difficult for me.
The official guide calls for either using the Assisted installer which is only available for OpenShift, not for OKD or using the manual way which is not that difficult, but I just couldn’t get it to work. The ignition process would start and hang with one of the issues I will mention below.
I searched and found several guides on how others have set up their small virtual “clusters”:
https://itnext.io/okd-4-5-single-node-cluster-on-windows-10-using-hyper-v-3ffb7b369245
https://cgruver.github.io/okd4-single-node-cluster/
https://blog.maumene.org/2020/11/18/OKD-or-OpenShit-in-one-box.html
All of them had very useful tips and general information to learn about OpenShift/OKD but I couldn’t get any of them to work properly. I ran into a number of issues with: etcd not working (etcdctl failed: retrying in 5 seconds), bootstrap machine not having enough storage (https://github.com/openshift/okd/issues/649), iptables SELinux trouble (https://bugzilla.redhat.com/show_bug.cgi?id=2008097) and others. Also some of the guides called for outdated OKD version use which meant that the container images were no longer available. The closest guide I got to be almost working is the last one (from the maumene blog). There, everything worked apart from the console operator (which may have been an error on my part).
Now I have to say that some of the issues I had were my configuration faults, either with DNS, haproxy or DHCP configuration. But a lot of them came from various installation problems which were difficult to find. My final successful installation for example only works with one certain OKD version (without specific reasoning why).
Code Ready Containers #
If you made it this far, you are probably wondering why not just use CRC and it is a good question. While I used CRC before and it works great I was really interested in setting up my own version. Funnily enough, it later proved that CRC was indeed the answer to my problems because it has automated ways of setting up Single Node Clusters (SNC’s)!
Setting up your own libvirt OKD (OpenShift) Single Node Cluster (a short guide) #
There is a fantastic post here: https://upstreamwithoutapaddle.com/home-lab/okd-crc/ on how to build your own OKD version of CRC. In its process it actually builds a libvirt Single Node Cluster (SNC, https://github.com/code-ready/snc) to create an image for CRC. So you can just use this process to build your own single node cluster and skip building CRC.
Follow the guide until this step:
./snc.sh
or before creating of the single node cluster. In the snc.sh file you can inspect and change the whole process and configuration on how your cluster will be built. You can change OKD version (only one that worked for me was 4.8.0-0.okd-2021-10-10-030117, hardware configuration, you can disable the “not for production warning” by commenting it out in the code, and you can even change the default crc.testing hostname (I looked for it with grep -Ri crc.testing and replaced it everywhere with sed.
When the process is done, you will have a single libvirt VM with the entire functioning OKD cluster in it. DNS and DHCP are configured automatically with libvirt. Also storage and registry are automatically setup for you.
The VM is not setup to start OKD by default but you can ssh into it (ssh key id_ecdsa_crc is generated during the process and saved in your working directory) and execute: sudo systemctl enable kubelet && sudo systemctl start kubelet
Caveats and Warning (December 2021) #
Here I list some of the issues with this process in hope of helping others save time.
Version of OKD is important #
The only version that worked for me is this one: 4.8.0-0.okd-2021-10-10-030117, there are later versions of OKD 4.8 but they didn’t seem to work, be careful about it.
DNS #
On my Fedora 35 laptop, systemd-resolv was not playing nicely with dnsmasq setup of libvirt. There is a comment here on how to make it work, but I ended up disabling it and using dnsmasq (default one from NetworkManager).
SNC needs you #
The SNC building process is not fully automatic, it needs your sudo password several times (to work with virsh) and it will time out making you run the lengthy process from the beginning. Don’t ask me how I know. :)
Future will be better #
Take a look also at this method: https://github.com/eranco74/bootstrap-in-place-poc which also mentions ways of doing thing automatically. Since the SNC/CRC method worked for me, I didn’t test this one our but it looks very promising.
Conclusion #
Big thanks to @cgruver for his guide on how to build CRC with OKD. In the end it was a learning curve for me, but I am now comfortable with setting up and using OKD in my development environment and it is truly amazing software.