Background
I’ve been working with HydraDX recently and attempting to run it in Docker. Unfortunately, the last runnable version of HydraDX is significantly outdated, and it no longer connects to active peers. Furthermore, when attempting to use a more recent HydraDX Docker image, an error arises indicating “version `GLIBC_2.34′ not found.” As of the most recent version available on September 4th, 2023, identified as pr-664, this issue still persists without resolution. Following a period of challenges and without delving into the underlying causes of the issue, I decided to immerse myself in the Rust program and scrutinize the code. Through this process, I identified some outdated configurations and instances of Docker misuse.
It seems that encountering similar issue when compiling the rust program and running it in Docker environment is quite common, especially within the blockchain industry.
However, please note that the focus of this post is only on building a Docker image for a Rust program (or at least a program linked with glibc). Understanding HydraDX or blockchain concepts is not necessary to grasp the content of this discussion. But with a minimal knowledge of Docker is required.
Reproduce Problem
To address the issue, the initial step is to reliably reproduce it. Fortunately, with the HydraDX Docker image, achieving this is straightforward. I anticipate that the following command should directly run HydraDX. Please note that my environment is a Macbook Pro M1, thus the command line parameter –platform linux/x86_64 is required.
$ docker run -it --rm --platform linux/x86_64 galacticcouncil/hydra-dx:pr-664
Alas, this command will yield the following lines of error messages.
/usr/local/bin/hydradx: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/local/bin/hydradx)
/usr/local/bin/hydradx: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /usr/local/bin/hydradx)
Identify Issue
If you’ve worked with C/C++ on Linux, you probably have a broad comprehension of the prevalent problem — typically tied to absent libraries. Furthermore, you might speculate about a potential resolution, which often involves copying the appropriate library versions to either the standard system libraries path or the location specified by $LD_LIBRARY_PATH environment variable.
That’s precisely what crossed my mind when I encountered it for the first time.
Hence, I proceeded to execute the following command in an attempt to promptly identify the issue.
First of all, accessing the shell of the Docker image with root privileges.
$ docker run --user root -it --rm --platform linux/x86_64 --entrypoint /bin/bash galacticcouncil/hydra-dx:pr-664
Then it is easy to check the “not-found” libraries status. The result surprised me.
root@28b1cd5da8b8:/# ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Mar 31 2021 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.33.so
root@28b1cd5da8b8:/# ls -l /lib/x86_64-linux-gnu/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Apr 27 2021 /lib/x86_64-linux-gnu/libstdc++.so.6 -> libstdc++.so.6.0.29
The Docker image indeed contains libc.so.6 and libstdc++.so.6; however, they are of an older version.
With a swift examination of the container’s OS, it can be a evident that HydraDX is erroneously utilising an end-of-support version of Ubuntu, specifically, hirsute 21.04.
root@28b1cd5da8b8:/# cat /etc/debian_version
bullseye/sid
root@28b1cd5da8b8:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.04
DISTRIB_CODENAME=hirsute
DISTRIB_DESCRIPTION="Ubuntu 21.04"
But why the main program is asking a newer version of libraries? The Dockerfile can answer the question.
FROM ubuntu:21.04
LABEL org.opencontainers.image.source = "https://github.com/galacticcouncil/HydraDX-node"
RUN useradd -m -u 1000 -U -s /bin/sh -d /hydra hydra && \
mkdir -p /hydra/.local/share && \
chown -R hydra:hydra /hydra
ADD target/release/hydradx /usr/local/bin/hydradx
RUN chmod +x /usr/local/bin/hydradx
USER hydra
EXPOSE 30333 9933 9944 9615
VOLUME ["/hydra/.local/share"]
ENTRYPOINT ["/usr/local/bin/hydradx"]
CMD ["--prometheus-external", "--", "--execution=wasm" ,"--telemetry-url", "wss://telemetry.hydradx.io:9000/submit/ 0"]
Evidently, the Docker image lacks a build phase and directly copies the binary program built on the host into the container. Consequently, when the Docker host runs a more recent version of glibc than the older version present in the container’s execution environment, it triggers the “lib not found” error. Interestingly, this error mirrors the same issue one might have encountered two decades ago with C++ programs.
Solution
Upon pinpointing the precise cause of the “lib not found” error, the solution becomes remarkably straightforward. It involves upgrading the container to Ubuntu 22.04, which is equipped with glibc 2.35. This version can offer backward compatibility with programs relying on glibc 2.34, and it can even support versions as early as glibc 2.27.
For a comprehensive solution, merely upgrading the container environment won’t suffice. This is because when the glibc version in the building environment surpasses that of the running environment, the same error will resurface.
Introducing a build phase and ensuring that the library versions in both the build and runtime phases are coordinated should indeed represent the most effective and sustainable solution for the long term.
Upon conducting a further investigation, I found that up until April of last year, HydraDX indeed had a build phase within its Dockerfile. The commit comment for this change reads “fixed docker build.” Prior to this fix, the Docker image used rust:lastest as the builder, which, as of that time, corresponded to rust:1.67 based on Debian bullseye. Simultaneously, the runtime environment utilized an older version of Debian, specifically buster. It’s highly plausible that HydraDX encountered build issues due to this version discrepancy, leading to the implementation of a patch in the Dockerfile, albeit with an incorrect approach.
Conclusion
When confronted with a similar “lib not found” issue in a Docker image, it is strongly advisable to meticulously examine the version compatibility between the build and runtime libraries. It’s imperative to ensure that the runtime glibc is capable of backward compatibility with the build glibc version, meaning that the version of the runtime glibc must be equal to or higher than the build environment’s version. This alignment is crucial for resolving such issues effectively.
You can use the realelf command to determine the lowest supported version of the current glibc. For instance, you can use a command like the following:
$ readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6 | grep glob64
Here is a good article to read: How the GNU C Library handles backward compatibility