localhost isn’t always what you expect
I spent the better part of a day working on this issue as it was a deep one. For some reason our test web server started using npx http-server out stopped being accessible in our pipeline. Everything worked locally, so it was one of those that could only be figured out by making changes and sending it to the pipeline. Time consuming to say the least.
The setup
I was moving our Cypress tests to use the static file output of our NextJS export command instead of the NextJS next start command. Testing what is supposed to be a fully static site using a node server didn’t seem right.
In order to serve the files to Cypress we need to run http-server which is a simple http server. Seems simple enough.
We then us wait-on to tell us when the server is up so we can start our Cypress tests.
npm install -g wait-onnpx http-server -p 3000 out & wait-on -i 1000 -v <http://localhost:3000>
However, wait-on never detected the website and therefore hung up the pipeline.
making HTTP(S) head request to url:<http://localhost:3000> ...HTTP(S) error for <http://localhost:3000> Error: connect ECONNREFUSED ::1:3000
It took a while since I’m not accustomed to IPv6 addresses that it was right in front of my nose what address was rejecting the connection (::1:3000 )
I should mention that this only started happening when we moved to node 18.
Node 17 changed the rules
The issue lies in how Node 17 changed how it relates to localhost as it relates to ipv4 vs ipv6. Apparently it now defers to the operating system’s preferences.
This is the reason why everything works fine on my Mac but fails in the pipeline which is running Debian bullseye.
I’d love to find the official docs for this change, but instead I did find several issues of people reporting the same issue.
https://github.com/nodejs/node/issues/40537
It’s apparently a bigger issue relating to the slow adoption of IPv6 and how devices treat the different protocols and fallback.
The official fix is lovingly called Happy Eyeballs.
There is no clear blessed fix yet
I tried seeing if http-server could serve both IPv4 and IPv6, whether wait-on could support both IPv4 and IPV6 and even replacing both those libraries with different alternatives. Nothing seemed to work.
Answer - well the lo-tech patch at least
What ended up working with the least amount of fuss was modifying the etc/hosts file with my script in the docker image. I don’t have access to change the docker image but I am running a script with my Bitbucket pipeline.
The hosts file in the Debian bullseye docker image looked like:
127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters
You can see there are entries for localhost for both the IPv4 address and the IPv6 address (::1). Even though the IPv4 entry is first, something in the operating system configuration still chooses the IPv6 version and finds no web server there, which is why wait-on just hangs.
Just remove the line!
My lo-tech solution for now is to just remove that IPv6 line from the hosts file.
Since I can’t modify the docker image itself, I’ll just do it in the script with my pipeline.
sed "/loopback/d" /etc/hosts > hosts cp hosts /etc/hosts
sed is a little tricky since there are different version of it with quite different options. I found the Debian version of seddidn’t support the -i which modifies the files itself. Instead I need a second line that takes the new file I created and overwrites the etc/hosts file.
Docker docker docker
I really need to learn docker better so I can test this stuff locally. My feedback loop would have been way quicker and probably could have solved it in less than half the time it took me. Yet another thing to learn…