Docker push fails with 504 Gateway Time-out

I was solving issues during large docker layers being uploaded to the SaaS Artifactory docker repository. Docker failed on large layers with the following error message:

received unexpected HTTP status: 504 Gateway Time-out

High-level description of how traffic traveled:

Client > Nginx proxy < AWS LB > SaaS Artifactory (docker repository)

In most cases, you will have misconfigured Nginx proxy or AWS LB. In this case, we raised in Nginx config:

Which didn’t help. Nothing suspicious was logged in the Nginx error/access log. Only Artifactory reported the following error:

We also needed to adjust the AWS LB configuration because large layers can take some minutes to be saved in the desired repository. By default, Idle Timeout is 60 seconds. After raising this value we fixed this issue.

AWS LB Attributes – Set proper Idle timeout

AWS Terraform provider supports this configuration – https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lb#idle_timeout

If you’re using AWS ALB Ingress in k8s:

alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.