Docker push fails with 504 Gateway Time-out

I was solving issues during large docker layers being uploaded to the SaaS Artifactory docker repository. Docker failed on large layers with the following error message:

received unexpected HTTP status: 504 Gateway Time-out

High-level description of how traffic traveled:

Client > Nginx proxy < AWS LB > SaaS Artifactory (docker repository)

In most cases, you will have misconfigured Nginx proxy or AWS LB. In this case, we raised in Nginx config:

      proxy_read_timeout XXs;
      client_header_timeout XXs;
      client_body_timeout XXs;
      proxy_connect_timeout XXs;
      proxy_send_timeout XXs;

Which didn’t help. Nothing suspicious was logged in the Nginx error/access log. Only Artifactory reported the following error:

[ERROR] [c.e.m.GlobalExceptionMapper:48] [p-nio-8081-exec-6666] - Failed to transform BinaryStream to PreProcessChecksumBinaryStream
org.jfrog.storage.StorageException: Failed to transform BinaryStream to PreProcessChecksumBinaryStream

We also needed to adjust the AWS LB configuration because large layers can take some minutes to be saved in the desired repository. By default, Idle Timeout is 60 seconds. After raising this value we fixed this issue.

AWS LB Attributes – Set proper Idle timeout

AWS Terraform provider supports this configuration – https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lb#idle_timeout

If you’re using AWS ALB Ingress in k8s:

alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.