How Cloudflare Broke My Build and How I Fixed It

Mysterious Build Failure

About a month ago, my open-source project EntityFramework.Exceptions failed to build on AppVeyor, the continuous integration service that I have been using for several years. I hadn’t made any changes that could have caused the build to fail, so I set out to investigate the cause of the failure.

It turned out that the build failed because it failed to upload source code coverage data to The error message was very minimal:

BadRequest - <html>
<head><title>400 Bad Request</title></head>
<center><h1>400 Bad Request</h1></center>

I checked coveralls status page, but there wasn’t any ongoing incident, so I reached out to coveralls support. Coveralls support responded that they had recently migrated their services to different servers and were now using Cloudflare as a service provider but these changes should have been transparent to the end users. To ensure this wasn’t a DNS issue, I added ping -n 1 to the appVeyor.yml file. The resolved IP address was correct, so the issue wasn’t related to DNS.

Coveralls support tried to check the cause of the Bad Request error in their logs but surprisingly my attempts to upload the coverage data didn’t show up. This meant that the Bad Request didn’t originate from and was coming from Cloudflare. Why was Cloudflare suddenly rejecting my attempts to upload coverage that was working flawlessly for the last couple of years?

Finding the Root Cause of the Issue

Now that I knew that the failure was Cloudflare’s fault, I asked the team to get in touch with Cloudflare support to find out the cause of the error response. I downloaded (Coveralls uploader for .Net Code coverage) to my local machine, attempted to submit the coverage data, and captured the request with Fiddler. I saved the HTTP request as a har file and sent it to support, hoping Cloudflare would help solve the issue. Unfortunately, Cloudflare was not helpful at all.

My next step, as suggested by James from, was to submit the coverage file with curl and see if that worked. I ran the following command: curl --form "json_file=@uploadedCoverage.json" and to my surprise, the coverage file was successfully received by Now that I was able to submit coverage data with curl, all I had to do, was to compare the HTTP requests issued by curl and to find the difference between them that was causing the failure.

The HTTP request looked like this (request bodies omitted): HTTP request:

Content-Type: multipart/form-data; boundary="e5d8bb1a-4ab2-4bb8-9d8b-152c5b704cfa"
Content-Length: 3093

Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=json_file; filename=coverage.json; filename*=utf-8''coverage.json

curl HTTP request:

User-Agent: curl/7.83.1
Accept: */*
Content-Length: 3080
Content-Type: multipart/form-data; boundary=------------------------c8eebbf0920b1cc0

Content-Disposition: form-data; name="json_file"; filename="uploadedCoverage.json"
Content-Type: application/octet-stream

There are several differences between these requests, but can you guess what was causing Cloudflare to fail? It turned out that Cloudflare returned Bad Request because of double quotes around the boundary directive value in the Content-Type header!

Content-Type boundary with double quotes is perfectly valid, so why does Cloudflare treat it as an error? Unfortunately, I don’t have an answer to this question. The pull request to handle quotes was merged more than two years ago, so either Cloudflare uses an older version of nginx upload module, or Cloudflare uses another module that cannot handle double quotes around the boundary directive.

Fixing Coveralls Uploader

Now that I knew what was causing the failure, it was easy to fix the

var boundary = formData.Headers.ContentType?.Parameters.FirstOrDefault(o => o.Name == "boundary");
if (boundary != null)
    boundary.Value = boundary.Value?.Replace("\"", string.Empty);

I submitted a pull request to the repo, and as soon as it was merged, my build was green again!

Giorgi Dalakishvili
World-Class Software Engineer