Fetching CircleCI Artifacts

Do you use CircleCI for your continuous workflows? Do you use their scheduled jobs? Recently we had a need to retrieve some performance benchmarking via a Lighthouse service that records page load times out to a text file. This job is scheduled to run daily.

Unfortunately it can be difficult to find a build in the CircleCI UI for a given project since there is no search, and only 20 builds at a time are shown in order of most recently run. Fortunately CircleCI has an API that lets us automate the task of trying to find a build and view the artifacts from the run: https://circleci.com/docs/api/#recent-builds-across-all-projects

We can fetch up to 100 builds a given project a time. Some scripting allows us to narrow our results down to just the build types we are interested in. From here we now have the build number from the most recent build of a given type. In our case the type is the pageload times from Lighthouse.

Once we have found a specific build for a given project we can use the API again to ask about its artifacts: https://circleci.com/docs/api/#artifacts-of-a-build . This allows us to get the container information and paths of any artifacts produced by the job. Including our page load times.

We now have the URL for a given artifact, and it is just a matter of downloading the file by suffixing the CircleCI token to our URL: https://circleci.com/docs/api/#download-an-artifact-file

We now have the output from our artifact. From here we can put this information into our company Slack, or even push it to a collaborate spreadsheet that the team routinely reviews. The specifics of how to automate this script, and what to do with its output is outside the scope of this post, but I will share our Ruby script for interacting with CircleCI. Should be easily adaptable to other languages. It can be viewed below:

	# Finds a CircleCI job of a given name, and retrieves an artifact from a given CircleCI project build
	# Usage:
	# $ API_TOKEN=xxx GITHUB_USERNAME=bsimpson GITHUB_PROJECT=some-project TARGET_JOB=lighthouse ARTIFACT=averages_pageload ruby ./circleci.rb

	require "net/http"
	require "json"

	API_TOKEN = ENV["API_TOKEN"]
	LIMIT = 100
	GITHUB_USERNAME = ENV["GITHUB_USERNAME"]
	GITHUB_PROJECT = ENV["GITHUB_PROJECT"]
	TARGET_JOB = ENV["TARGET_JOB"]
	ARTIFACT = ENV["ARTIFACT"]

	# Recent builds for a single project
	# curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project?circle-token=:token&limit=20&offset=5&filter=completed
	def find_job(job=TARGET_JOB)
	offset = 0

	while offset < LIMIT * 10 do
	url = "https://circleci.com/api/v1.1/project/github/%{github_username}/%{github_project}?circle-token=%{token}&limit=%{limit}&offset=%{offset}"
	uri = URI.parse url % {
	github_username: GITHUB_USERNAME,
	github_project: GITHUB_PROJECT,
	token: API_TOKEN,
	limit: LIMIT,
	offset: offset
	}

	response = Net::HTTP.get(uri)
	jobs = JSON.parse(response)
	matching_job = jobs.detect { \|job\| job["build_parameters"]["CIRCLE_JOB"].match(TARGET_JOB) }
	if matching_job
	return matching_job
	end

	puts "Trying offset #{offset}…"
	offset += LIMIT
	end

	puts "Exhausted pages"
	end

	# Return artifacts of a build
	# curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/:build_num/artifacts?circle-token=:token
	def find_artifacts(job, artifact=ARTIFACT)
	build_num = job["build_num"]

	url = "https://circleci.com/api/v1.1/project/github/%{github_username}/%{github_project}/%{build_num}/artifacts?circle-token=%{token}"
	uri = URI.parse url % {
	github_username: GITHUB_USERNAME,
	github_project: GITHUB_PROJECT,
	build_num: build_num,
	token: API_TOKEN
	}

	response = Net::HTTP.get(uri)
	artifacts = JSON.parse(response)

	matching_artifact = artifacts.detect { \|artifact\| artifact["path"].match(ARTIFACT) }

	return matching_artifact
	end

	# Download an artifact
	# https://132-55688803-gh.circle-artifacts.com/0//tmp/circle-artifacts.7wgAaIU/file.txt?circle-token=:token
	def download_artifact(artifact)
	url = "#{artifact["url"]}?circle-token=%{token}"
	uri = URI.parse url % {
	token: API_TOKEN
	}

	response = Net::HTTP.get(uri)

	puts response
	return response
	end

	job = find_job
	artifact = find_artifacts(job)
	download_artifact(artifact)