The heart of Continuous Integration is running tests. Whenever a test fails, you want to know why ASAP so you can correct it. Whenever a CI build fails, you want to see that failing test and how it failed.
CircleCI’s Test Summary feature puts this info front-and-center so you can respond directly to the test failure without anything getting in your way. The trick is to feed CircleCI your test info the way it expects.
The build log might be fine to start. You expand the failing step, scroll to the end of the page, and then scroll up till you hit the test failure.
This is not too bad. At first.
But with a big enough project, the build and test logs grow too long to view in-place on the web page. Then you find yourself downloading the log file first.
Sometimes the failing test isn’t really that near the end of the file. Then you’re fumbling around trying to find it.
Across a lot of developers on a long project, this time and friction adds up.
If you’re building an iOS app, and you copy-paste the Example Configuration for Using Fastlane on CircleCI, you should luck into something that works.
But you’ll want to better understand what the Test Summary feature is looking for if:
CircleCI’s Collecting Test Metadata doc calls out one big thing:
store_test_results step reference
calls out another:
This subdirectory name is used to identify the test suite.
There’s one more requirement that I haven’t seen documented anywhere, though:
The rest of the filename doesn’t seem matter for the test summary, but if you have the wrong path extension, you won’t see any test summary.
You’ll wind up with a directory layout like:
/Users/distiller/project/ └── fastlane └── test_output └── xctest └── junit.xml 3 directories, 1 file
This ticks all the boxes:
(Fastlane only produces a single test report, so the nesting of report-in-folder-in-folder admittedly looks a little silly.)
Scan provides a lot of config knobs.
You can view a table of the full list and their default values by running
fastlane action scan.
We need to arrange three things:
Conveniently enough, Scan has three config settings, one for each of those needs.
Scan also happens to have three different ways to set those three options:
In your Fastfile, you can set them using keyword arguments to the
scan method call:
scan( # … other arguments … output_types: 'junit', output_files: 'junit.xml', output_directory: './fastlane/test_output/xctest')
If you’re invoking
you can set them using CLI options:
fastlane scan \ --output_directory="./fastlane/test_output/xctest" \ --output_types="junit" \ --output_files="junit.xml"
Because Ruby is a true descendant of Perl, TMTOWTDI, so you could also configure Scan using environment variables:
env \ SCAN_OUTPUT_DIRECTORY=./fastlane/test_output/xctest \ SCAN_OUTPUT_TYPES=junit \ SCAN_OUTPUT_FILES=junit.xml \ fastlane scan
(You could also set those environment variables in the
environment stanza in
your CircleCI config. Six one way, half-dozen the other.)
Now you have Fastlane Scan writing its test report using the JUnit format into
*.xml file under a suggestively-named subdirectory.
To get CircleCI to actually process this carefully arranged data,
you’ll need tell the
store_test_results step to snarf everything at and under
That’s right: not just the
xctest subdirectory that holds the test report
XML, but the directory.
Add this step to the pipeline that runs
- store_test_results: path: "./fastlane/test_output"
At some point, you’ll probably want to be able to look at the test report yourself, as well as the overall build logs.
You can send both of those on up to CircleCI as build artifacts using a couple
- store_artifacts: path: "./fastlane/test_output" destination: scan-test-output - store_artifacts: path: ~/Library/Logs/scan destination: scan-logs
You’re not limited to just one artifact or just one test output. In fact, handling multiple kinds of test output is precisely why there’s the folder-in-folder nesting.
Say you wanted to have CircleCI call out SwiftLint nits.
You could drop this snippet into your
lint: docker: - image: dantoml/swiftlint:latest steps: - checkout - run: name: Run SwiftLint command: | mkdir -p ./test_output/swiftlint swiftlint lint --strict --reporter junit | tee ./test_output/swiftlint/junit.xml - store_test_results: path: "./test_output" - store_artifacts: path: "./test_output"
The key links in the chain here are:
store_test_resultsat that “ALL the tests” directory:
Any output you can massage into meeting those requirements, you can cadge CircleCI into calling out in your Test Summary.
There you have it:
Interested in learning more about our basic and advanced iOS Courses?
Learn from the experts at a Big Nerd Ranch Bootcamp!
Interested in leveling up your coding skills from the same authors of the Big Nerd Ranch Guide? Subscribe to The Frontier today!