C&I Wiki Portal/ToolsDevelopmentGuideline
- 1 Overview
- 2 Basics
- 2.1 Bazaar
- 2.2 Practices
- 3 Project Hierarchy
- 3.1 opencompute-certification
- 3.2 checkbox
- 3.3 Putting it all together
- 4 Obtaining the Code
- 5 Adding new code
- 6 Submitting Code to the Project
- 7 Questions?
The opencompute-testing project contains all the code for generating the OCP Certification and OCP Ready testing tools. It is the code that creates the Plainbox Provider and Launcher Packages, hosting the job descriptions that are OCP Specific, scripts and whitelists. Package builds occur automatically once per day if the code base has been updated.
If you have ANY questions at all about the code base, contributing, or anything else related, contact Jeff Lane For anything related to Open Compute Certification (including code questions):
- Mailing List: http://lists.opencompute.org/mailman/listinfo/opencompute-ci
- IRC: Freenode IRC Servers #ocp-ci
This section will cover some of the basics for contributing to the code for these tools. This is a general overview, more specific information can be found on line at the links below.
Bazaar is a version control system that helps you track project history over time and to collaborate easily with others. Whether you're a single developer, a co-located team or a community of developers scattered across the world, Bazaar scales and adapts to meet your needs. Part of the GNU Project, Bazaar is free software sponsored by Canonical
You can learn more at http://bazaar.canonical.com/en/
To install Bazaar on a debian or ubuntu system:
sudo apt-get install bazaar
For other Operating Systems, including Windows, Red Hat, SuSE, OSX and more, please see http://wiki.bazaar.canonical.com/Download
Bazaar is easy to learn. There is an excellent 5 Minute Tutorial that should get you started. Beyond that, bzr has a fairly well developed man page and in-program help system.
Please read and do the 5 Minute Tutorial now, which will teach you the basics like creating branches, logging into LP, pushing and merging.
Trees, Branches, Trunks?
The basic top level of a codebase is called the Trunk. The Trunk is the mainline code that all branches are a part of. Branches are localized copies of Trunk. Branches are where you make changes for submission and merge consideration. Once accepted, Branches are merged into Trunk and become part of the mainline codebase.
- The most basic command is:
- To get a local copy of a project:
bzr branch lp:opencompute/opencompute-testing
- To review a change summary:
- To add new files that are not yet managed:
bzr add path/to/filename
- Note that /path/to/filename needs to be a path INSIDE your branch. When adding files not in the branch, you need to copy those files into the branch before adding them.
- To see a diff of changes:
bzr diff path/to/filename
- To commit your changes to a new revision
bzr commit -m "Some descriptive summary of your changes"
- To push your changes to launchpad in preparation for a Merge Request:
bzr push lp:~<Your Launchpad Username>/opencompute/<Your Branch Name>
- for example:
bzr push lp:~bladernr/opencompute/ocp-testing-job-updates
- for example:
There are many MANY more commands but those are the ones you'll likely use most frequently.
Languages to use
Plainbox, Checkbox, the providers and such are all written in Python3.
Job Files and Whitelists
Job Files are the same as they were in Checkbox, that is RFC822 compliant text files. See any file in the
jobs/ directory for examples.
Whitelists are also simply text files, HOWEVER, you can also use regular expressions in job naming to pattern match jobs.
These are scripts that perform actual testing. For example, a storage IO test, or a network performance test. Scripts can be written in pretty much anything that is compiled or has a valid interpreter installed on the System Under Test. The following are just a few examples of the languages you can write test scripts in:
And some more complex ones
'Please note that compiled languages will need special consideration and handling in order to get the packages to build properly. Packages are built for AMD64 and i386 for now. We have the capability to also compile them for OpenPower, ARM and ARM64, but that will require some porting work.
In general, we advise the following for test scripts:
- Python 3 for pretty much everything
- BASH for one-liners and short simple scripts. Generally, if it starts feeling complex, it's time to re-write in Python3
- If you MUST use compiled languages, C but this is exceedingly rare and should only be done when something easier to maintain is not appropriate
We currently do not have any Translations for the tooling, nor are there plans to enable this.
What this provides
This is the trunk for the Open Compute Project's certification tooling. It can be found on Launchpad: https://code.launchpad.net/opencompute/opencompute-testing
This code is used to build the launcher packages:
and the provider package:
What's a Provider?
A Provider is a package that provides the bits necessary for Plainbox/Checkbox-NG to perform testing. These files include job descriptions, whitelists, test scripts and wrappers, helpers, custom libraries and anything else that you may need.
What's a Launcher?
A Launcher is a package that does two things.
- It provides the actual program launcher (the command you issue to start the testing) such as
- It ensures the lower-level packages and dependencies are installed
What this provides
This is the main project for Canonical's certification tool. This provides the back-end and underlying support structure as well as some of the general test scripts and jobs. The following packages make up our use of Checkbox, in order from top to bottom.
This package provides the actual checkbox test tools used by Canonical Certification. Many tests in OCP Certification and OCP Ready are based on the jobs, scripts and data in this provider.
This package provides system resource jobs and libraries. These are used at run-time to gather system information and provide that data to CheckboxNG for job generation, to the results files for system hardware inventory, and to provide the validation that allows certain jobs to run.
To explain that last statement, one job of this is to determine if certain hardware exists, and if that hardware exists, then the jobs that test that hardware are configured and executed. For example, there is a USB resource that determines if a given motherboard has a USB controller. If the USB Controller is detected by the USB Resource, then the USB test jobs will be run. If the USB Resource does NOT detect a USB controller, then the USB jobs will be skipped.
This package is an infrastructure package that provides several python3 support libraries that Checkbox needs to run.
This package is the back-end. It provides Checkbox-NG, the Plainbox based test harness that replaces the old Checkbox tool. It provides the internals that organize and run all the various test cases, performs all the parsing and heavy work of Certification Testing.
For the most part, Documentation for the OCP Tools can be found here on the OCP Certification Wiki. For Checkbox, documentation can be found:
As CheckboxNG and Plainbox are under active development, this documentation is constantly evolving and improving.
How can I contribute to this?
There are several ways to contribute. For the most part, you'll likely want to contribute to test scripts, jobs and whitelist changes, so you will mostly be interested in the opencompute-testing project.
To get started, follow the bits above to branch the project (lp:opencompute/opencompute-testing) and start hacking!
If you'd like to contribute to the Checkbox project, please see the Questions? section below.
Putting it all together
So now that you know about the various packages that make up the OCP Certification and OCP Ready test tools, let's install them so you can try them out to see what they do.
Installation of Packages
First, we need to install the PPAs. Because this is based on Canonical's actual certification tooling instead of a fork, we need to add two PPAs:
sudo apt-add-repository ppa:opencompute-developers/ocp-certification-tools-ppa
sudo apt-add-repository ppa:hardware-certification/public
Next, we need to update the apt cache:
sudo apt update
And finally, install the packages:
sudo apt-get install opencompute-certification
After you have done the above, you will see a long list of packages that need to be installed. These all provide testing tools, libraries and other things necessary for Certification testing. The number of packages will vary depending on how you installed the OS onto the SUT, what version is being used, what tool is used (Ready vs Certification) and so forth.
Hit Y and the installation will begin.
Once installation is complete, to check it out qickly, simply run the command
To do proper testing, there is some configuration and other things that need to be performed first, but this will get you an idea of how the tool works and what it looks like.
Obtaining the Code
For now, there are multiple locations to find testing tools. In the near future, these will be merged into the Launchpad branches so that all code is easily found and maintained.
The main project code is housed in the following projects on Launchpad:
ITRI has placed the majority of the Certification test scripts, job files and such on GitHub:
Pulling the code
See above about using bzr to obtain local branches of the Checkbox or OpenCompute Testing trunks.
For the ITRI github location, you'll need to use git.
Adding new code
The following is meant to help you understand where to add things in the code.
The trunk contains the following directories:
- This directory contains all the test scripts that are not part of the lp:checkbox branch. These are the OCP specific test scripts.
- This is a copy of the license. All code is Open Source under the GPLv3. Proprietary code is not allowed in the certification suite.
- This directory is used in case sample test data needs to be installed along with the test suite.
- This directory contains all the necessary Debian packaging information. It also contains the Changelog. Likely, you will only ever need to edit the changelog in this directory.
- This directory contains sample config files that are installed when a given package is installed.
- This directory contains the job description files. Jobs are the units of Test that are performed.
- This directory contains the launcher file that creates the command used to execute the test tool on the SUT
- This is a helper program for testing and building packages.
- UNUSED CURRENTLY, but this would house translation data if implemented.
- A README file that gives you basic development information. Please read this file.
- A description of the trunk contents (the same thing you're reading right now)
- Package building requirements.
- The whitelists that define each test suite.
As discussed before, job files contain the descriptions that define each test case or unit of test.
Basic Job Description
An example job description looks like this:
plugin: shell name: memory/ocp/stress_30min user: root command: stressapptest -s 1800 _description: Test and exercise memory.
This is a very basic example that contains most of the common KEY:VALUE pairs that are used to describe a test.
- Plugins are the type of job to perform.
- shell is a fully automated job that is executed by the test harness automatically with no user intervention.
- manual is a test/job that requires the user to perform some task and verify the results.
- user-interact which also requires the user to initiate the test with the PASS or FAIL results being automatically determined by Checkbox
- user-interact-verify requires the user to initiate the test AND validate the results, choosing PASS or FAIL afterwards.
- This is a descriptive name of the test. In this case, the hierarchy means that it's a Memory test that is part of the OCP Specific tests and the test case is 'stress_30min'
- Some jobs can be run as an alternate user (in fact, some require this). So specifying root here will tell Checkbox to run the command as the root user (essentially using sudo to elevate privileges for this one test only).
- This is the command to perform. This item can be in-line code (like a bash oneliner or test) or the name of a command that is in the bin/ directory.
- This is a text description of the job. For a Manual job, description would list the steps necessary to perform the test. For an automated test, this is just a summary of what the test does.
There are also things we refer to as "Metajobs". These jobs are actually run early on and are simply used to generate jobs on the fly. In general, you would use a metajob to create separate jobs for multiple pieces of the same hardware. In the following example, we have a metajob that creates a separate storage test for any disk device (NON-USB) discovered.
plugin: local name: disk/ocp/io_stress requires: device.category == 'DISK' package.name == 'stressapptest' _description: Verify that storage devices, such as Fibre Channel and RAID can be detected and perform under stress. command: cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"' plugin: shell name: disk/ocp/io_stress_`ls /sys$path/block` user: root requires: device.path == "$path" block_device.`ls /sys$path/block`_state != 'removable' description: Disk I/O stress test for $product command: disk_stress `ls /sys$path/block | sed 's|!|/|'` EOF
Note the differences between a Metajob and a normal Job.
- local is a special plugin that tells Checkbox to run this job as soon as the tool starts. This means this job is executed well before the user ever sees a Suite or Test Selection screen.
- This job also introduces the requires directive. This means that the following conditions must be met before the job will be run.
- Also note that you can specify multiple requirements by placing each one on a new line with a single-space indention.
- In the basic example, I mentioned that a Command can be embedded shell code. This is an example of that embedded shell code. This runs the shell code that follows. In this example, it cats all the text between the 'cat' line and the 'EOF' marker (which looks exactly like the Basic Job Description) and pipes all that through the 'run-templates' helper tool which creates on-the-fly jobs. In this case, 'run-templates' also executes the udev-resource resource gathering script and ensures that child jobs are ONLY created for hardware devices that fit into the DISK category (so we don't create these Disk jobs for network devices or CPUs).
Scritps are found in the bin/ directory. These can be anything from bash to python to ruby to anything you feel like coding in. Scripts can be actual tests themselves, or wrappers that call other tools or test suites. In the past, we have had wrapper scripts that ran things like Phoronix Test Suite and Piglit (for desktop testing).
Whitelists are text lists that define the test suite. This is a list of tests to run. Below is a sample whitelist:
# Resource Jobs miscellanea/submission-resources #Info attachment jobs __info__ cpuinfo_attachment dmesg_attachment dmi_attachment dmidecode_attachment efi_attachment lspci_attachment lshw_attachment mcelog_attachment meminfo_attachment modprobe_attachment modules_attachment sysctl_attachment sysfs_attachment udev_attachment lsmod_attachment acpi_sleep_attachment info/hdparm info/hdparm_.*.txt installer_debug.gz info/disk_partitions # Actual test cases __TC-003__ TC-003-0003-001/Network_performance TC-003-0003-001/Network_performance_.*
The first job, miscellanea/submission-resources is always required in a whitelist. It is what triggers all the various resource jobs to run as well as jobs necessary to create a full submission file. After that, add whatever jobs you need. The first group in the example above are jobs that attach files containing the output of various commands.
In the next section is an example of two things. The job TC-003-0003-001/Network_performance is a "metajob" that generates other jobs. IN this case, it's a job that creates a separate network test for each NIC discovered on the SUT. The next job, TC-003-0003-001/Network_performance_.* is an example of using a regular expression. In this case, this line item tells Checkbox to run any job that starts with the text 'TC-003-0003-001/Network_performance_' (which when run would be jobs like 'TC-003-0003-001/Network_performance_eth0' and 'TC-003-0003-001/Network_performance_eth1'.