Editing C&I Wiki Portal/ToolsDevelopmentGuideline

== Overview ==
The official repository for the source code to OCP Certification and OCP Ready is [http://launchpad.net/opencompute Launchpad].  There are a number of projects hosted there but the most important one is [http://code.launchpad.net/opencompute-testing opencompute-testing].

The '''opencompute-testing''' project contains all the code for generating the OCP Certification and OCP Ready testing tools.  It is the code that creates the Plainbox Provider and Launcher Packages, hosting the job descriptions that are OCP Specific, scripts and whitelists.  Package builds occur automatically once per day if the code base has been updated.

=== Contacts ===
If you have ANY questions at all about the code base, contributing, or anything else related, contact [mailto:jeff@canonical.com Jeff Lane]
For anything related to Open Compute Certification (including code questions):
* Mailing List: http://lists.opencompute.org/mailman/listinfo/opencompute-ci
* IRC: [http://freenode.net Freenode IRC Servers] #ocp-ci

== Basics ==
This section will cover some of the basics for contributing to the code for these tools.  This is a general overview, more specific information can be found on line at the links below.

=== Bazaar ===
==== About Bazaar ====
Bazaar is a version control system that helps you track project history over time and to collaborate easily with others. Whether you're a single developer, a co-located team or a community of developers scattered across the world, Bazaar scales and adapts to meet your needs. Part of the GNU Project, Bazaar is free software sponsored by Canonical

You can learn more at http://bazaar.canonical.com/en/

==== Obtaining Bazaar ====
To install Bazaar on a debian or ubuntu system:
<code>sudo apt-get install bazaar</code>

For other Operating Systems, including Windows, Red Hat, SuSE, OSX and more, please see http://wiki.bazaar.canonical.com/Download

==== Learning Bazaar ====
Bazaar is easy to learn.  There is an excellent [http://doc.bazaar.canonical.com/latest/en/mini-tutorial/ 5 Minute Tutorial] that should get you started.  Beyond that, bzr has a fairly well developed man page and in-program help system.  

Please read and do the 5 Minute Tutorial now, which will teach you the basics like creating branches, logging into LP, pushing and merging.

===== Trees, Branches, Trunks? =====
The basic top level of a codebase is called the Trunk.  The Trunk is the mainline code that all branches are a part of.  Branches are localized copies of Trunk.  Branches are where you make changes for submission and merge consideration.  Once accepted, Branches are merged into Trunk and become part of the mainline codebase.  

===== Basic Commands =====
* The most basic command is: <code>bzr</code>
* To get a local copy of a project: <code>bzr branch lp:opencompute/opencompute-testing</code>
* To review a change summary: <code>bzr status</code>
* To add new files that are not yet managed: <code>bzr add path/to/filename</code>
** Note that /path/to/filename needs to be a path INSIDE your branch.  When adding files not in the branch, you need to copy those files into the branch before adding them.
* To see a diff of changes: <code>bzr diff path/to/filename</code>
* To commit your changes to a new revision <code>bzr commit -m "Some descriptive summary of your changes"</code>
* To push your changes to launchpad in preparation for a Merge Request: <code>bzr push lp:~<Your Launchpad Username>/opencompute/<Your Branch Name></code>
** for example: <code>bzr push lp:~bladernr/opencompute/ocp-testing-job-updates</code>

There are many MANY more commands but those are the ones you'll likely use most frequently.

===== Learning More =====
* http://doc.bazaar.canonical.com/en/
* http://wiki.bazaar.canonical.com/Tutorials
* https://yade-dem.org/wiki/Quick_Bazaar_tutorial
* http://wiki.bazaar.canonical.com/Tutorials

=== Practices ===
==== Languages to use ====
===== Main Tooling =====
Plainbox, Checkbox, the providers and such are all written in Python3.

===== Job Files and Whitelists =====
Job Files are the same as they were in Checkbox, that is RFC822 compliant text files.  See any file in the <code>jobs/</code> directory for examples.
Whitelists are also simply text files, HOWEVER, you can also use regular expressions in job naming to pattern match jobs.

===== Test Scripts =====
These are scripts that perform actual testing.  For example, a storage IO test, or a network performance test. Scripts can be written in pretty much anything that is compiled or has a valid interpreter installed on the System Under Test.  The following are just a few examples of the languages you can write test scripts in:
* Python3
* Bash
* Ruby
* Perl

And some more complex ones
* C/C++
* Go
'Please note that compiled languages will need special consideration and handling in order to get the packages to build properly.  Packages are built for AMD64 and i386 for now.  We have the capability to also compile them for OpenPower, ARM and ARM64, but that will require some porting work.

In general, we advise the following for test scripts:
* Python 3 for pretty much everything
* BASH for one-liners and short simple scripts.  Generally, if it starts feeling complex, it's time to re-write in Python3
* If you MUST use compiled languages, C but this is exceedingly rare and should only be done when something easier to maintain is not appropriate

==== Translations ====
We currently do not have any Translations for the tooling, nor are there plans to enable this.

==== Style Guide ====
The general rule is Python code should adhere to [https://www.python.org/dev/peps/pep-0008/ Python Foundation's PEP-8].  There are several tools to help you check your code for adherence such as [https://pypi.python.org/pypi/pep8/1.6.2 pep8] and [http://www.pylint.org/ Pylint] and many others.

== Project Hierarchy ==
=== opencompute-certification ===
==== What this provides ====
This is the trunk for the Open Compute Project's certification tooling.  It can be found on Launchpad:
https://code.launchpad.net/opencompute/opencompute-testing

This code is used to build the launcher packages:
* opencompute-ready
* opencompute-certification

and the provider package:
* plainbox-provider-opencompute-certification

==== What's a Provider? ====
A Provider is a package that provides the bits necessary for Plainbox/Checkbox-NG to perform testing.  These files include job descriptions, whitelists, test scripts and wrappers, helpers, custom libraries and anything else that you may need.

==== What's a Launcher? ====
A Launcher is a package that does two things.  
# It provides the actual program launcher (the command you issue to start the testing) such as <code>opencompute-certification</code>
# It ensures the lower-level packages and dependencies are installed
=== checkbox ===
==== What this provides ====
This is the main project for Canonical's certification tool.  This provides the back-end and underlying support structure as well as some of the general test scripts and jobs.  The following packages make up our use of Checkbox, in order from top to bottom.

===== plainbox-provider-checkbox =====
This package provides the actual checkbox test tools used by Canonical Certification. Many tests in OCP Certification and OCP Ready are based on the jobs, scripts and data in this provider.

===== plainbox-provider-resource-generic =====
This package provides system resource jobs and libraries.  These are used at run-time to gather system information and provide that data to CheckboxNG for job generation, to the results files for system hardware inventory, and to provide the validation that allows certain jobs to run.

To explain that last statement, one job of this is to determine if certain hardware exists, and if that hardware exists, then the jobs that test that hardware are configured and executed.  For example, there is a USB resource that determines if a given motherboard has a USB controller.  If the USB Controller is detected by the USB Resource, then the USB test jobs will be run.  If the USB Resource does NOT detect a USB controller, then the USB jobs will be skipped.

===== python3-checkbox-support =====
This package is an infrastructure package that provides several python3 support libraries that Checkbox needs to run.

===== checkbox-ng =====
This package is the back-end.  It provides Checkbox-NG, the Plainbox based test harness that replaces the old Checkbox tool.  It provides the internals that organize and run all the various test cases, performs all the parsing and heavy work of Certification Testing.

==== Documentation? ====
For the most part, Documentation for the OCP Tools can be found here on the OCP Certification Wiki.  For Checkbox, documentation can be found:
* http://checkbox.readthedocs.org/en/latest/
* http://plainbox.readthedocs.org/en/latest/

As CheckboxNG and Plainbox are under active development, this documentation is constantly evolving and improving.

==== How can I contribute to this? ====
There are several ways to contribute.  For the most part, you'll likely want to contribute to test scripts, jobs and whitelist changes, so you will mostly be interested in the '''opencompute-testing''' project.

To get started, follow the bits above to branch the project (lp:opencompute/opencompute-testing) and start hacking!

If you'd like to contribute to the Checkbox project, please see the [[#Questions?|Questions?]] section below.

=== Putting it all together ===
So now that you know about the various packages that make up the OCP Certification and OCP Ready test tools, let's install them so you can try them out to see what they do.
==== Installation of Packages ====
First, we need to install the PPAs.  Because this is based on Canonical's actual certification tooling instead of a fork, we need to add two PPAs:

<code>
sudo apt-add-repository ppa:opencompute-developers/ocp-certification-tools-ppa

sudo apt-add-repository ppa:hardware-certification/public
</code>

Next, we need to update the apt cache:

<code>sudo apt update</code>

And finally, install the packages:

<code>sudo apt-get install opencompute-certification</code>

==== Interaction ====
After you have done the above, you will see a long list of packages that need to be installed.  These all provide testing tools, libraries and other things necessary for Certification testing.  The number of packages will vary depending on how you installed the OS onto the SUT, what version is being used, what tool is used (Ready vs Certification) and so forth.  

Hit Y and the installation will begin.

Once installation is complete, to check it out qickly, simply run  the command <code>opencompute-certification</code> or <code>opencompute-ready</code>

To do proper testing, there is some configuration and other things that need to be performed first, but this will get you an idea of how the tool works and what it looks like.

== Obtaining the Code ==
=== Locations ===
For now, there are multiple locations to find testing tools.  In the near future, these will be merged into the Launchpad branches so that all code is easily found and maintained.
==== Launchpad ====
The main project code is housed in the following projects on Launchpad:
* [https://code.launchpad.net/opencompute/opencompute-testing lp:opencompute/opencompute-testing]
* [https://code.launchpad.net/checkbox lp:checkbox]
==== GitHub ====
ITRI has placed the majority of the Certification test scripts, job files and such on GitHub:
* https://github.com/opencomputeproject/ocp_ci_server_certification_test_cases
=== Pulling the code ===
See above about using '''bzr''' to obtain local branches of the Checkbox or OpenCompute Testing trunks.

For the ITRI github location, you'll need to use git.

== Adding new code ==
The following is meant to help you understand where to add things in the code.
=== opencompute-certification ===
==== Trunk Contents ====
The trunk contains the following directories:
;bin
:This directory contains all the test scripts that are not part of the lp:checkbox branch.  These are the OCP specific test scripts.
;COPYING
:This is a copy of the license.  All code is Open Source under the GPLv3.  Proprietary code is not allowed in the certification suite.
;data
:This directory is used in case sample test data needs to be installed along with the test suite.
;debian
:This directory contains all the necessary Debian packaging information.  It also contains the Changelog.  Likely, you will only ever need to edit the changelog in this directory.
;examples
:This directory contains sample config files that are installed when a given package is installed.
;jobs
:This directory contains the job description files.  Jobs are the units of Test that are performed.
;launcher
:This directory contains the launcher file that creates the command used to execute the test tool on the SUT
;manage.py
:This is a helper program for testing and building packages.
;po
:UNUSED CURRENTLY, but this would house translation data if implemented.
;README.development
:A README file that gives you basic development information.  Please read this file.
;README.rst 
:A description of the trunk contents (the same thing you're reading right now)
;requirements 
:Package building requirements.
;whitelists
:The whitelists that define each test suite.

==== Job Files ====
As discussed before, job files contain the descriptions that define each test case or unit of test.  
===== Basic Job Description =====
An example job description looks like this:

<pre>
plugin: shell
name: memory/ocp/stress_30min
user: root
command: stressapptest -s 1800
_description:
 Test and exercise memory.
</pre>

This is a very basic example that contains most of the common KEY:VALUE pairs that are used to describe a test.
;plugin
:Plugins are the type of job to perform.  
:'''shell''' is a fully automated job that is executed by the test harness automatically with no user intervention.  
:'''manual''' is a test/job that requires the user to perform some task and verify the results.
:'''user-interact''' which also requires the user to initiate the test with the PASS or FAIL results being automatically determined by Checkbox
:'''user-interact-verify''' requires the user to initiate the test AND validate the results, choosing PASS or FAIL afterwards.
;name
:This is a descriptive name of the test.  In this case, the hierarchy means that it's a Memory test that is part of the OCP Specific tests and the test case is 'stress_30min'
;user 
:Some jobs can be run as an alternate user (in fact, some require this).  So specifying '''root''' here will tell Checkbox to run the '''command''' as the root user (essentially using sudo to elevate privileges for this one test only).
;command
:This is the command to perform.  This item can be in-line code (like a bash oneliner or test) or the name of a command that is in the bin/ directory.
;_description
:This is a text description of the job.  For a Manual job, '''description''' would list the steps necessary to perform the test.  For an automated test, this is just a summary of what the test does.

===== Meta-Jobs =====
There are also things we refer to as "Metajobs".  These jobs are actually run early on and are simply used to generate jobs on the fly.  In general, you would use a metajob to create separate jobs for multiple pieces of the same hardware.  In the following example, we have a metajob that creates a separate storage test for any disk device (NON-USB) discovered.

<pre>
plugin: local
name: disk/ocp/io_stress
requires:
 device.category == 'DISK'
 package.name == 'stressapptest'
_description: Verify that storage devices, such as Fibre Channel and RAID can be detected and perform under stress.
command:
 cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
 plugin: shell
 name: disk/ocp/io_stress_`ls /sys$path/block`
 user: root
 requires:
  device.path == "$path"
  block_device.`ls /sys$path/block`_state != 'removable'
 description: Disk I/O stress test for $product
 command: disk_stress `ls /sys$path/block | sed 's|!|/|'`
 EOF
</pre>

Note the differences between a Metajob and a normal Job.
;Plugin
:'''local''' is a special plugin that tells Checkbox to run this job as soon as the tool starts. This means this job is executed well before the user ever sees a Suite or Test Selection screen.
;requires
:This job also introduces the '''requires''' directive.  This means that the following conditions must be met before the job will be run.  
:Also note that you can specify multiple requirements by placing each one on a new line with a single-space indention.
;Command
:In the basic example, I mentioned that a '''Command''' can be embedded shell code.  This is an example of that embedded shell code.  This runs the shell code that follows.  In this example, it cats all the text between the 'cat' line and the 'EOF' marker (which looks exactly like the Basic Job Description) and pipes all that through the 'run-templates' helper tool which creates on-the-fly jobs.  In this case, 'run-templates' also executes the udev-resource resource gathering script and ensures that child jobs are ONLY created for hardware devices that fit into the DISK category (so we don't create these Disk jobs for network devices or CPUs).

==== Scripts ====
Scritps are found in the bin/ directory.  These can be anything from bash to python to ruby to anything you feel like coding in.  Scripts can be actual tests themselves, or wrappers that call other tools or test suites.  In the past, we have had wrapper scripts that ran things like Phoronix Test Suite and Piglit (for desktop testing).

==== Whitelists ====
Whitelists are text lists that define the test suite.  This is a list of tests to run.  Below is a sample whitelist:
<pre>
# Resource Jobs
miscellanea/submission-resources
#Info attachment jobs
__info__
cpuinfo_attachment
dmesg_attachment
dmi_attachment
dmidecode_attachment
efi_attachment
lspci_attachment
lshw_attachment
mcelog_attachment
meminfo_attachment
modprobe_attachment
modules_attachment
sysctl_attachment
sysfs_attachment
udev_attachment
lsmod_attachment
acpi_sleep_attachment
info/hdparm
info/hdparm_.*.txt
installer_debug.gz
info/disk_partitions
# Actual test cases
__TC-003__
TC-003-0003-001/Network_performance
TC-003-0003-001/Network_performance_.*
</pre>

The first job, '''miscellanea/submission-resources''' is always required in a whitelist.  It is what triggers all the various resource jobs to run as well as jobs necessary to create a full submission file.  After that, add whatever jobs you need.  The first group in the example above are jobs that attach files containing the output of various commands. 

In the next section is an example of two things.  The job '''TC-003-0003-001/Network_performance''' is a "metajob" that generates other jobs.  IN this case, it's a job that creates a separate network test for each NIC discovered on the SUT.  The next job, '''TC-003-0003-001/Network_performance_.*''' is an example of using a regular expression.  In this case, this line item tells Checkbox to run any job that starts with the text 'TC-003-0003-001/Network_performance_' (which when run would be jobs like 'TC-003-0003-001/Network_performance_eth0' and 'TC-003-0003-001/Network_performance_eth1'.

== Submitting Code to the Project ==
=== Launchpad ===
=== Pushing Changes ===
=== Merge Requests ===

== Questions? ==
=== OCP Ready or OCP Certification===
=== CheckboxNG or Plainbox ===