C&I Wiki Portal/ToolsDevelopmentGuideline
Jump to navigation
Jump to search
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
fill this in!
=== opencompute-certification === ==== Trunk Contents ==== The trunk contains the following directories: ;bin :This directory contains all the test scripts that are not part of the lp:checkbox branch. These are the OCP specific test scripts. ;COPYING :This is a copy of the license. All code is Open Source under the GPLv3. Proprietary code is not allowed in the certification suite. ;data :This directory is used in case sample test data needs to be installed along with the test suite. ;debian :This directory contains all the necessary Debian packaging information. It also contains the Changelog. Likely, you will only ever need to edit the changelog in this directory. ;examples :This directory contains sample config files that are installed when a given package is installed. ;jobs :This directory contains the job description files. Jobs are the units of Test that are performed. ;launcher :This directory contains the launcher file that creates the command used to execute the test tool on the SUT ;manage.py :This is a helper program for testing and building packages. ;po :UNUSED CURRENTLY, but this would house translation data if implemented. ;README.development :A README file that gives you basic development information. Please read this file. ;README.rst :A description of the trunk contents (the same thing you're reading right now) ;requirements :Package building requirements. ;whitelists :The whitelists that define each test suite. ==== Job Files ==== As discussed before, job files contain the descriptions that define each test case or unit of test. ===== Basic Job Description ===== An example job description looks like this: <pre> plugin: shell name: memory/ocp/stress_30min user: root command: stressapptest -s 1800 _description: Test and exercise memory. </pre> This is a very basic example that contains most of the common KEY:VALUE pairs that are used to describe a test. ;plugin :Plugins are the type of job to perform. :'''shell''' is a fully automated job that is executed by the test harness automatically with no user intervention. :'''manual''' is a test/job that requires the user to perform some task and verify the results. :'''user-interact''' which also requires the user to initiate the test with the PASS or FAIL results being automatically determined by Checkbox :'''user-interact-verify''' requires the user to initiate the test AND validate the results, choosing PASS or FAIL afterwards. ;name :This is a descriptive name of the test. In this case, the hierarchy means that it's a Memory test that is part of the OCP Specific tests and the test case is 'stress_30min' ;user :Some jobs can be run as an alternate user (in fact, some require this). So specifying '''root''' here will tell Checkbox to run the '''command''' as the root user (essentially using sudo to elevate privileges for this one test only). ;command :This is the command to perform. This item can be in-line code (like a bash oneliner or test) or the name of a command that is in the bin/ directory. ;_description :This is a text description of the job. For a Manual job, '''description''' would list the steps necessary to perform the test. For an automated test, this is just a summary of what the test does. ===== Meta-Jobs ===== There are also things we refer to as "Metajobs". These jobs are actually run early on and are simply used to generate jobs on the fly. In general, you would use a metajob to create separate jobs for multiple pieces of the same hardware. In the following example, we have a metajob that creates a separate storage test for any disk device (NON-USB) discovered. <pre> plugin: local name: disk/ocp/io_stress requires: device.category == 'DISK' package.name == 'stressapptest' _description: Verify that storage devices, such as Fibre Channel and RAID can be detected and perform under stress. command: cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"' plugin: shell name: disk/ocp/io_stress_`ls /sys$path/block` user: root requires: device.path == "$path" block_device.`ls /sys$path/block`_state != 'removable' description: Disk I/O stress test for $product command: disk_stress `ls /sys$path/block | sed 's|!|/|'` EOF </pre> Note the differences between a Metajob and a normal Job. ;Plugin :'''local''' is a special plugin that tells Checkbox to run this job as soon as the tool starts. This means this job is executed well before the user ever sees a Suite or Test Selection screen. ;requires :This job also introduces the '''requires''' directive. This means that the following conditions must be met before the job will be run. :Also note that you can specify multiple requirements by placing each one on a new line with a single-space indention. ;Command :In the basic example, I mentioned that a '''Command''' can be embedded shell code. This is an example of that embedded shell code. This runs the shell code that follows. In this example, it cats all the text between the 'cat' line and the 'EOF' marker (which looks exactly like the Basic Job Description) and pipes all that through the 'run-templates' helper tool which creates on-the-fly jobs. In this case, 'run-templates' also executes the udev-resource resource gathering script and ensures that child jobs are ONLY created for hardware devices that fit into the DISK category (so we don't create these Disk jobs for network devices or CPUs). ==== Scripts ==== Scritps are found in the bin/ directory. These can be anything from bash to python to ruby to anything you feel like coding in. Scripts can be actual tests themselves, or wrappers that call other tools or test suites. In the past, we have had wrapper scripts that ran things like Phoronix Test Suite and Piglit (for desktop testing). ==== Whitelists ==== Whitelists are text lists that define the test suite. This is a list of tests to run. Below is a sample whitelist: <pre> # Resource Jobs miscellanea/submission-resources #Info attachment jobs __info__ cpuinfo_attachment dmesg_attachment dmi_attachment dmidecode_attachment efi_attachment lspci_attachment lshw_attachment mcelog_attachment meminfo_attachment modprobe_attachment modules_attachment sysctl_attachment sysfs_attachment udev_attachment lsmod_attachment acpi_sleep_attachment info/hdparm info/hdparm_.*.txt installer_debug.gz info/disk_partitions # Actual test cases __TC-003__ TC-003-0003-001/Network_performance TC-003-0003-001/Network_performance_.* </pre> The first job, '''miscellanea/submission-resources''' is always required in a whitelist. It is what triggers all the various resource jobs to run as well as jobs necessary to create a full submission file. After that, add whatever jobs you need. The first group in the example above are jobs that attach files containing the output of various commands. In the next section is an example of two things. The job '''TC-003-0003-001/Network_performance''' is a "metajob" that generates other jobs. IN this case, it's a job that creates a separate network test for each NIC discovered on the SUT. The next job, '''TC-003-0003-001/Network_performance_.*''' is an example of using a regular expression. In this case, this line item tells Checkbox to run any job that starts with the text 'TC-003-0003-001/Network_performance_' (which when run would be jobs like 'TC-003-0003-001/Network_performance_eth0' and 'TC-003-0003-001/Network_performance_eth1'.
Please note that all contributions to OpenCompute may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
Do not submit copyrighted work without permission!
(opens in new window)
Retrieved from "
Not logged in
Help about MediaWiki
What links here