Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

x-common

Shared metadata for exercism exercises.

Subscribe to updates I use x-common


Statistics on x-common

Number of watchers on Github 138
Number of open issues 85
Average time to close an issue 10 days
Main language Shell
Average time to merge a PR 3 days
Open pull requests 80+
Closed pull requests 36+
Last commit over 1 year ago
Repo Created almost 6 years ago
Repo Last Updated over 1 year ago
Size 1.47 MB
Organization / Authorexercism
Contributors65
Page Updated
Do you use x-common? Leave a review!
View open issues (85)
View x-common activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating x-common for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

problem-specifications

Shared metadata for Exercism exercises.

Contributing Guide

Please see the contributing guide

Problem metadata

Each problem's data lives in a directory under exercises/

exercises/
 accumulate
  description.md
  metadata.yml
 ...
 minesweeper
  canonical-data.json
  description.md
  metadata.yml
 ...
 zipper
     description.md
     metadata.yml

There are three metadata files:

  • description.md - the basic problem description
  • metadata.yml - additional information about the problem, such as where it came from
  • canonical-data.json - standardized test inputs and outputs that can be used to implement the problem

Test Data Format (canonical-data.json)

This data can be incorporated into test programs manually or extracted by a program. The file format is described in canonical-schema.json, but it is easier to understand with an example:

{ "exercise": "foobar"
, "version" : "1.0.0"
, "comments":
    [ " Comments are always optional and can be used almost anywhere.      "
    , "                                                                    "
    , " They usually document how the exercise's readme ('description.md') "
    , " is generally interpreted in test programs across different         "
    , " languages.                                                         "
    , "                                                                    "
    , " In addition to a mainstream implementation path, this information  "
    , " can also document significant variations.                          "
    ]
, "cases":
    [ { "comments":
          [ " A test case must have a 'description' and a 'property'.  "
          , " Anything else is optional.                               "
          , "                                                          "
          , " The 'property' is a string in lowerCamelCase identifying "
          , " the type of test, but most of the times it is just the   "
          , " name of a function being tested.                         "
          , "                                                          "
          , " Test cases can have any number of additional keys, and   "
          , " most of them also have an 'expected' one, defining the   "
          , " value a test should return.                              "
          ]
      , "description": "Foo'ing a word returns it reversed"
      , "property"   : "foo"
      , "input"      : {
          "word"       : "lion"
        }
      , "expected"   : "noil"
      }
    , { "description": "Bar'ing a name returns its parts combined"
      , "property"   : "bar"
      , "input"      : {
          "firstName"  : "Alan",
          "lastName"   : "Smithee"
        }  
      , "expected"   : "ASlmainthee"
      }
    , { "comments":
          [ " Test cases can be arbitrarily grouped with a description "
          , " to make organization easier.                             "
          ]
      , "description": "Abnormal inputs: numbers"
      , "cases":
          [ { "description": "Foo'ing a number returns nothing"
            , "property"   : "foo"
            , "input"      : {
                "word"       : "42"
              }
            , "expected"   : null
            }
          , { "description": "Bar'ing a name with numbers gives an error"
            , "property"   : "bar"
            , "input"      : {
                "firstName"  : "HAL",
                "lastName"   : "9000"
              }  
            , "expected"   : { "error": "You should never bar a number" }
            }
          ]
      }
  ]
}

Keep in mind that the description should not simply explain what each case is (that is redundant information) but also why each case is there. For example, what kinds of implementation mistakes might this case help us find?

There are also some conventions that must be followed:

  • All keys should follow the lowerCamelCase convention.
  • If the input is valid but there is no result for the input, the value at "expected" should be null.
  • If an error is expected (because the input is invalid, or any other reason), the value at "expected" should be an object containing exactly one property, "error", whose value is a string.
    • The string should explain why the error would occur.
    • A particular track's implementation of the exercise need not necessarily check that the error includes that exact string as the cause, depending on what is idiomatic in the language (it may not be idiomatic to check strings for error messages).

Test Data Versioning

Test data should be versioned according to Semantic Version 2.0, which is defined as follows:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes. Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

MAJOR version changes

The MAJOR version should be changed when the test suite is modified in a fundamentally incompatible way.

There are examples of changes requiring a MAJOR version change:

  • A new property (test type).
  • Renaming a property.
  • Insertion, deletion or renaming of keys in the test data object.
  • Changing the type of one of the test data keys.

MAJOR changes should be expected to break even well-behaved test generators.

MINOR version changes

The MINOR version should change when you add functionality in a backwards-compatible manner, make non-breaking changes that alter the meaning of the test suite, make previously passing solutions possibly fail, or failing solutions fail at a different spot.

There are examples of changes requiring a MINOR version change:

  • Adding or deleting test cases.
  • Changing the test cases inputs and/or outputs.
  • Changing the test cases ordering.

MINOR changes would never break well-designed test generators, because the test-generation logic remains exactly the same.

PATCH version changes

The PATCH version should change when you make backwards-compatible bug fixes or whenever the meaning of the tests does not change.

There are examples of changes requiring a PATCH version change:

  • Regrouping/Renesting test cases without changing test case ordering.
  • Changing descriptions or comments.
  • Changing keys' ordering or formatting (would result in an equivalent JSON file).

PATCH changes would never break well-designed test generators, because the test data remains exactly the same.

Automated Tests

canonical-data.json for each exercise is checked for compliance against the canonical-schema.json. In order to run these tests, you will need to have yarn installed on your system. Install them from here.

Install the required packages:

yarn install

Run for all exercises:

yarn test

Run for single exercise:

yarn run test-one exercises/<exercise>/canonical-data.json

Replace <exercise> by the name of exercise which you want to check.

x-common open issues Ask a question     (View All Issues)
  • about 3 years How do we express the intention of non-obvious test cases in our canonical-data.json files?
  • about 3 years Update or remove non-ASCII test cases.
  • about 3 years sublist: Possible missing edge case
  • about 3 years CryptoSquare: add a test for "perfect square" normalized cipher text
  • about 3 years Discussion: Non-Ascii test cases.
  • about 3 years [isogram] Test descriptions
  • about 3 years Update implementations of Bowling
  • about 3 years simple-cipher: fix the description
  • about 3 years Representation of undefined/exceptional values in canonical-data.json
  • about 3 years README of atbash-ciper mentions group size, but there are no tests for it
  • about 3 years Should queen-attack creation tests be improved?
  • about 3 years Add a philosophy document to exercises directories?
  • over 3 years crypto-square examples have trailing spaces stripped, not in README
  • over 3 years canonical-data.json standardisation discussion (was: Malformed data?)
  • over 3 years [secret-handshake]: Clarify readme
  • over 3 years Should the existing text-based exercises be merged?
  • over 3 years Idea for new exercise (Polar bear)
  • over 3 years [hamming] Do we need to restrict this exercise to DNA?
  • over 3 years Is "write a program" accurate?
  • over 3 years Saddle Point: Uses y, x coordinate values, rather than x, y
  • over 3 years Sublist type clarification
  • over 3 years Exercise Proposal: Parsing Math expressions
  • over 3 years [hello-world] Consistency of the empty string
  • over 3 years Deprecate binary, trinary, octal, and hexadecimal in favor of new all-your-base exercise
  • over 3 years The protein-translation and nucleotide-codons exercises are the same
  • over 3 years pangram: missing case-sensitivity edge case
  • over 3 years [binary-search]: Do not implement it for linked lists
  • over 3 years Incomplete specification for run-length encoding?
  • over 3 years Binary search should not require a check for sorted input
  • over 3 years Assemble exercise README including the docs/TESTS.md?
x-common open pull requests (View All Pulls)
  • Add commonised JSON to scrabble score problem
  • secret-handshake: added test json
  • Added difference-of-squares.json test definition
  • add diagram about exercism.io workflow
  • Fix expected output of an anagram test
  • series, largest-series-product: clarify "consecutive"
  • add pangram tests
  • sublist: Add JSON test data
  • sum-of-multiples: Remove default behavior
  • Edit README to clarify exercise instructions
  • Add crypto-square.json
  • added metadata for flatten-array question
  • Shrink & clean up custom_set.json
  • Add canonical tests for Wordy
  • Reorder custom-set tests to improve flow
  • Add space-age.json, extracted from Haskell track.
  • Pangram: Updated test to verify case insensitivity count
  • Added an extra test for the isogram exercise. …
  • Explicitly describe robot name format
  • Common Test Suite for Robot Simulator
  • Add exercise all-your-base.
  • Add ocr-numbers test suite, update description
  • rectangles.json: Explain purpose of large input test
  • tournament: Add JSON file
  • transpose: add .md and .yml
  • Introduce Track Anatomy section from x-api
  • forth: Add canonical data
  • Update bowling canonical-data
  • pov: Add canonical data
  • [Alphametics] Add more easy cases.
  • Triangle: Update tests to check properties, not types
  • Rename flatten array to flatten
  • Amended description.md files to be formatted to 80 chars.
  • pangram: Add case-insensitive test
  • sublist: Add digits edge case test
  • isogram: Revise canonical test data
  • matrix: description: add newlines in sample string
  • [WIP] queen-attack: use Algebraic Notation for coordinates?
  • Add descriptions to Roman numerals canonical data.
  • Note on luhn description for single digits
  • secret-handshake: Add canonical data (revives #159)
  • circular-buffer: Add canonical data
  • variable-length-quantity: Add canonical data
  • [WIP] Make description.md self contained.
  • Implement the two-fer exercise
  • Add Extensions to RobotName
  • New exercise: complex numbers
  • proverb: fix description formatting
  • scale-generator: fix table markdown
  • Delete documentation about updating exercise READMEs
  • Delete documentation about porting exercises
  • Delete documentation about canonical data
  • Delete stub about writing test generators
  • Delete documentation about maintaining a track
  • secret-handshake: remove tests for numbers > 31
  • reverse-string: apply "input" policy
  • rectangles: apply "input" policy
  • raindrops: apply "input" policy
  • rail-fence-cipher: apply "input" policy
  • queen-attack: apply "input" policy
  • proverb: apply "input" policy
  • scale-generator: add canonical data
  • protein-translation: apply "input" policy
  • prime-factors: apply "input" policy
  • poker: apply "input" policy
  • palindrome-products: apply "input" policy
  • ocr-numbers: apply "input" policy
  • kindergarten-garden: apply "input" policy
  • space-age: Update description.md
  • complex-numbers: exponential of complex number
  • travis: Choose schema based on USE_OLD_SCHEMA file (strict)
  • problem-specifications: Replace use of node/npm with yarn
  • series: Implement canonical-data.json
  • meetup: grouping testcase and update descriptions
  • simple-cipher: Add canonical data
  • pig-latin: improve documentation
  • react: Test callbacks on multiple cells
  • react: Fold expect_callback_values into set_value
  • go-counting: Updating documentation
  • crypto-square: Clarify rectangular output requirement in description.md
x-common questions on Stackoverflow (View All Questions)
  • How can I exclude jbossws libs (in Jboss 5.1.x common/lib) from being loaded by my project?
x-common list of languages used
Other projects in Shell