Filter functions

Filters can change the look and format of the source data, or even generate new data derived from the input values. What's important is that the original data is replaced by the result of transformations, and that's what ends up in rendered templates.

Overview

The following functions are supported in addition to the built-in functions provided by Nunjucks.

Low level functions

High level functions

Named arguments

Some functions support named arguments, many of these repeat in different functions.

term - a single string, used as a substring to match with the matched item.

list - a list of strings, trying to match any of the listed substrings with the matched item.

regex - a single string, used as a regular expression with the matched item. A regular expression can be created just like JavaScript, but needs to be prefixed with r, for example, r/^foo.*/g, for more info see Nunjucks.

attr - a key in the element to use when doing the requested operation.

For example, the following expressions provide an identical result:

- {{ 'something' | includes(regex=r/^some.*/) }}
- {{ 'something' | includes(term='some') }}
- {{ 'something' | includes(list=['some']) }}

Reference

capture

Extract the first match of the regex in the input string. If no match is found, the function returns an empty string.

For example, the following line will extract the substring "hello wo" from the input

{{ "hello world" | capture(regex=r/he.+o/) }}

difference

Given two lists, keep only items that are in the 1st list but not in the 2nd.

every

Checks whether all element in the list are true. In case the list of elements is empty, it will return false.

For example, check that all changes are in either 'src' or 'dest' directories:

{{ files | match(list=['src', 'dest']) | every }}

filter

Creates a shallow copy of a portion of a given list, filtered down to just the elements that match the given term. You can use either a single term, regex, or a list of terms to match with.

Examples: Check if all changes to JavaScript files are in the tests directory:

{{ files | filter(regex=r/\.js$/) | match(regex=r/tests\//) | every }}

Check if all changes to JavaScript files are formatting:

{{ source.diff.files | filter(attr='new_file', regex=r/\.js$/) | isFormattingChange }}

Check if the PR has new Python files:

{{ branch.diff.files_metadata | filter(attr='original_file', regex=r/^$/) | filter(attr='new_file', regex=r/\.py$/) | some }}

includes

Determines whether a string includes a certain substring. You can use either a single term, regex, or a list of terms to match with.

Check string matches either of the terms:

{{ 'something' | includes(list=['any', 'thing']) }}

intersection

Given two lists, keep only items that are in both lists.

map

Creates a new list populated with the values of the selected attribute of every element in the input list.

For example, the source.diff.files context holds a list of FileDiff , each has new_file attribute. You can create a list of all the new file names by mapping to the new_file attribute and then check if there are changes to any handler.js file:

{{ source.diff.files | map(attr='new_file') | match(term='handler.js') | some }}

match

Return true for each element in the list that match the search term.

Examples:

Check if all code changes are in the tests directory:

{{ files | match(regex=r/tests\//) | every }}

Check if there are code changes with specific function call:

{{ source.diff.files | match(attr='diff', term='myFunction') | some }}

nope

The inverse of every, checks whether all elements in the list are false. In case the list of elements is empty, it will return true.

For example, check that no changes in either 'src' or 'dest' directories:

{{ files | match(list=['src', 'dest']) | nope }}

reject

Creates a shallow copy of a portion of a given list, filtered down to just the elements that do not match the given term. You can use either a single term, regex, or a list of terms to match with.

Examples:

Check if all changes, but JavaScript files are in tests directory:

{{ files | reject(regex=r/\.js$/) | match(regex=r/tests\//') | every }}

Check if all changes except for config.json files are formatting:

{{ source.diff.files | reject(attr='new_file', regex=r/config\.json$/) | isFormattingChange }}

some

Checks whether any element in the list is true. In case the list of elements is empty it will return false.

{{ files | match(list=['src', 'dest']) | some }}

allDocs

Return true if the input list includes only documents based on file extensions.

Doc files extensions are: md, mkdown, txt, rst, adoc, except for requirements.txt.

{{ files | allDocs }}

In case you want to exclude more files, like all txt under the requirements directory, add another check:

{{ (files | allDocs) and (files | match(regex=r/requirements\/.*\.txt$/) | nope ) }}

allImages

Return true if the input list includes only images based on file extensions.

Image file extensions are: svg, png, gif.

{{ files | allImages }}

allTests

Return true if the input list includes only tests based on the file's path and name.

To identify as test the file must include the word test or spec in its name or path, it is checked using this regex: [^a-zA-Z0-9](spec|test|tests)[^a-zA-Z0-9].

{{ files | allTests }}

codeExperts

When requesting a review for a pull request, it's important to select a reviewer who has a deep understanding of the relevant code area, the domain problem, and the framework being used. This ensures that the reviewer can provide specific and informed feedback, rather than general comments that may not take into account the context in which the issue was solved.

The filter provides the list of most qualified contributors based on git-blame and git-commit results to determine who has been most active in the relevant code area, and then combines this information into a score between 0 and 100. The commit activity is scored higher for recent commits, which ensures that those who are actively contributing to the codebase are given higher priority as potential reviewers. The result will be limited to 2 users and shall not include the PR author.

The output lists the Git provider users, e.g., GitHub users, which are mapped from the Git users included in the git-blame output. When gitStream cannot map the Git user to a Git provider user it will be dropped from the output list, hence the list may contain less than 100% of the lines.

For example:

automations:
  code_experts:
    if:
      - true
    run:
      - action: add-reviewers@v1
        args:
          reviewers: {{ repo | codeExperts(gt=10) }}

decode

Decode Base64 encoded string into an object. Encoded strings are formatted: "base64: <encoded_string>"

{{ "base64: SGVsbG8gV29ybGQ=" | decode }} # Output: "Hello World"

encode

Encode data into Base64 encoded string. When an encoded string is passed as input for add-comment, the action automatically detects and decodes it.

{{ "Hello World" | encode }} # Output: "base64: SGVsbG8gV29ybGQ="

estimatedReviewTime

Returns the estimated review time in minutes based on an ML model. The model estimation is computed based on the PR metadata data (e.g. branch name, commits) and mainly by the number of additions and deletions for each type of change (Code, Data, Configuration, etc..)

{{ branch | estimatedReviewTime }}

The following files are automatically excluded from the estimated review time calculation.

extensions

Expects files and provide a list of all unique file extensions.

For example, check that only one file type was changed:

{{ files | extensions | length == 1 }}

extractJitFindings

Get an object with a summary of the findings found by Jit scan. This filter is relevant only for repos that use Jit to scan PRs

The pr context includes all the reviews in the pull request, including the reviews written by the Jit bot, along with all the comments (conversations) to the review.

This filter reads and parses the reviews with Jit's findings, making them available for use inside the .cm file automations.

The output is an object of the following format:

{
  "vulnerabilities": [{
    "security_control": 'string',
    "type": 'string',
    "description": 'string',
    "severity": 'string',
    "summary": 'string'
  }],
  "metrics": {
    "HIGH": number,
    "MEDIUM": number,
    "LOW": number,
    "INFO": number
  }
}

Example of the filter output

{
  "vulnerabilities": [
    {
      "security_control": "Static Code Analysis Js",
      "type": "Codsec.Javascriptnosql-Injection.Nosql-Injection",
      "description": "Putting request data into a mongo query can leadto a NoSQL Injection. Be sure to properly sanitize thedata if you absolutely must pass request data into a query.",
      "severity": "HIGH",
      "summary": "Jit Bot commands and options (e.g., ignore issue)"
    },
    {
      "security_control": "Secret Detection",
      "type": "Private-Key",
      "description": "Private Key",
      "severity": "HIGH",
      "summary": "Jit Bot commands and options (e.g., ignore issue)"
    }
  ],
  "metrics": {
    "HIGH": 2,
    "MEDIUM": 0,
    "LOW": 0,
    "INFO": 0
  }
}

Assign the output to a variable

jit: {{ pr | extractJitFindings }}

Add a label if Jit detected secrets in the PR

automations:
    add_bugs_label:
      if:
        - {{ jit.metrics.HIGH > 0 }}
      run:
        - action: add-label@v1
          args:
            label: "Vulnerable code!""

extractSonarFindings

Get an object with a summary of the findings found by the SonarCloud scan. This filter is relevant only for repos that use SonarCloud to scan PRs

The pr context includes all the comments added to the pull request, including the comment written by the SonarCloud bot that holds a summary of its scan.

This filter reads and parses the comment with SonarCloud's scan summary and makes them available to use inside the .cm file automations.

The output is an object of the following format:

{
  "bugs": {
      "count": number,
      "rating": 'string' //('A'-'E')
    },
  "code_smells": {
      "count": number,
      "rating": 'string' //('A'-'E')
    },
  "vulnerabilities": {
      "count": number,
      "rating": 'string' //('A'-'E')
    },
  "security_hotspots": {
      "count": number,
      "rating": 'string' //('A'-'E')
    },
  "duplications": number,
  "coverage": number
}

Example of the filter output

{
  "bugs": {
      "count": 1,
      "rating": 'B'
    },
  "code_smells": {
      "count": 2,
      "rating": 'B'
    },
  "vulnerabilities": {
      "count": 2,
      "rating": 'E'
    },
  "security_hotspots": {
      "count": 0,
      "rating": 'A'
    },
  "duplications": 3,
  "coverage": 70
}

Assign the output to a variable

sonar: {{ pr | extractSonarFindings }}

Add a label with the number of bugs if the bugs rating is other than 'A', and use mapToEnum to set its color

automations:
# Add Bugs label
  show_bugs_count:
    if:
      - {{ sonar.bugs.count > 0}}
    run:
      - action: add-label@v1
        args:
          label: '🐞 x {{ sonar.bugs.count }} Bugs'
          color: {{ sonar.bugs.rating | mapToEnum(enum = colors) }}

colors:
  A: '05AA02'
  B: 'B6D146'
  C: 'EABE05'
  D: 'DF8339'
  E: 'D4343F'

explainRankByGitBlame

This filter helps to explain the results of rankByGitBlame, the output is in Markdown format that can be used in a PR comment.

The output lists the Git provider users, e.g., GitHub users, which are mapped from the Git users included in the git-blame output. Git users that could not be automatically mapped are marked with *. To map these users, you can add user_mapping see instructions here.

For example:

automations:
  the_right_reviewer:
    if:
      - true
    run:
      - action: add-reviewers@v1
        args:
          reviewers: {{ repo | rankByGitBlame(gt=50) }}
      - action: add-comment@v1
        args:
          comment: |
            {{ repo | explainRankByGitBlame(gt=50) }}

Note the comment starts with | and a new-line as explainRankByGitBlame generates a multiline comment.

isFirstCommit

Return true if it's the author first commit in the repo.

if:
  - {{ repo.contributors | isFirstCommit(branch.author) }}
run:
  - action: add-comment@v1
    args:
      comment: "Welcome {{branch.author}}!"

isFormattingChange

Return true if all file diffs are validated as formatting changes. This filter function works for JavaScript, TypeScript, Python, JSON, YAML and HTML.

gitStream determines formatting changes by minifying the source code for the incoming changes and the existing code and comparing them. If they are identical, this filter function returns true. If any unsupported languages are contained in the PR, gitStream will return false.

{{ source.diff.files | isFormattingChange }}

mapToEnum

Get the enum value matches to the input key

For example, set a label color according to names in the enum:

automations:
  label_color:
    if:
       - true
    run:
      - action: add-label@v1
        args:
          label: 'Blue label'
          color: {{ "blue" | mapToEnum(enum = colors) }}

colors:
  red: 'FF0000'
  green: '00FF00'
  blue: '0000FF'
  yellow: 'FFFF00'

matchDiffLines

Checks diff for matching lines.

For example, to check if all the changes are of adding prints and ignore white spaces:

{{ source.diff.files | matchDiffLines(regex=r/^\+.*console\.log/, ignoreWhiteSpaces=true) | every }}

rankByGitActivity

Get list of contributors based on git-commit activity.

The repo context includes all the changed files, for each file it includes each contributor number of lines changed every week over the last 52 weeks, based on git-commit.

These functions compare each contributor changes per week and yield an average percentage of contribution for any given file. For example, in a certain week a file had 500 line changed, 200 by a first user, while 3 other users changed 100 lines each. So the score for the first user in that week will be 40 (200/500 in %). The function then average the score for each user for the selected time period.

Then you can use the thresholds to get the right reviewer.

Check if the branch author is a rookie

active_coders: {{ repo | rankByGitActivity(gt=50, weeks=12) }}

rankByGitBlame

Get list of contributors based on git-blame results

The repo context includes all the changed files, for each file it includes the contributors' percentage of lines in the file, based on git-blame.

This function sums all these percentages per user and yield an average percentage of contribution. Then you can use the thresholds to get the right reviewer.

The output lists the Git provider users, e.g., GitHub users, which are mapped from the Git users included in the git-blame output. When gitStream cannot map the Git user to a Git provider user it will be dropped from the output list, hence the list may contain less than 100% of the lines.

Example of the filter output, note the output are GitHub users in the example:

[
  "PopeyeUser",
  "olive_user",
  "BRUTUS_USER"
]

Get the most significant contributors for the PR files:

contributors: {{ repo | rankByGitBlame(gt=30) }}

Check if the branch author is a rookie

is_rookie: {{ repo | rankByGitBlame(lt=15) | match(term=branch.author) | some }}

readFile

Reads the contents of a file from the current branch or the cm repo and returns it as a string.

Examples:

Add a comment with a file's content:

automations:
  add_readme_comment:
    if:
      - true
    run:
      - action: add-comment@v1
        args:
          comment: |
            {{ README_CONTENT }}

README_CONTENT: {{ "./README.md" | readFile() }}

Read JSON configuration file from the cm repo and use some of the properties in a comment:

automations:
  describe_teams:
    if:
      - {{ true }}
    run:
      - action: add-comment@v1
        args:
          comment: |
              We have {{ TEAMS | length }} teams with {{ TEAMS.front.members | length + TEAMS.back.members | length }} members in total:
              FrontEnd: include {{ TEAMS.front.members | length }} members
              BackEnd: include {{ TEAMS.back.members | length }} members


TEAMS: {{ "../cm/TEAMS.json" | readFile(output="json") }}

Configuration file example:

{
  "front": {
    "name": "Frontend",
    "description": "Frontend team",
    "members": [
      "John",
      "Jane"
    ]
  },
  "back": {
    "name": "Backend",
    "description": "Backend team",
    "members": [
      "Alice",
      "Bob"
    ]
  }
}