Maximize Code Quality with Minimal Effort

Intro

---

Pobody's nerfect, or so they say. Which is why people much smarter than me figured out ways to minimize our flaws through carefully-crafted tools and techniques.

But the problem is I'm both flawed and lazy. Lucky for us both, there are ways around that too. By taking advantage of a few tools at nearly everyone's disposal, we can drastically improve the quality of our work. And when all is said and done, we won't even have to lift a finger to know how to do it.

SAST

---

Expressed in terms of effort-to-implement vs potential payoff, SAST (Static Application Security Testing) is basically a must in any kind of dev environment. You don't have to spring for a very fancy scanner to be told where your code is vulnerable and, just as importantly, how to fix it.

SAST scanning allows you to identify bugs early on in your lifecycle and can potentially save you a mountain of grief down the road.

Like anything else, it isn't a silver bullet and should only be a single tool at your disposal, but nevertheless should be in your arsenal all the same.

Bandit

Bandit is one SAST tool that I use in nearly every Python-based project. Developed by the Python Security Response Team (PSRT), it's open-source, easy to use, and can be integrated into your CI/CD pipeline very quickly (more on that later). Generally run from the CLI, it generates a report that highlights potential security issues and categorizes them based on confidence and severity.

Linting

---

In a perfect world, we would all code exactly alike and we would do so perfectly every time. But to err is human, as they say, so we have things called 'linters' that ensure we code somewhat alike, at least some of the time. Linters check to ensure our projects are syntactically correct and conform to agreed-upon standards to preserve readability and maintainability so we don't drive anyone to madness should they dive into our codebase.

Flake8

Flake8 is one such linter which works hard to ensure my Python projects are slightly less of a trainwreck. It checks for things like:

• Syntax errors
• PEP8 non-compliance, which includes things like:
• Over/under indentation
• Line length
• Whitespace
• Import order
• Unused variables/imports/etc
• Redundant code
• Light type checking

Below you can see a couple of Errors (in red) and Warnings (in yellow). I assure you, I intentionally eroded quality here to give Flake8 something to do.

Pictured above: Flake8 after a long day

Work Smarter

---

Now, linting and scanning might only take a couple more minutes of your time, but that will add up in short order - Especially if you're a busy dev. Remember, if we can automate something safely, securely, and sensibly, we should've done so yesterday.

Actions and Workflows

Github Workflows are automations that live in your repositories and can be triggered by specific events like push/pull requests or run on a schedule. They are defind by YAML files located in the .github/workflows directory of your repo and can be as simple or complex as you need them to be.

The business logic of Workflows is composed of reusable functions called Actions. Actions can written from scratch in just about any language, or you can get them prebuilt from the Github Marketplace.

Each Action is composed of one or more steps that define the individual tasks that make up the Action. These steps can be run on different machines, in parallel, or in a specific order.

Here's a very simple example of a Workflow that echoes a message to the console:

name: Echo Message
branches:
  only:
    - dev
on: [push]
jobs:
  echo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Echo Message
        run: echo "Workflow completed!"

Let's break down the major components of this Workflow: - name is the title of the Workflow - branches define the branches on which our Workflow will run - on define events that will trigger our Workflow (in this case, push events) - jobs define the Actions that make up our Workflow - echo is the name of our Action - runs-on defines the machine on which our Action will run - steps define the individual tasks that make up our Action - uses defines the Action that will be used - name is the title of the step - run is the actual command that will be executed

Putting it all together

---

Now that we understand the ins and outs of SAST, linting, and Workflows, it's time to take the quality of our code to the next level. Let's create a basic Workflow that will run Bandit and Flake8 against our repo.

name: Code Quality and Security Check

# Trigger the workflow on pushes/PRs to the main branch
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest # Run on the latest version of Ubuntu

    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v3
      with:
        python-version: '3.11'
    - name: Install Dependencies # Setup our tools
      run: |
        pip install bandit
        pip install flake8

    # We've included '|| true' after our commands below to ensure the entire Workflow doesn't fail if there are any issues
    - name: Run Bandit
      run: bandit -r . || true # Use the -r flag to recursively scan our repo
    - name: Run Flake8 || true
      run: flake8 .

Now SAST scanning and linting will run automatically whenever a push or PR occurs against the main branch. If there are any issues, they will be reported in the Actions tab. But wait...

There's more

Ideally, an issue should probably be opened if there are any problems with our code's quality, security, or both. But I'm especially lazy, so not only do I not want to open these issues myself, I don't even want to log into Github to see if there are any. To do that, we'll need to:

1. Create a simple script to open issues for us
2. Adjust our bandit/flake8 commands to dump their findings into a file
3. Add a step to upload said file as an artifact
4. Add a step to run the script we created in step 1

Issue Automation

This will be fairly straightforward - We'll have a function to read in the contents of our reports, and another to create an issue if findings are present. We'll use a couple of environmental variables to authenticate to Github's API and identify the repo under which it's running, but no need to worry about these right away - We'll supply them through our Workflow file.

You can place this script anywhere you'd like, just make sure you update the path in your Workflow file accordingly.

import os
import requests


def create_github_issue(title: str, body: str, token: str, repo: str) -> bool:
    """    Create an issue on github using the given parameters.

    Args:
        title (str): Title of the issue.
        body (str): Body of the issue.
        token (str): Github token.
        repo (str): Github repository.

    Returns:
        bool: True if the issue was created successfully, False otherwise.
    """

    headers = {
        "Authorization": f"token {token}",
        "Accept": "application/vnd.github.v3+json"
    }

    data = {"title": title,
            "body": body}

    response = requests.post(f"https://api.github.com/repos/{repo}/issues",
                             json=data, headers=headers, timeout=10)

    if response.status_code == 201:
        print(f"Issue created: {response.json()['html_url']}")
        return True

    print(f"Failed to create issue: {response.content}")
    return False


def read_report(file_path: str) -> str:
    """Read the given file and return its contents.

    Args:
        file_path (str): File path to read.

    Returns:
        str: Contents of the file.
    """

    try:
        with open(file_path, 'r') as file:
            return file.read().strip()

    except FileNotFoundError:
        return ""


def main():
    bandit_report = read_report("bandit-report.txt")
    flake8_report = read_report("flake8-report.txt")

    if (bandit_report and 'No issues identified' not in bandit_report) or flake8_report:
        title = "Code Quality Issues Detected"
        body = f"**Bandit Findings:**\n```\n{bandit_report}\n```\n**Flake8 Findings:**\n```\n{flake8_report}\n```"
        token = os.getenv("GITHUB_TOKEN")
        repo = os.getenv("GITHUB_REPOSITORY")

        if token and repo:
            create_github_issue(title, body, token, repo)
        else:
            print("Environment variables GITHUB_TOKEN and GITHUB_REPOSITORY are required")
    else:
        print("No issues found in Bandit and Flake8 reports.")


if __name__ == "__main__":
    main()

Setting Up Our Notifications

I'm going to relay our notifications through a Tines-based Slack bot I've built. This gives me a little more flexibility in terms of how and where I'm notified, but you can use whatever you'd like (including Github's own Slack-based notifications).

First, I'll have Github send a POST request to a Tines Webhook I've setup. This is done using Github's Webhooks, which can be found in the Settings tab of your repo. You'll want to create a new Webhook with the following settings:

• Payload URL: The URL of your Tines Webhook
• Content Type: application/json
• Events: Issue Comments, and Issue activity in general (opened/closed/etc)

Then I'll head over to Tines and add two more Actions to my story board: A trigger to check that the action is 'opened', and an HTTP request to relay the Slack DM:

Interested in copying the actions into a story of your own? Grab the JSON below:

    {"standardLibVersion":"40","actionRuntimeVersion":"5","agents":[{"disabled":false,"name":"Receive Issue Notifications","description":null,"options":"{\"path\":\"6e7d1688da8e42b2615173bec115fb9c\",\"secret\":\"4a1b997c39f032fca8570513ca61cd8b\",\"verbs\":\"get,post\"}","position":{"x":810,"y":120},"type":"webhook","timeSavedUnit":"minutes","timeSavedValue":0,"monitorAllEvents":false,"monitorFailures":false,"monitorNoEventsEmitted":null,"recordType":null,"recordWriters":[],"form":null,"cardIconName":null,"createdFromTemplateGuid":null,"createdFromTemplateVersion":null,"originStoryIdentifier":"cloud:98bf19cf1391e1805daf0cbdc3239e4a:bae0d194c24575af33284228ec0a2a2d"},{"disabled":false,"name":"Issue Opened","description":null,"options":"{\"rules\":[{\"type\":\"field==value\",\"value\":\"opened\",\"path\":\"<<receive_issue_notifications.body.action>>\"}]}","position":{"x":810,"y":225},"type":"trigger","timeSavedUnit":"minutes","timeSavedValue":0,"monitorAllEvents":false,"monitorFailures":false,"monitorNoEventsEmitted":null,"recordType":null,"recordWriters":[],"form":null,"cardIconName":null,"createdFromTemplateGuid":null,"createdFromTemplateVersion":null,"originStoryIdentifier":"cloud:98bf19cf1391e1805daf0cbdc3239e4a:bae0d194c24575af33284228ec0a2a2d"},{"disabled":false,"name":"Send Slack DM","description":null,"options":"{\"url\":\"https://slack.com/api/chat.postMessage\",\"content_type\":\"application_json\",\"method\":\"post\",\"payload\":{\"channel\":\"C068DA8AKC4\",\"blocks\":[{\"type\":\"header\",\"text\":{\"type\":\"plain_text\",\"text\":\"Issue Created\",\"emoji\":true}},{\"type\":\"divider\"},{\"type\":\"section\",\"fields\":[{\"type\":\"mrkdwn\",\"text\":\"*Issue:* \\\\<<<receive_issue_notifications.body.issue.html_url>>|<<receive_issue_notifications.body.issue.title>>>\"},{\"type\":\"mrkdwn\",\"text\":\"*User:* <<receive_issue_notifications.body.issue.user.login>>\"},{\"type\":\"mrkdwn\",\"text\":\"*Repo:* <<receive_issue_notifications.body.repository.name>>\"},{\"type\":\"mrkdwn\",\"text\":\"*Created:* <<receive_issue_notifications.body.issue.created_at |> DATE(%, \\\"%Y-%m-%d\\\")>>\"}]}]},\"headers\":{\"Authorization\":\"Bearer <<CREDENTIAL.slack_qa_token>>\"}}","position":{"x":810,"y":345},"type":"httpRequest","timeSavedUnit":"minutes","timeSavedValue":0,"monitorAllEvents":false,"monitorFailures":false,"monitorNoEventsEmitted":null,"recordType":null,"recordWriters":[],"form":null,"cardIconName":null,"createdFromTemplateGuid":null,"createdFromTemplateVersion":null,"originStoryIdentifier":"cloud:98bf19cf1391e1805daf0cbdc3239e4a:9f14dfcd7d12904a7c5651dbe89b3389"}],"links":[{"sourceIdentifier":0,"receiverIdentifier":1},{"sourceIdentifier":1,"receiverIdentifier":2}],"diagramNotes":[]}

Updating Our Workflow

The reason we created our script first is because our updated Workflow will be looking for it in the .github/workflows directory, otherwise it'll fail when it gets to that step. So now that that's out of the way, we can revamp our Workflow a bit. Additionally, if an issue is opened, I'd like to test our bot's functionality at the same time.

The main changes are as follows:

1. Added the permissions our script needs to create issues and read our repo
2. Added requests to our dependencies
3. Dump our findings into their respective reports
4. Upload our reports as artifacts
5. Finally, kick off our script using the environmental variables we defined

name: Python Code Quality and Security Check

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

# Important! We need to give our script permission to create issues and read our repo
permissions:
  issues: write
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v3
      with:
        python-version: '3.11'

      # Add requests to our dependencies
    - name: Install Dependencies
      run: |
        pip install bandit
        pip install flake8
        pip install requests

      # Dump our findings into reports with expected names
    - name: Run Bandit
      run: bandit -r . > bandit-report.txt || true
    - name: Run Flake8
      run: flake8 --max-line-length 120 . > flake8-report.txt || true

      # Upload our reports as artifacts
    - name: Upload Reports as Artifacts
      uses: actions/upload-artifact@v3
      with:
        name: reports
        path: |
          bandit-report.txt
          flake8-report.txt

      # Run our new script using the pertinent environmental variables
    - name: Check for Findings and Create Issue
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        GITHUB_REPOSITORY: ${{ github.repository }}
      run: python ./workflows/create_scan_issues.py  # This is where I put our script, but you can put it wherever you'd like

Commit and push your changes, and if all goes well you're likely to see a new issue in just a couple of minutes. Since my code is never perfect, I had a new issue waiting for me - But thanks to Tines, I didn't have to go to my email or log into Github to know about it:

Wrapping Up

---

We've covered a lot of ground here, so let's recap a bit. Now, SAST scanning and linting happens automatically with every update to our codebase. If there's a hitch with our code's quality, an issue is automatically opened so we can document and address these problems. Not only that, but we're notified wherever Slack is installed. And the best part? The hard part is done - Now we can focus on the code itself without getting bogged down with the needless minutiae. Far out, man.